## Summary
- add ZO_FEATURE_METRICS_PUSHDOWN_FILTER_ENABLED as an opt-in filter
pushdown switch for metrics queries
- group DataFusion and Tantivy feature configuration under the search
config struct
- move inverted-index result cache sizing, footer caches, skip
threshold, TopN group limit, and bloom footer cache settings from limit
to search
- preserve the moved cache settings' environment variable names and
defaults
- apply the metrics-specific setting to object storage, WAL, and Flight
query contexts
## Testing
- cargo fmt --all -- --check
- cargo check for config, infra, search, tantivy_utils, search_service,
and promql-service
- cargo test -p config every_env_config_default_parses --lib
- cargo test -p config inverted_index_config --lib
- cargo test -p search test_create_session_config --lib
- cargo clippy for all affected crates with -D warnings
## Summary
Two coupled alerting features:
**1. Notification Templates v2 ("content" templates)**
- New `kind: content` template type: a structured spec (title, body
markdown, fields, links, rows, per-channel title overrides,
severity-filtered visibility) rendered natively per channel — existing
`custom` (raw body) templates are untouched and byte-identical on the
send path.
- Per-channel renderers: Slack Block Kit, Teams (Adaptive Card +
MessageCard), Discord embeds, PagerDuty Events v2, Opsgenie, ServiceNow,
SNS, HTML/text email, and a stable machine `webhook` envelope.
- Escaping is renderer-owned (mrkdwn entities, HTML encoding, markdown
stripping) with a shared URL-scheme allowlist; golden-fixture tests per
channel.
- Template editor UI: guided content form (fields/links row tables,
compact rows config, channel title overrides), live preview panel with
per-channel preview + payload tab, test-send (rate-limited,
`[TEST]`-marked).
- Migration `m20260802_000001_add_template_kind`; prebuilt default
template + org backfill.
**2. Alert chart images (stateless, storage-free)**
- Alerts can attach a metric-history chart PNG rendered from the
triggers self-reporting stream (`actual_value` per evaluation), with
warn/crit threshold lines and the firing point appended.
- Zero storage: the chart URL *is* the data — deflate+base64url payload
signed with truncated HMAC-SHA256 (org-bound, expiry embedded), rendered
on fetch by the HTTP-terminating node (`GET
/api/v2/{org}/alerts/charts/render`). Signature verified before inflate
(decompression-bomb guard), uniform 404s, per-node render semaphore +
per-minute query budget + small LRU.
- Renderer: plotters + embedded DejaVu Sans (Bitstream Vera license
included), gap-broken line segments so sparse evaluation history doesn't
draw false plateaus.
- Channel delivery: Slack image block, email inline CID (no remote
fetch), Discord multipart attachment, Teams/PagerDuty image URLs;
per-destination opt-in via template chart flag.
- Slack layout live-A/B'd on a real channel: content + buttons/chart
split across two same-colored attachments so the severity stripe spans
everything while staying under Slack's "Show more" fold; clickable link
title; "View in OpenObserve" keeps a reserved primary-button slot.
- Slack image-delivery fallback: Slack validates image URLs server-side
and rejects the ENTIRE message with `400 invalid_attachments` when its
proxy can't fetch the chart (VPN-only/private `ZO_WEB_URL`) —
live-verified. On that rejection the send strips the image blocks and
resends once (alert always delivers), sets a process-wide flag so later
sends pre-strip (reachability is deployment-wide, not per-destination),
and re-probes hourly. Zero config; observable via `slack_image_rejected`
/ `slack_image_pre_stripped` metric events.
## Testing
- 147 notification unit tests (resolve/render/preview/test-send incl.
per-channel goldens and hostile-input/injection coverage), 8 chart
payload tests, renderer PNG tests.
- Live-verified end to end: Slack (Slackbot chart fetches ~100ms,
cache-immutable semantics), SES email inline image, multi-severity
firing scenario (warning→critical) with correct threshold lines.
- FE: AddTemplate/ContentTemplateForm/preview specs; UX audit pass with
fixes.
- `cargo clippy -D warnings` clean in both OSS and enterprise
arrangements.
## Enterprise pairing
Must merge together with openobserve/o2-enterprise#2353 (workspace
twin-manifest mirror for the new OSS deps).
Bumps the `openobserve/vortex` pin from `57ffbf2b` to `1cb48912` in the
workspace manifest.
## Upstream API changes handled
- Arrow interop moved into a dedicated `vortex-arrow` crate, surfaced as
`vortex::arrow`. `vortex::array::arrow::*` and `vortex::dtype::arrow::*`
no longer exist, so `FromArrowArray`, `FromArrowType`, `ToArrowType` and
`ArrowSessionExt` are now imported from `vortex::arrow`.
- `ToArrowType` is no longer in scope implicitly; added the explicit
import where `DType::to_arrow_schema()` is called.
- The `vortex::compressor` facade narrowed to `BtrBlocksCompressor`,
`BtrBlocksCompressorBuilder`, `Scheme`, `SchemeId`. `SchemeExt` and the
scheme list are no longer re-exported there, so `search` takes a direct
`vortex-btrblocks` dependency (same git rev) for `SchemeExt` /
`schemes::integer::IntDictScheme`, which the UTF8 compressor uses to
exclude the int-dict scheme.
No behavior change intended — the compressor configuration and write
strategies are identical.
## Verification
- `cargo check --workspace --all-targets` clean
- `cargo fmt --all -- --check` clean
- `cargo test -p search --lib datafusion::vortex` — 6 passed
- `cargo test -p tantivy_utils --lib vortex` — 3 passed
- `cargo test -p config --lib utils::parquet` — 21 passed
Paired enterprise PR (same branch name):
https://github.com/openobserve/o2-enterprise/pull/2303
## Summary
- decouple stream cleanup from enrichment-data and remove the transitive
compaction/DataFusion dependency from stream
- extract sourcemaps and organization storage services from core into
dedicated crates
- extract GitHub built-in regex access and model pricing into
openobserve-builtins
- rename pipeline and organization-storage persistence modules from
store to db
- remove now-unused direct dependencies from core
- synchronize the enterprise workspace overlay in
openobserve/o2-enterprise#2266
## Validation
OSS:
- cargo check -p openobserve-builtins -p openobserve-core -p
openobserve-jobs -p openobserve-api
- cargo test -p openobserve-builtins --lib
- cargo test -p openobserve-builtins --doc
- cargo clippy -p openobserve-builtins -- -D warnings
- cargo check -p openobserve-org-storage --features enterprise
Enterprise overlay:
- copied ../o2-enterprise/Cargo.toml.openobserve over Cargo.toml
- cargo fmt --all -- --check
- cargo check --all-targets --no-default-features --features enterprise
- cargo clippy --all-targets --no-default-features --features enterprise
-- -D warnings
The default enterprise + vectorscan build was attempted and reached
vectorscan-rs-sys, but the validation host does not have cmake
installed. The enterprise-only build and clippy run completed
successfully.
## Summary
Moves Vortex file-format support out of the enterprise build and makes
it a core, always-compiled capability in open source.
- **Code move**: the enterprise `search/vortex.rs` module
(`Utf8Compressor`, `VORTEX_RUNTIME`, `write_vortex`,
`generate_vortex_access_plan`, plus its unit tests) now lives at
`src/search/src/datafusion/vortex.rs`; the three call sites
(`merge/downsampling.rs`, `merge/mod.rs`, `table_provider/helpers.rs`)
reference the local module.
- **Feature removal**: the `vortex` cargo feature is deleted from the
root and all 11 sub-crate manifests. All `#[cfg(feature = "vortex")]` /
`#[cfg(all(feature = "enterprise", feature = "vortex"))]` gates and
their "requires the vortex feature" error branches are gone —
`FileFormat::Vortex` match arms compile unconditionally in config,
search, and tantivy_utils.
- **Dependencies**: `vortex` / `vortex-datafusion` are now direct git
dependencies (`openobserve/vortex` rev `57ffbf2b`, the same rev the
enterprise workspace pinned); the empty dependency-resolution stub
crates under `src/enterprise/vortex{,-datafusion}` are deleted.
Companion PR in o2-enterprise removes the vortex module/feature there
and syncs `Cargo.toml.openobserve`; that PR should merge together with
(or right after) this one, since its template relies on this repo no
longer declaring the feature.
## Verification
- `cargo check --workspace` (OSS, default features) passes
- `cargo check -p search -p tantivy_utils -p config --all-targets`
passes (vortex tests compile)
- Enterprise combined workspace (`Cargo.toml.openobserve` swapped in)
`cargo check -p openobserve` passes
- `cargo fmt` applied to touched crates
## Summary
This PR begins decomposing `openobserve-core` into focused service
crates while preserving the existing application behavior and dependency
direction.
It extracts six independently owned areas:
- `ingestion-common`: shared ingestion contracts and response types
- `schema`: schema inference, validation, evolution, and persistence
helpers
- `compaction`: merge, retention, and compaction services
- `enrichment-data`: enrichment-table lifecycle and runtime support
- `promql-service`: PromQL service orchestration
- `stream`: stream metadata, settings, retention, and lifecycle
operations
## What changed
- registered the extracted crates in the workspace and propagated
enterprise/cloud/vectorscan/vortex features
- moved implementations out of Core and updated API, Jobs, query, and
queue consumers to use the new crates
- removed Core facade aliases for extracted crates and converted
remaining callers to direct crate paths
- retained the small cross-service stream-delete coordinator in Core for
now because it still coordinates Pipeline and Alert cleanup
- restored required root/Core dependencies after validating the full
enterprise build
- added the missing `anyhow` dependency for enterprise-only
`search_service::search_jobs`
- updated the lockfile for the new crate dependency graph
## Dependency direction
The extracted crates do not depend on `openobserve-core`. Core remains a
temporary upper-level consumer while the remaining domain services are
split incrementally.
```text
ingestion-common -> common/config
schema -> ingestion-common/db/infra
stream -> schema/enrichment-data/db/infra
compaction -> schema/search/search-service/db/infra
openobserve-core -> ingestion-common/schema/stream/compaction/...
api/jobs -> openobserve-core + extracted services
```
## Enterprise compatibility
The private OpenObserve manifest was updated in
openobserve/o2-enterprise#2257 to register the extracted workspace
members and dependencies.
Related Enterprise PR:
https://github.com/openobserve/o2-enterprise/pull/2257
## Validation
OSS manifest:
- `cargo fmt --all -- --check`
- `cargo clippy -- -D warnings`
Using `o2-enterprise/Cargo.toml.openobserve` as the OpenObserve root
manifest:
- `cargo check --no-default-features --features enterprise`
- `cargo check --no-default-features --features cloud`
- `cargo clippy --workspace --no-default-features --features enterprise
-- -D warnings`
- `cargo clippy --workspace --no-default-features --features cloud -- -D
warnings`
Enterprise repository:
- `cargo fmt --all -- --check`
- `cargo clippy -- -A clippy::question-mark -D warnings`
All checks above pass in the development profile.
## Summary
- remove compatibility re-exports introduced during the recent crate
refactors
- migrate callers to the crates that own each API
- delete five pure facade modules and add the required direct crate
dependencies
- keep OSS and enterprise manifests aligned
## Validation
- cargo fmt --all -- --check
- cargo clippy --workspace -- -D warnings
- cargo test --locked --workspace --no-run
- enterprise overlay: cargo test --offline --workspace
--no-default-features --features enterprise --no-run
- enterprise overlay: cargo clippy --offline --workspace
--no-default-features --features enterprise -- -D warnings
- git diff --check
Companion enterprise PR uses the same branch name:
codex/remove-refactor-reexports.
## TODO / Follow-ups
- [ ] **Re-establish the pipeline write-guardrail lost with
`db::pipeline`.** That module was not a pure facade: it re-exported only
the six read-only accessors from `infra::pipeline` and its doc comment
recorded the invariant that *mutations must go through
`service::pipeline::store`* (which keeps the `ExecutablePipeline` caches
and the coordinator/super-cluster watch events in sync). Deleting it
points callers straight at `infra::pipeline`, which also exposes
`put`/`delete`/`drop_table` with no guardrail. Decide the layering and
enforce it:
- **Reads** may hit `infra` directly (raw persistence, no side effects).
- **Writes** must go through the wrapper that persists *and* emits the
coordinator/super-cluster event + refreshes the in-memory cache. That
wrapper is `db::<entity>::{set,delete}` for most entities (e.g.
`db::alerts::alert::set` →
`infra::coordinator::alerts::emit_put_event`), but for pipeline it is
`core::pipeline::store` (not `db`), because the
`STREAM_EXECUTABLE_PIPELINES` cache it maintains is a core-level
construct the `db` crate cannot build. The super-cluster receiver
(`super_cluster_queue::pipelines`) is the one sanctioned caller of
`infra::pipeline::{put,delete}` directly, and it manually pairs each
with `emit_*_event` to avoid a re-broadcast loop.
- Net invariant to protect: never call `infra::pipeline::{put,delete}`
without emitting the coordinator event — i.e. nowhere outside
`core::pipeline::store` and the super-cluster handler. A bare `infra`
write leaves in-memory caches stale until restart (see the warning
comment still live at `src/core/src/org_cleanup.rs:484`).
- [ ] Consider whether the pipeline write-wrapper living in
`core::pipeline::store` while every other entity's lives in
`db::<entity>` is worth unifying, or at least documenting as an
intentional exception.
## Summary
- extract persistence, cache, watcher data access, and DB service
modules from openobserve-core into a standalone workspace db crate
- keep application orchestration that depends on search, organization
provisioning, and other Core services behind the existing Core db facade
- preserve existing call paths through re-exports while making the
dependency direction openobserve-core -> db
- move HTTP error mapping back to the Core/API boundary so db has no
direct axum dependency
- avoid provider/provisioner callback traits and function-pointer hooks
## Validation
- cargo fmt --all
- cargo check --all-targets
- cargo check -p db --all-targets
- Enterprise workspace: cargo check --all-targets --offline
- Enterprise workspace: cargo check -p db --all-targets --features
enterprise --offline
Companion Enterprise manifest PR: openobserve/o2-enterprise#2230.
## Summary
Two crate-split refactors continuing the workspace decomposition:
**1. `transform` crate (renamed from `openobserve-vrl`, `src/vrl` →
`src/transform`)**
- Now the only crate depending on `vrl`, `vector-enrichment` and
`rquickjs` directly; downstream crates use `transform::vrl::` /
`transform::vector_enrichment::` re-export paths and drop their direct
dependencies
- The QuickJS runtime (`common/utils/js.rs`) moved into `transform::js`
(js_security checks gated behind the crate's `enterprise` feature);
`apply_vrl_fn`, `convert_to_vrl`/`convert_from_vrl` and
`TRANSFORM_FAILED` moved from core
- Exception: `src/config` keeps its direct `vrl` dependency (it sits
below `transform` in the dependency graph)
**2. `super_cluster_queue` extracted from the root crate into a
workspace member**
- Only depends on `openobserve-core`, `infra`, `config` and
`o2_enterprise`, so it builds in parallel with `api` and edits no longer
recompile the root crate (which re-embeds `web/dist` on every rebuild)
- Gated behind its own `enterprise` feature — compiles as an empty crate
in OSS builds so `cargo --workspace` keeps working against the
`o2_enterprise` stub; `vectorscan` wired through for `re_pattern.rs`
- Root `lib.rs` re-exports the crate, so
`openobserve::super_cluster_queue::init()` in `main.rs` is unchanged
Pairs with the `Cargo.toml.openobserve` template sync PR in
o2-enterprise (must merge together).
## Test plan
- `cargo check --workspace --all-targets` (OSS) passes
- `cargo check -p openobserve --no-default-features --features
enterprise` with the synced enterprise template passes
- `cargo clippy -p super_cluster_queue --features enterprise,vectorscan`
— zero warnings
- `cargo clippy -- -D warnings` (CI invocation) passes; `cargo fmt
--all` clean
## Summary
- move the MCP protocol, tool discovery, schema simplification,
execution, and response filtering implementation into a new OSS
`openobserve-mcp` workspace crate
- enable the real MCP POST, GET, and DELETE handlers in community builds
- initialize the MCP tool catalog from the OpenAPI document for every
edition
- keep OAuth authorization-server discovery enterprise-only because it
depends on Dex
- add OSS handler and OpenAPI initialization coverage
Closes#13266
Design at: #13266
## Impact
Community users can connect MCP clients directly to OpenObserve and use
the OpenAPI-backed tool catalog. Enterprise builds continue to use the
same MCP implementation from the OSS workspace crate.
Companion PR openobserve/o2-enterprise#2206 removes the old private copy
and updates the enterprise workspace overlay.
## Summary
- add a dedicated promql workspace crate for parsing and evaluation
- keep cluster routing, cache, WAL, file-list, and request orchestration
in openobserve-core
- preserve existing service::promql call sites through a compatibility
facade; the facade re-exports an explicit item list (not a glob), and
the promql search orchestration in core depends on the promql crate
directly
- inject enterprise query cancellation through TableProvider instead of
coupling the evaluator to core search state
- replace the core regex-UDF dependency with DataFusion regexp_like
expressions
## Companion PR
- openobserve/o2-enterprise#2198 updates the enterprise workspace
override for the new promql crate
## Validation
- cargo fmt --all -- --check
- cargo check --workspace
- cargo clippy -p promql --all-targets -- -D warnings
- cargo clippy -p openobserve-core
- cargo test -p promql: 363 passed
- verified the promql dependency tree has no openobserve-core or search
dependency
- cargo check -p o2_enterprise --lib in the paired private workspace
## Notes
The evaluator moves atomically in this PR. The existing
service::promql::search subtree remains in core because it owns
OpenObserve-specific orchestration rather than PromQL evaluation.
The two web/ file changes only update source-path references in comments
that pointed at the moved Rust files (src/service/promql/... ->
src/promql/src/...).
## Summary
- add `src/vrl` as a leaf package named `openobserve-vrl` with Rust path
`openobserve_vrl`
- move `QUERY_FUNCTIONS`, enrichment-table ownership, GeoIP/MaxMind
table registration, VRL compiler configuration, and shared runtime
helpers into `openobserve-vrl`
- let ingestion, jobs, core, and search use the shared VRL integration
without depending on one another
- keep DataFusion UDF registration direct through `Sql.org_id`; no
`QueryExecutionContext`, `PreparedQueryTransform`, or
`prepare_query_transforms`
- move the cipher registry, query parameters, and file-cache helpers to
`search`
- move the generic file downloader implementation to `infra`
- add feature forwarding for the `search` crate
- keep compatibility re-exports for existing API, ingestion, gRPC, jobs,
and Enterprise call sites
Enterprise companion:
https://github.com/openobserve/o2-enterprise/pull/2196
## Summary
- add the minimal `openobserve-search` workspace crate and move shared
search/cache types into it
- move the concrete enrichment table type and cache ownership to the
search crate while retaining normal compatibility re-exports
- move trace schema constants to `config` so the SQL engine no longer
reaches into the traces service
- expose concrete enrichment read functions from `infra` and make
DataFusion call them directly
- construct the stateless VRL runtime directly in the transform UDF
- keep search execution and orchestration in `openobserve-core` for the
later physical moves
- do not introduce `SearchRuntime`, provider traits, callbacks,
`#[path]`, or a search-to-core dependency
This combines the original crate-boundary step with the dependency
cleanup needed before SQL/DataFusion files can move safely.
## Validation
- `cargo test -p openobserve-search --lib` (11 passed)
- `cargo test -p openobserve-core service::enrichment::tests --lib` (19
passed)
- `cargo check -p openobserve --features mimalloc`
- enterprise + mimalloc check using the o2-enterprise overlay
- cloud + mimalloc check using the o2-enterprise overlay
- `cargo fmt --all -- --check`
- strict all-target clippy for `openobserve-search` and
`openobserve-core`
- strict library clippy for `infra`
- dependency check confirms `openobserve-search` does not depend on
`openobserve-core`
`infra --all-targets` currently reports three pre-existing inconsistent
digit-grouping warnings in
`src/infra/src/table/entity/alert_dedup_state.rs`; they are unrelated to
this change and are not included here.
## Problem
Every rebuild of `openobserve-api` re-runs the `RustEmbed` macro
expansion, which embeds `web/dist` (53MB, 1035 files) and compresses
every file with gzip + brotli(q11) — single-threaded, non-incremental,
and it peaks at 8GB RSS. `-Ztime-passes` attributes ~540s of the 769s
api compile to `expand_crate`. Net effect: touching one file in
`openobserve-core` costs a **16m45s** rebuild on an M-series laptop, and
neither incremental compilation nor codegen-unit tuning can help,
because macro expansion happens before both.
## Fix
Move the `#[derive(RustEmbed)]` into a new `src/web` leaf crate with no
workspace dependencies. The embed + compression now only recompiles when
`web/dist` changes; edits anywhere in the backend no longer pay for it.
**Measured (same machine, default profile settings, `--profile
release-profiling`):**
| | before | after |
|---|---|---|
| `openobserve-api` compile | 769s | 231s |
| touch `src/core` → binary | 16m45s | **6m44s** |
## Also in this PR
- **`src/web/build.rs`**: the derive has no file tracking, so a rebuilt
`web/dist` silently shipped stale embedded assets until something else
dirtied the api crate. `rerun-if-changed` on the embed folder fixes
that. The folder is parsed from the `#[folder]` attribute rather than
hardcoded, so it stays correct when the e2e workflows rewrite it. A
missing `index.html` now prints a build warning instead of silently
embedding nothing (the #13197 failure mode).
- **Playwright workflows**: the embed-folder `sed` now targets
`src/web/src/lib.rs` and is guarded with `grep -q`, so a future path
drift fails at the sed step instead of silently shipping an empty UI and
failing at the login page (see openobserve/o2-enterprise#2175 for that
incident).
- **`deny.toml`**: allow the new crate's AGPL-3.0 license (CI runs
`cargo deny check licenses`, which requires a per-crate exception).
## Validation
- `cargo deny check licenses` passes
- Smoke-tested the binary: `/healthz` ok, `/web/` serves `index.html`,
JS assets return 200 from the embedded bundle
- Enterprise overlay resolves: `cargo metadata` passes with the
companion `Cargo.toml.openobserve` change
## Companion
openobserve/o2-enterprise#2184 carries the overlay + e2e workflow sync
on the **same branch name** (`perf/web-embed-crate`); the two should
merge together.
## Summary
- add the `openobserve-api-management` crate under `src/api_management`
- physically move management handlers, models, and authentication
helpers out of `openobserve-api`
- preserve the existing API module paths through re-exports
- move shared stream permission checking into `openobserve-core` to
avoid a management-to-query dependency
- allow the new crate's `AGPL-3.0` license in `deny.toml` (fixes the
cargo-deny `check` job)
- keep router, OpenAPI generation, and rate-limit path mapping in
`openobserve-api`
## Dependency boundary
`openobserve-api-management` depends on `openobserve-api-common` and
`openobserve-core`; it does not depend on `openobserve-api-query`.
## Note on `split-debuginfo`
Removing `split-debuginfo = "packed"` from `[profile.release-profiling]`
is intentional (not a formatting artifact): profiling builds now keep
DWARF debuginfo embedded in the binary, so profiling artifacts can be
symbolized at runtime without shipping a separate debuginfo file. The
o2-enterprise overlay `Cargo.toml.openobserve` is kept in sync.
## Companion PR
- openobserve/o2-enterprise#2175
## Validation
- OSS check with `mimalloc`
- OSS strict clippy with all targets
- enterprise check and strict clippy
- cloud check and strict clippy
- `cargo test -p openobserve-api-management --lib`: 300 passed
- `cargo deny check licenses`
- `cargo fmt --all`
- `git diff --check`
Build-time benchmarking is intentionally deferred for this stage.
## Summary
- add `openobserve-api-common` for HTTP extractors and request DTOs
shared across API domain crates
- add `openobserve-api-query` and physically move the search, PromQL,
and traces handlers (about 16.4K lines)
- preserve the existing `handler::http::request::{search,promql,traces}`
paths through re-exports so routing, Swagger generation, and rate-limit
metadata remain compatible
- forward enterprise, cloud, vectorscan, and vortex features through the
new crate boundary
- avoid `#[path]` source inclusion
## Enterprise overlay
Requires openobserve/o2-enterprise#2174, which registers the two new
workspace crates in `Cargo.toml.openobserve`.
## Validation
- `cargo check -p openobserve --features mimalloc`
- enterprise check with `--no-default-features --features
enterprise,mimalloc`
- cloud check with `--no-default-features --features cloud,mimalloc`
- OSS and cloud/enterprise all-targets Clippy with `-D warnings`
- `openobserve-api-common`: 9 tests passed
- `openobserve-api-query`: 136 independent tests passed single-threaded;
the unchanged DB-dependent saved-view test was excluded because
standalone crate tests do not initialize the infra database
- `cargo fmt --all -- --check`
Build-time benchmarking is intentionally deferred until the crate-split
series is ready for a controlled `release-profiling` comparison.
Design at: #13200
## What changed
Adds a max-age admission policy for the file download cache:
- New configs `ZO_DISK_CACHE_MAX_AGE_DAYS` and
`ZO_MEMORY_CACHE_MAX_AGE_DAYS` (default `0` = disabled, current behavior
unchanged).
- `queue_download()` now skips files whose `max_ts` is older than the
configured max age — they never enter the download queue; queries read
them directly from object storage (the existing cache-miss fallback).
- The `cache_latest_files` gRPC event handler applies the same filter
when collecting files, covering the batch `download_from_node` path that
writes to the cache via `file_data::set` without going through the queue
(e.g. merged files produced by compaction of old partitions).
- The check is exposed as `file_downloader::exceeds_cache_max_age(ts,
cache_type)`; `ts <= 0` (unknown timestamp) is never skipped.
## Why
`time_lru` has an eviction policy but no admission policy. With a cache
sized for the recent hot window (e.g. 3 days), a query on old data (e.g.
30 days ago):
1. Triggers gc **before** insert: the first old-file insert evicts
`max(release_size, data_size * 100)` — by default ~10% of the cache —
from the oldest hours of the *hot* window.
2. Once inserted, the old files themselves become the oldest `time_lru`
bucket and are evicted first, often within the same query — S3 download
→ disk write → immediate delete, pure wasted work with zero retention
for repeat queries.
With a max age matching the cache window, deep-history queries no longer
disturb the hot cache or waste S3 bandwidth and disk IO.
## Coverage
All query-side download paths funnel through `queue_download()` with the
file's real `max_ts` and are covered: search (parquet), tantivy index
(`.ttv`), promql/metrics, and file_list dump. Deliberately **not**
gated: compactor working downloads (`cache_remote_files` and
merge-output writes — required reads/writes, not speculative caching)
and the results/aggregations/metrics_results caches (separate concern).
## Tests
Unit tests for the age check (disabled, unknown ts, recent file, old
file); `cargo test -p openobserve-core --lib file_downloader` passes.
## Summary
Second step of the crate split (after #13168 which extracted
`openobserve-core`): move `src/handler` and `src/router` into a new
`openobserve-api` crate at `src/api`, so the HTTP/gRPC transport
frontend (routing, auth, extractors, OpenAPI generation) builds as its
own compilation unit.
All 155 moved source files are byte-identical moves (`git mv` only, zero
content changes). The real changes are limited to manifests and
scaffolding:
- **`src/api/src/lib.rs`** re-exports `openobserve_core::{common, job,
service}` (plus `cipher` under `enterprise`) at the crate root, so the
existing `crate::common::...` / `crate::service::...` paths inside the
moved files keep resolving without touching them.
- **`src/lib.rs`** re-exports `openobserve_api::{handler, router}`, so
external paths like `openobserve::handler::...` used by `main.rs` and
`tests/integration_test.rs` are unchanged.
- **Feature forwarding**: `enterprise` / `cloud` / `vectorscan` /
`vortex` / `profiling` on the root crate now forward to
`openobserve-api` as well.
- **`src/enterprise-stubs` → `src/enterprise`**: directory rename, plus
a new `o2_ratelimit` dependency-resolution stub (the api crate depends
on it under the `enterprise` feature; there was no stub for it before).
- `#![feature(variant_count)]` moves with its only use site
(`models/dashboards.rs`) into the api crate.
- **CI**: added `deny.toml` license exceptions for the two new AGPL-3.0
workspace crates (`openobserve-api`, `o2_ratelimit`); updated the
`ui.rs` path patched by the playwright workflows.
`src/api/Cargo.toml` is shared verbatim between OSS and enterprise
builds — only the root manifest is replaced by the enterprise overlay;
the `o2_*` deps resolve via `workspace = true` to either the stubs (OSS)
or the real crates (enterprise).
Companion PR: openobserve/o2-enterprise#2168
## Testing
- `cargo check --workspace` passes (OSS).
- Verified all moved files are 0-diff renames (`git diff -M90% --stat`).
- Verified every `crate::` reference in the moved code resolves through
the re-exports (`service`/`common`/`handler`/`job`/`router`/`cipher`),
with the single `crate::cipher` use correctly gated behind `enterprise`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Summary
Extract the application core (`src/common`, `src/service`, `src/job`,
`src/cipher`) out of the root binary crate into a new workspace crate
`src/core` (package `openobserve-core`). The root crate keeps the binary
entrypoints and edges: `cli`, `handler`, `router`, `migration`, and
`super_cluster_queue`.
This builds on #13162 (remove service → handler dependencies), which
made the extraction possible.
> Naming note: the directory is `src/core` but the package stays
`openobserve-core` — a Cargo dependency literally named `core` shadows
Rust's built-in `core` crate and breaks the `::core::` paths emitted by
derive macros (thiserror, serde, ...).
## What moved
- `src/common`, `src/service`, `src/job`, `src/cipher` →
`src/core/src/...` (467 files moved; 385 of them byte-identical, the
rest only needed `pub(crate)` → `pub` widening and clippy fixes required
by the new crate boundary)
- `ua_regex/regexes.yaml` → `src/core/ua_regex/` (same content, consumer
moved with it)
- `src/handler/grpc/request/search/mod.rs` →
`src/core/src/service/search/grpc_server.rs` (byte-identical; old path
kept via `pub use ... as search`)
- `impl From<MetricsQueryRequest> for cluster_rpc::MetricsQueryRequest`
moved from `handler/grpc/mod.rs` into `service/promql` (orphan rule;
field-for-field identical)
- The `impl From<XError> for Response` blocks scattered across HTTP
handler files (alerts, dashboards, reports, folders, pipeline,
eval_jobs, providers, ratelimit) are consolidated into
`src/core/src/service/http.rs` with unchanged status-code mappings
## Compatibility
`src/lib.rs` re-exports the moved modules (`pub use
openobserve_core::{common, job, service}`, and `cipher` under the
`enterprise` feature), so all existing `crate::service::...` /
`openobserve::common::...` paths keep working — handlers, tests, and
downstream code need no changes.
## Enterprise dependency resolution
`openobserve-core` optionally depends on `o2_enterprise` / `o2_dex` /
`o2_openfga` / `vortex` behind features. So that the OSS repo resolves
on its own, workspace deps point at empty stub packages under
`src/enterprise-stubs/` (never compiled into default builds). Enterprise
builds replace the workspace manifest with `Cargo.toml.openobserve`,
which points these at the real crates.
⚠️ Must be merged together with openobserve/o2-enterprise#2162 (same
branch name) — merging either PR alone breaks the enterprise build.
## Feature forwarding
Root `enterprise` / `cloud` / `vectorscan` / `vortex` features forward
to the corresponding `openobserve-core` features; the core crate
forwards to `config` / `infra` / `o2_enterprise` exactly as the root
crate did before.
## Unit-test fix
The `service::sourcemaps` tests relied on the workspace root being the
test cwd (shared zip fixture via a relative path, sqlite db dir/tables
created by `init-db` from `coverage.sh`). Unit tests of a sub-crate run
with the crate dir as cwd, so they now read the fixture via
`CARGO_MANIFEST_DIR` and create the db dir + the tables they touch
themselves; `src/core/data` is cleaned between the postgres and sqlite
CI passes and gitignored, mirroring `src/infra/data`.
## Cleanup
The extracted crate's manifest is pruned to what core actually uses (45
dependency entries copied from the root crate plus a dead `profiling`
feature removed), and the stale `roaring` dependency (unused since the
tantivy result cache rewrite in #7890) is dropped from the root
manifest. All new files carry the standard AGPL header.
## Testing
- `cargo check --workspace --all-targets`, `cargo clippy -D warnings`,
and `cargo fmt --check` pass
- `service::sourcemaps` unit tests pass under `--test-threads=1` (the CI
setting; they share one db group and are not parallel-safe, same as
before)
- No logic changes anywhere: the diff is moves + visibility widening +
clippy appeasement; error-to-HTTP-response mappings verified
variant-by-variant
- e2e label kept so Playwright runs
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
related to https://github.com/openobserve/o2-enterprise/issues/1860
- [x] limit parallel by the target partitions
## Summary
This PR improves Flight `do_get` response throughput by executing and
encoding output partitions in parallel instead of funneling everything
through a single sequential stream/encoder path.
## Changes
- Switches Flight `do_get` from `execute_stream` to
`execute_stream_partitioned`.
- Adds per-partition encoder tasks that coalesce RecordBatches, encode
Arrow IPC, and apply ZSTD compression independently.
- Splits Flight encoding into:
- `ChunkCoalescer` for 8192-row batch coalescing.
- `FlightDataEncoder` for IPC/ZSTD encoding of completed chunks.
- Adds `PartitionEncoderStream` to handle per-partition batch coalescing
and encoding.
- Preserves custom Flight messages behavior:
- early-emitted messages are sent before the first data message;
- stats/metrics/peak-memory messages are sent after data completes.
- Uses a fresh dictionary tracker per encoded chunk so
dictionary-encoded columns remain decodable when chunks are encoded
independently.
- Cleans up stream lifecycle handling by aborting partition tasks before
clearing session resources.
## Performance
100M records
single node optimzier disable(to simulate the distribute query)
### Q1
```sql
SELECT
trace_id,
count(*) AS cnt,
max(_timestamp) AS ts
FROM
"k8s_logs"
GROUP BY
trace_id ORDER BY cnt DESC LIMIT 100
```
12M trace_id unique value
12M records send from follower to leader
main branch: 6s
this branch: 3s
### Q2
```sql
SELECT count(distinct request_id) FROM "k8s_logs"
```
100M request_id unique value
100M records send from follower to leader
main branch: 30s
this branch: 11s
### Q3
```sql
SELECT
kubernetes_namespace_name,
COUNT(DISTINCT trace_id) AS cnt
FROM
k8s_logs
GROUP BY
kubernetes_namespace_name ORDER BY cnt DESC
```
10 kubernetes_namespace_name unique value, 12M trace_id unique value
63M records send from follower to leader
main branch: 21s
this branch: 8.3s
### Q4
```sql
SELECT
host,
COUNT(DISTINCT trace_id) AS cnt
FROM
k8s_logs
GROUP BY
host ORDER BY cnt DESC
```
2160 host unique value, 12M trace_id unique value
99.57M records send from follower to leader
main branch: 56s
this branch: 20s
- [x] ~add sort when to many groups, we can do this, but for now do not
do this~
- [x] remove the unsafe for memory pool
- [x] add approx_distinct improve, already fork, will add in next pr
- [x] fork vortex to add cast fix, and export
`.exclude_schemes([IntDictScheme.id()])`
- [x] ~enable_physical_uncorrelated_scalar_subquery test~
Design at: #12620Closes#12620Fixes#12564
## What
Rewrites both tantivy histogram optimize-rule collectors following the
same approach as the recent `TopNCollector` /
`SingleSegmentDocIdCollector` rewrites, and adds timezone support to
both rules.
- `MultiHistogramCollector` (replaces the aggregation framework for
`SimpleMultiHistogram`)
- `SimpleHistogramCollector` (replaces
`tantivy::collector::HistogramCollector` for `SimpleHistogram`)
### Timezone support (#12564)
`histogram(ts, interval, tz)` rewrites to `date_bin(interval,
to_timestamp_micros(_timestamp + offset), origin)`; the optimizer's
`get_column_name` hit the `BinaryExpr` and silently skipped both rules.
Now:
- the visitors extract the fixed offset from that shape
(`get_timestamp_offset`), so the rules fire with a timezone
- `IndexOptimizeMode::SimpleHistogram` / `SimpleMultiHistogram` carry a
`ts_offset` field (no proto change — histogram modes never cross gRPC,
they're re-derived per node)
- bucket edges are computed in shifted wall-clock space and the
collectors bucket on `_timestamp + ts_offset`, so emitted keys match
`date_bin` exactly when merging with non-tantivy partials
## Testing
- New collector unit tests: dense vs sparse equality, ts_offset
shifting, bounds filtering, missing column / missing breakdown docs,
per-bucket truncation with tie-breaks
- Visitor tests now include real-DataFusion-plan timezone cases
(`'+08:00'` simple, `'-05:30'` multi) asserting the extracted shifted
bounds and offset
- `cargo test -p config` (1936 passed), plus `index_optimizer` /
`search::tantivy` / `tantivy_optimize_exec` suites in the main crate;
clippy clean
- Low cardinality: fast `20%` than main
```
histogram(_timestamp, '1h') AS ts, count(*) FROM default GROUP BY ts ORDER BY ts ASC
```
- Low cardinality: fast `30%` than main
```
histgoram(_timestamp, '1h') AS ts, breakdown, count(*) FROM default GROUP BY ts, breakdown ORDER BY ts ASC
```
- High cardinality: faster `5x` than main
```
histgoram(_timestamp, '1h') AS ts, breakdown, count(*) FROM default GROUP BY ts, breakdown ORDER BY ts ASC
```
Close#12345
## Problem
OTLP histogram metrics sent over HTTP/JSON (e.g. otel-collector with
`encoding: json`) created streams but stored 0 events, with no
`_count`/`_sum`/`_bucket` streams. Int-valued counters/gauges were
silently recorded as 0. The protobuf path was unaffected, which is why
this only surfaced for collectors explicitly configured with JSON
encoding.
## Root cause
`otlp_json` deserializes the body directly into opentelemetry-proto's
serde-derived types, which only partially implement the proto3 JSON
mapping (64-bit integers encoded as strings, enums by name, zero/empty
fields omitted). `Metric.data` is a flattened `Option` oneof, and serde
converts any inner deserialization error of a flattened `Option` into
`None` instead of failing — so the whole metric silently became `data:
None`, while the stream itself had already been created by the schema
setup. Same mechanism on `NumberDataPoint.value` turned string `asInt`
values into 0.
## Changes
- Upgrade opentelemetry stack 0.31 → 0.32, which fixes string-encoded
u64 handling for plain histograms and summaries upstream.
- Add `otlp_json_compat` normalizer for the gaps still present in 0.32
(each verified empirically, with unit tests):
- string `asInt` values on data points and exemplars
- `aggregationTemporality` as enum name
- `Exemplar`'s value oneof is not flattened in 0.32
- missing `serde(default)` on
`ExponentialHistogramDataPoint`/`Buckets`/`ValueAtQuantile`/`Exemplar`,
which rejected omitted zero/empty fields
- Count and log metrics whose data oneof failed to decode
(`partial_success.rejected_data_points`) instead of silently skipping
them, and stop overwriting specific partial-success error messages with
the generic retention message.
- Adapt to OTLP 0.32 API additions (`KeyValue.key_strindex`,
`AnyValue::StringValueStrindex`).
- Upgrade workspace reqwest 0.12 → 0.13, required because
opentelemetry-http 0.32 only implements `HttpClient` for reqwest 0.13
(`rustls` feature replaces `rustls-tls-native-roots` via the platform
verifier; `json`/`query`/`form` are now explicit features). The
`segment` crate is pinned to reqwest 0.12, so `TELEMETRY_CLIENT` moved
into `telemetry.rs` as a plain `reqwest::Client` that posts the Track
message itself, replicating `segment::HttpClient::send`.
- Remove the unused pyroscope profiling integration (optional feature,
dependencies, agent setup, `ZO_PROF_PYROSCOPE_*` config).
## Verification
- 10 unit tests in `otlp_json_compat` covering the full payload matrix
(collector-style string integers, enum names, omitted zero fields,
exemplars with and without trace context, already-normalized payloads,
malformed input).
- End-to-end: posted the issue's collector-style JSON payload to
`/api/default/v1/metrics` — `_bucket` (3 events, correct cumulative
counts with exemplars attached), `_count`, `_sum` populated for
histogram and exponential histogram, summary quantiles stored, `asInt`
counter stored as 42 instead of 0.
- `cargo clippy -- -D warnings` clean; lib test failures are
pre-existing local-environment issues that reproduce identically on
main.
Build frontend and Rust binary in parallel using /tmp/dist trick
Remove VITE_COVERAGE/COVERAGE env vars and coverage instrumentation
Remove generate_coverage_report job and all nyc upload/report steps
Rename artifact debug-binary → release-ci-binary, add frontend-dist
artifact
Update playwright_summary needs to drop generate_coverage_report
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Huaijin
Ashish Kolhe
Hengfei Yang
Harsh Mahajan
Don Kendall
ktx-kirtan
漂流