Design at: #13465
PR 2 of the plan in #13465 (finding 1: the settings cache O(S) deep
clone per mutation).
## Background
The stream settings cache is a locked map (`STREAM_SETTINGS`) plus an
ArcSwap read snapshot (`STREAM_SETTINGS_ATOMIC`) introduced in
`405794db08` for lock-free ingestion reads. That architecture is kept —
the read/write ratio justifies it. The problem was the constant factor:
- every mutation republished the snapshot by **deep-cloning every cached
`StreamSettings`** (vectors, UDS field lists, maps). The snapshot is
republished on every schema change event cluster-wide (any new stream or
field, on every node) plus every cache-miss backfill — at 100k+ streams
this is a full deep copy of the map at minute-level frequency;
- every point read (`get_settings`, called on each ingestion request)
deep-cloned the returned `StreamSettings`, including UDS lists that can
hold hundreds of strings;
- the two maps each held a complete deep copy of all settings.
## Changes
**Arc-ified values.** Both maps now store `Arc<StreamSettings>`.
Snapshot republish is a shallow clone (String keys + refcount bumps),
point reads return an `Arc` clone, and the maps share one set of values,
roughly halving the cache's memory.
**Centralized mutation.** The map statics are private; all writes go
through `put_stream_settings` / `put_stream_settings_batch` /
`remove_stream_settings`, which republish the snapshot under the write
lock so snapshots cannot be published out of order. Startup load
collects settings and publishes once via the batch API. Callers can no
longer accidentally rebuild the snapshot — the pattern that caused the
original quadratic startup. Lock-free reads stay available via
`get_stream_settings_atomic`; the status API uses a new
`get_stream_settings_stats()` accessor.
**API ripple, kept minimal.** `get_settings()` now returns
`Option<Arc<StreamSettings>>`. Field access and `unwrap_or_default()`
call sites compile unchanged; the few paths that edit a local copy
before persisting (`update_stream_settings`, LLM/gen-ai settings,
dashboard distinct fields) do an explicit `(*s).clone()`. The
`get_stream_setting_*` helpers are generic over
`Borrow<StreamSettings>`, so both `&Option<StreamSettings>` and
`&Option<Arc<StreamSettings>>` callers work — including the downstream
enterprise call sites, which were checked and need no changes.
**`SchemaCache::fields_map` Arc-ified** — cloning a cached schema entry
no longer deep-copies the field-name index, which holds thousands of
entries for wide streams.
## Measurements (criterion bench used during development, not kept
in-tree)
| operation | result |
|---|---|
| bulk load 10k / 50k / 100k streams | 0.80ms / 4.4ms / 8.7ms — linear,
~87ns per stream |
| point write against 100k-entry cache | 3.5ms (shallow snapshot
republish) |
| point read against 100k-entry cache | 7.9ns (lock-free) |
For comparison, the field report in #13465 measured ~27ms per stream
insert at 58k cached streams under the old per-mutation deep clone;
reads previously deep-cloned the settings value on every call.
## Testing
- new unit test for the mutation API (put/batch/remove, snapshot sharing
via `Arc::ptr_eq`, republish keeps other keys)
- full test suites pass for all touched crates (infra, db, schema,
stream, core, api-search, api-management, search_service, compaction,
enrichment-data); the only failures are pre-existing local-environment
ones also failing on main (sqlite-path retention tests, one
env-dependent index test, one enterprise doctest)
- `cargo clippy --workspace --all-targets`: no findings in touched files
Finding 2 (grouped stream enumeration) landed in #13536; this branch has
been rebased onto main (the #13531 overlap is resolved).
## Summary
- decouple stream cleanup from enrichment-data and remove the transitive
compaction/DataFusion dependency from stream
- extract sourcemaps and organization storage services from core into
dedicated crates
- extract GitHub built-in regex access and model pricing into
openobserve-builtins
- rename pipeline and organization-storage persistence modules from
store to db
- remove now-unused direct dependencies from core
- synchronize the enterprise workspace overlay in
openobserve/o2-enterprise#2266
## Validation
OSS:
- cargo check -p openobserve-builtins -p openobserve-core -p
openobserve-jobs -p openobserve-api
- cargo test -p openobserve-builtins --lib
- cargo test -p openobserve-builtins --doc
- cargo clippy -p openobserve-builtins -- -D warnings
- cargo check -p openobserve-org-storage --features enterprise
Enterprise overlay:
- copied ../o2-enterprise/Cargo.toml.openobserve over Cargo.toml
- cargo fmt --all -- --check
- cargo check --all-targets --no-default-features --features enterprise
- cargo clippy --all-targets --no-default-features --features enterprise
-- -D warnings
The default enterprise + vectorscan build was attempted and reached
vectorscan-rs-sys, but the validation host does not have cmake
installed. The enterprise-only build and clippy run completed
successfully.
## Summary
This PR begins decomposing `openobserve-core` into focused service
crates while preserving the existing application behavior and dependency
direction.
It extracts six independently owned areas:
- `ingestion-common`: shared ingestion contracts and response types
- `schema`: schema inference, validation, evolution, and persistence
helpers
- `compaction`: merge, retention, and compaction services
- `enrichment-data`: enrichment-table lifecycle and runtime support
- `promql-service`: PromQL service orchestration
- `stream`: stream metadata, settings, retention, and lifecycle
operations
## What changed
- registered the extracted crates in the workspace and propagated
enterprise/cloud/vectorscan/vortex features
- moved implementations out of Core and updated API, Jobs, query, and
queue consumers to use the new crates
- removed Core facade aliases for extracted crates and converted
remaining callers to direct crate paths
- retained the small cross-service stream-delete coordinator in Core for
now because it still coordinates Pipeline and Alert cleanup
- restored required root/Core dependencies after validating the full
enterprise build
- added the missing `anyhow` dependency for enterprise-only
`search_service::search_jobs`
- updated the lockfile for the new crate dependency graph
## Dependency direction
The extracted crates do not depend on `openobserve-core`. Core remains a
temporary upper-level consumer while the remaining domain services are
split incrementally.
```text
ingestion-common -> common/config
schema -> ingestion-common/db/infra
stream -> schema/enrichment-data/db/infra
compaction -> schema/search/search-service/db/infra
openobserve-core -> ingestion-common/schema/stream/compaction/...
api/jobs -> openobserve-core + extracted services
```
## Enterprise compatibility
The private OpenObserve manifest was updated in
openobserve/o2-enterprise#2257 to register the extracted workspace
members and dependencies.
Related Enterprise PR:
https://github.com/openobserve/o2-enterprise/pull/2257
## Validation
OSS manifest:
- `cargo fmt --all -- --check`
- `cargo clippy -- -D warnings`
Using `o2-enterprise/Cargo.toml.openobserve` as the OpenObserve root
manifest:
- `cargo check --no-default-features --features enterprise`
- `cargo check --no-default-features --features cloud`
- `cargo clippy --workspace --no-default-features --features enterprise
-- -D warnings`
- `cargo clippy --workspace --no-default-features --features cloud -- -D
warnings`
Enterprise repository:
- `cargo fmt --all -- --check`
- `cargo clippy -- -A clippy::question-mark -D warnings`
All checks above pass in the development profile.
## Summary
- remove compatibility re-exports introduced during the recent crate
refactors
- migrate callers to the crates that own each API
- delete five pure facade modules and add the required direct crate
dependencies
- keep OSS and enterprise manifests aligned
## Validation
- cargo fmt --all -- --check
- cargo clippy --workspace -- -D warnings
- cargo test --locked --workspace --no-run
- enterprise overlay: cargo test --offline --workspace
--no-default-features --features enterprise --no-run
- enterprise overlay: cargo clippy --offline --workspace
--no-default-features --features enterprise -- -D warnings
- git diff --check
Companion enterprise PR uses the same branch name:
codex/remove-refactor-reexports.
## TODO / Follow-ups
- [ ] **Re-establish the pipeline write-guardrail lost with
`db::pipeline`.** That module was not a pure facade: it re-exported only
the six read-only accessors from `infra::pipeline` and its doc comment
recorded the invariant that *mutations must go through
`service::pipeline::store`* (which keeps the `ExecutablePipeline` caches
and the coordinator/super-cluster watch events in sync). Deleting it
points callers straight at `infra::pipeline`, which also exposes
`put`/`delete`/`drop_table` with no guardrail. Decide the layering and
enforce it:
- **Reads** may hit `infra` directly (raw persistence, no side effects).
- **Writes** must go through the wrapper that persists *and* emits the
coordinator/super-cluster event + refreshes the in-memory cache. That
wrapper is `db::<entity>::{set,delete}` for most entities (e.g.
`db::alerts::alert::set` →
`infra::coordinator::alerts::emit_put_event`), but for pipeline it is
`core::pipeline::store` (not `db`), because the
`STREAM_EXECUTABLE_PIPELINES` cache it maintains is a core-level
construct the `db` crate cannot build. The super-cluster receiver
(`super_cluster_queue::pipelines`) is the one sanctioned caller of
`infra::pipeline::{put,delete}` directly, and it manually pairs each
with `emit_*_event` to avoid a re-broadcast loop.
- Net invariant to protect: never call `infra::pipeline::{put,delete}`
without emitting the coordinator event — i.e. nowhere outside
`core::pipeline::store` and the super-cluster handler. A bare `infra`
write leaves in-memory caches stale until restart (see the warning
comment still live at `src/core/src/org_cleanup.rs:484`).
- [ ] Consider whether the pipeline write-wrapper living in
`core::pipeline::store` while every other entity's lives in
`db::<entity>` is worth unifying, or at least documenting as an
intentional exception.
## Summary
- extract persistence, cache, watcher data access, and DB service
modules from openobserve-core into a standalone workspace db crate
- keep application orchestration that depends on search, organization
provisioning, and other Core services behind the existing Core db facade
- preserve existing call paths through re-exports while making the
dependency direction openobserve-core -> db
- move HTTP error mapping back to the Core/API boundary so db has no
direct axum dependency
- avoid provider/provisioner callback traits and function-pointer hooks
## Validation
- cargo fmt --all
- cargo check --all-targets
- cargo check -p db --all-targets
- Enterprise workspace: cargo check --all-targets --offline
- Enterprise workspace: cargo check -p db --all-targets --features
enterprise --offline
Companion Enterprise manifest PR: openobserve/o2-enterprise#2230.
## Summary
- add a dedicated promql workspace crate for parsing and evaluation
- keep cluster routing, cache, WAL, file-list, and request orchestration
in openobserve-core
- preserve existing service::promql call sites through a compatibility
facade; the facade re-exports an explicit item list (not a glob), and
the promql search orchestration in core depends on the promql crate
directly
- inject enterprise query cancellation through TableProvider instead of
coupling the evaluator to core search state
- replace the core regex-UDF dependency with DataFusion regexp_like
expressions
## Companion PR
- openobserve/o2-enterprise#2198 updates the enterprise workspace
override for the new promql crate
## Validation
- cargo fmt --all -- --check
- cargo check --workspace
- cargo clippy -p promql --all-targets -- -D warnings
- cargo clippy -p openobserve-core
- cargo test -p promql: 363 passed
- verified the promql dependency tree has no openobserve-core or search
dependency
- cargo check -p o2_enterprise --lib in the paired private workspace
## Notes
The evaluator moves atomically in this PR. The existing
service::promql::search subtree remains in core because it owns
OpenObserve-specific orchestration rather than PromQL evaluation.
The two web/ file changes only update source-path references in comments
that pointed at the moved Rust files (src/service/promql/... ->
src/promql/src/...).
# Add DST-Aware Timezone Handling for Alert Scheduling
## Summary
Alerts previously used static timezone offsets that became outdated
during DST transitions, causing alerts to fire at incorrect times (e.g.,
an alert configured for "9 AM Pacific" would fire at 10 AM after DST
change).
## Solution
- Added `chrono-tz` dependency for IANA timezone database support
- Implemented DST-aware offset calculation at scheduling time using
timezone strings (e.g., `America/Los_Angeles`)
- Updated `get_next_trigger_time_non_aligned()` and `align_time()` to
dynamically calculate offsets based on the scheduled timestamp
- Maintains backward compatibility: alerts without timezone strings
continue using fixed offsets
- Ensure memtable isolation to prevent cross-test data contamination
- Fix dashboard API version compatibility (v5 -> version-agnostic dashboard_id)
- Fix indentation bug causing function/dashboard/stream deletions to run inside alert loop
- Add 3-second wait after dashboard deletion for propagation
### **User description**
In a multi-region super cluster setup, when a node starts up and caches
enrichment tables, it generally queries all regions for enrichment table
data. However, if user is sure that enrichment table is stored in one
primary region, then querying all regions is inefficient and might even
fail if other region queriers return error for some reason. Which can
eventually cause node start failure.
### Solution
This commit adds a new `apply_primary_region_if_specified` boolean
parameter throughout the enrichment table data fetching flow to control
whether to use only the primary region when fetching data.
### Key Changes
1. **New Parameter Added** (`enrichment_table.rs:40`, `mod.rs:190`):
- Added `apply_primary_region_if_specified` boolean parameter to:
- `get_enrichment_table_data()`
- `get_enrichment_table()`
- `get_enrichment_table_json()`
2. **Primary Region Logic** (`enrichment_table.rs:52-68`):
- When `apply_primary_region_if_specified` is true and enterprise
features are enabled:
- Checks if super cluster is enabled
- Reads the `enrichment_table_get_region` configuration
- If a primary region is specified, uses only that region for the search
request
- Otherwise, uses empty regions array (queries all regions)
3. **Cache Initialization** (`schema.rs:588-591`):
- When caching enrichment tables during node startup
(`cache_enrichment_tables()`), passes `true` for the parameter
- This ensures node startup only queries the primary region for
enrichment data
4. **Runtime Watch Operations** (`enrichment_table.rs:310-328`):
- Watch operations pass `false` for the parameter
- This maintains existing behavior for runtime updates
5. **Code Cleanup**:
- Removed unused `get()` function (lines 105-146 deleted)
- Simplified logging by removing redundant empty/non-empty checks
- Added better debug logging showing number of records fetched
### Impact
- **Performance**: Node startup is faster as it only queries the
designated primary region for enrichment table data
- **Correctness**: Ensures enrichment tables are fetched from the
authoritative source (primary region) during initialization
- **Flexibility**: Runtime operations can still query all regions if
needed
### Configuration Required
This feature requires enterprise edition with super cluster enabled and
the `enrichment_table_get_region` configuration set to specify which
region should be considered the primary source for enrichment table
data.
___
### **PR Type**
Enhancement
___
### **Description**
- Add primary-region-aware enrichment fetch
- Thread control flag across fetch APIs
- Use primary region on node startup
- Preserve all-regions for watch updates
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["schema::cache_enrichment_tables()"] -- "get_enrichment_table(..., true)" --> B["enrichment::get_enrichment_table_inner(..., true)"]
B -- "get_enrichment_table_data(..., true)" --> C["db::enrichment_table::search_inner(regions=primary)"]
D["db::watch()"] -- "get_enrichment_table(..., false)" --> B
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>enrichment_table.rs</strong><dd><code>Region-aware
enrichment data retrieval and watch updates</code> </dd></summary>
<hr>
src/service/db/enrichment_table.rs
<ul><li>Add <code>_apply_primary_region_if_specified</code> to
<code>get_enrichment_table_data</code><br> <li> Determine
<code>regions</code> based on enterprise config and flag<br> <li> Call
<code>search_inner</code> with computed regions and enable consistency
flag<br> <li> Update watch flow to call enrichment getter with
<code>false</code></ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8878/files#diff-6ce27ddf8a0d8b4e790bda90d088ff692a5da31a8d330beea8a9dd843294572b">+35/-6</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>schema.rs</strong><dd><code>Startup caching uses
primary region when specified</code>
</dd></summary>
<hr>
src/service/db/schema.rs
<ul><li>Cache startup uses <code>get_enrichment_table(...,
true)</code><br> <li> Add comment clarifying primary-region-only
behavior</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8878/files#diff-4696a44abde2df4ac48fed716f930bfdcc01759bdfcfd8d802821c6dd857bff9">+4/-1</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>mod.rs</strong><dd><code>Plumb primary-region flag
through enrichment APIs</code>
</dd></summary>
<hr>
src/service/enrichment/mod.rs
<ul><li>Plumb <code>apply_primary_region_if_specified</code> through
public APIs<br> <li> Document flag semantics and usage caveats<br> <li>
Pass flag to DB layer when fetching remote data</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8878/files#diff-0ea1823eb8a3a9845cde8bb2264ccdb749fad5be6d9acda8b17d76b2dbfdcc9f">+16/-3</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
Integrate Model Context Protocol server endpoints at `/api/{org_id}/mcp`
for both enterprise and non-enterprise builds.
Changes:
- Add POST and GET handlers for MCP requests with streaming support
- Extract and pass through Authorization header for API authentication
- Add non-enterprise stub endpoints that return "not available" error
- Update router to register MCP endpoints conditionally
Endpoints:
- POST `/api/{org_id}/mcp`: Single MCP request/response
- GET `/api/{org_id}/mcp`: Streaming MCP requests
Huaijin
Shrinath RaoandAshish Kolhe
Hengfei Yang
Omkar Kesarkhane
Prabhat Sharma
yujinqiu
Loakesh IndiranandHengfei Yang
ByteBakerandHengfei Yang
