The query half of the TSID-major layout: #13882 wrote `tsid-major-v3-*`
files ordered by `(__hash__, _timestamp)`; this PR adds the `.midx`
series index next to them and makes PromQL prune through it. Supersedes
the remaining scope of #13839.
## Write side
- Every `tsid-major-v3-*` file written by the hour-end merge gets a
`.midx` series index (Arrow IPC + zstd), stored like the Tantivy index
under a derived root distinct from the data tree:
`files/{org}/metrics/{stream}/{date}/{hour}/tsid-major-v3-{id}.parquet`
→ `files/{org}/midx/{stream}/{date}/{hour}/tsid-major-v3-{id}.midx`
(Tantivy: `files/{org}/index/{stream}_{type}/…/{id}.ttv`): **one row per
contiguous TSID run** — its `row_count: u32` plus the label columns of
the run's first row (source types preserved). Runs tile the data file,
so each run's starting row is the prefix sum of the preceding counts and
is not stored; `__hash__` is excluded (matchers never reference it). The
whole index is one ZSTD-compressed IPC batch. Runs crossing an input
RecordBatch boundary appear as two adjacent entries; query-time range
coalescing makes that representation equivalent without buffering
samples. File and row-group boundaries stay independent of TSID
boundaries.
- `TsidSeriesIndexWriter` lives inside
`search/datafusion/merge/tsid_major.rs` and collects run entries
alongside the Parquet writer (one buffered row per run), encoding and
compressing once at finish. `MergedFile` carries `series_index:
Option<Vec<u8>>`; the compactor uploads it to the `midx/` tree
(compliance mode included).
- `.midx` is intentionally **not** tracked in `file_list`; its path is
derived from the data file's key
(`MetricsFileLayout::series_index_path`) and it is deleted together with
the parent in the delayed-deletion pass (`deleted.rs`); like the
`index/` tree, the `midx/` day directories are covered by retention's
local-disk deletion and the gc tool.
## Query side (PromQL)
- `promql_service::…::tsid_series_index::search` runs where Tantivy used
to, before the metrics table is registered: for each TSID-major file it
loads the `.midx` (concurrent, `2×target_partitions ≤ 64`), projects the
row-range columns plus the matched labels **by name against each series
index's own schema** (label sets / column orders differ between files),
builds the matcher filter **per series index** with
`physical_expr::create_physical_expr` — the exact same predicates as
`apply_matchers` (`matcher_predicates`) — evaluates it over the run
rows, coalesces adjacent/overlapping ranges and attaches them as
`FileSelection::RowRanges` (converted into a Parquet access plan by the
generic helper from #13861, no per-row bitmap).
- **Applied per file**: files of any other layout in the same query
(still-open hours, legacy data, late arrivals) are left in place and
scanned in full. A matched label missing from a series index
over-selects that file; a query whose matchers cannot be evaluated at
all selects the whole file; any series index failure logs and falls back
to a full scan. The final PromQL matchers are always applied, so results
are exact in every case.
- Evaluated selections are cached (LRU, 256 MiB) keyed by account +
series index path + the full matcher set.
- The Tantivy path in the PromQL storage scan is removed
(`convert_matchers_to_index_condition`, index-field plumbing,
`is_add_filter_back` bookkeeping). SQL search is untouched.
## Benchmark (from the #13839 POC, same layout + series index)
478,144,896 rows, 7 CPUs, one-hour PromQL range, step 15s, result cache
off, release + mimalloc:
| Layout | Filter steady median | No-filter steady median | Data files |
Total storage |
|---|---:|---:|---:|---:|
| main, 2 GiB | 695.709 ms | 4.638 s | 88 | 4,925.81 MiB |
| TSID-major + `.midx` | 269.496 ms | 2.793 s | 80 | 1,294.61 MiB |
## Validation
- `cargo fmt --check`; `cargo check` / `clippy` on config, search,
compaction, promql-service, jobs, search_service (`--tests`) clean
- `config tsid_layout` 3 passed (incl. `series_index_path`); `search
tsid_major mode:: merge::` 10 passed (rotation test now also asserts the
series index, exact-run and coalescing series index tests); `compaction
merge deleted` 33 passed; `promql-service` 20 passed + 1 pre-existing
wall-clock flake (`cache::test_promql_cache_set_and_get`, fails
intermittently on main too)
- End-to-end on the #13839 branch (same code paths): hour-end merge
writes N `tsid-major-v3-*` + `.midx`; PromQL takes the series index path
(`selected 39 ranges across 4/4 files`), counts/sums match the synthetic
data, cold-cache identical after restart; a mixed hour (3 major + 1 late
`tsid-sorted`) reports `1 files without a series index left to
tantivy/full scan` with unchanged results; delayed deletion removes
`.parquet` **and** `.midx`
> ### ⚠️ This PR carries four changes — review alongside
openobserve/o2-enterprise#2420
>
> 1. **Super-cluster support (consumer half)** — `8ceb041c43 …
5885c57b3a`.
> 2. **Synthetics OSS migration** — `3a380087a3 … 2596300dbf`, nine
commits — one per build-order step, plus a ninth for fixes found by
running the binary. Moves synthetics out of `o2_enterprise` into
`src/synthetics` (`openobserve-synthetics`), leaving only the
private-VPC-agent path enterprise: `job_api::lease`, `agent/*`, private
locations and the agent-setup install flow. Paired with
openobserve/o2-enterprise#2420.
>
> 3. **Last-check columns read from the results stream** — `6af411e808`,
`e7a3f2c759`. Outside the region whose scheduler runs a check, the list
showed a stale status and a blank Last check / response time:
`last_check_status` replicates only on change, `last_triggered_at` is a
runtime column that does not replicate, and `last_response_ms` was never
populated anywhere. The `synthetics_results` stream already carries all
three and is federated, so the list API reads it instead of replicating
the columns — no queue traffic, no schema change. Best-effort: any miss
keeps the stored DB values, so search being unavailable cannot take the
list down. OSS-only; no enterprise counterpart.
>
> 4. **Check secrets are readable in every region** — `d47a6b2345` (+
o2-enterprise `da71d7ec`). The per-org DEK is minted by each region for
itself, so a check created outside the scheduler region replicated fine
and then silently never ran — `resolve` could not decrypt its
credentials (o2-enterprise#2451). Secrets now cross the queue **in
clear** and the applier re-encrypts them under its own DEK on arrival,
fixing the class at the replication boundary with no key to distribute
or keep in sync.
>
> ⚠️ **This puts customer secrets in plaintext on the inter-region bus**
— a deliberate, recorded trade-off (`o2oi_` ingest tokens and probe
tokens already travel it in clear). Securing the bus itself (TLS on the
gateway links, short retention on secret-carrying subjects) is the real
fix and is **not** in this PR.
>
> ⚠️ **Deploy consumers before producers.** The payload's bytes did not
change, only the meaning of the secret strings, so an un-upgraded
consumer handed the new payload would write customer passwords to disk
in clear. The plaintext-carrying messages therefore use new variant
names (old ones retired as `CreateEncrypted`/`UpdateEncrypted`), so an
old region fails serde and redelivers until upgraded instead of silently
persisting plaintext.
>
> An OSS build can now create, schedule, dispatch to Lambda, take
`resolve`/`ack` and report a check. 7 393 lines moved into the new
crate; 679 lines stay enterprise.
>
> **🛑 Deployment prerequisite (2 of 2).** `ZO_SYNTHETICS_ENCRYPTION_KEY`
must be set to the **same** base64 64-byte value in every region of a
super cluster. Without it synthetics falls back to the per-org DEK and
checks carrying secrets silently fail to run outside the scheduler
region. It is read once at startup and pinned — changing it later
requires a restart, and changing it at all orphans every secret already
encrypted under the old value.
>
> **🛑 Deployment prerequisite (1 of 2).** 14 `O2_SYNTHETICS_*` env vars
are hard-renamed to `ZO_SYNTHETICS_*` with **no fallback read**, and
`ZO_SYNTHETICS_ENABLED` defaults to `false`. A deployment still setting
the old names does not lose tuning — **synthetics stops running
entirely, and the first symptom is checks silently not firing.** Update
env vars *before* rolling the binary.
>
> #### Things a reviewer should look at first
>
> - **`super_cluster_queue` does not depend on
`openobserve-synthetics`** (`cargo tree` → 0), which is what keeps the
applier's loop-prevention guarantee a property of the crate graph rather
than a convention. `a7d5fdd969` adds the assertion that fires if that
ever changes.
> - **`job_upload` was skipping authorization in OSS builds**
(`4c3a0b8f0d`). Unreachable before, because no synthetics route was
registered in OSS; registering it would have made it an unauthenticated
artifact upload. `authorize_probe` is ungated and the bypass is gone.
> - **The dependency edge had to reverse in a single commit**
(`3247d4093f`). Cargo rejects an `o2_enterprise` ↔ OSS package cycle at
the manifest level — `optional` and feature gating do not help — so the
two edges could never coexist, and everything needed for the reversal
lands together. This is why the spec's steps 4 and 5 are one commit and
the config moved earlier than planned.
>
> #### Accepted OSS limitations, all documented in the commits
>
> - No notifications: `notify_check_result` ends in
`alerts::alert::dispatch_notification`, which is enterprise. An OSS
build runs the check and records the run; it has nowhere to send an
alert, exactly as for alerts themselves.
> - No per-resource RBAC on synthetics, and no OpenFGA ownership tuples
— both are enterprise capabilities across the platform.
> - Private locations are refused with a message rather than created.
> - Agent-served public `net-*` pools are unavailable, so an OSS user
needs a Lambda for every public pool (spec §1).
---
Consumer half of synthetics super-cluster support. **Depends on
openobserve/o2-enterprise#2420** (producer, message types, queue struct)
— and **this side must deploy first**: `MessageType` bytes 83–85 hit a
panicking `unreachable!()` in `From<u8>` on a consumer that does not
know them, which takes down the subscriber task rather than failing one
message.
Based on `fix/lease-claimed-by-2270` (#13785), not `main`, so the diff
is only this work.
## What this fixes
Synthetics writes are not querier routes, so a create lands on ingesters
in whichever region received the request and commits to that region's
Postgres. There is no synthetics topic on the super-cluster queue, so a
check created in a region that runs no scheduler is never seen by the
region that does: the UI shows it enabled, the scheduler has never heard
of it, and nothing reports the gap.
## Commits
| commit | what |
| --- | --- |
| `8ceb041c43` | orphan-detection query + metrics |
| `1c3e10da48` | apply replicated check config |
| `79e6abc052` | apply replicated locations and probe tokens |
| `16d4fa0fde` | render unknown live status instead of a false pending |
| `5885c57b3a` | share the job-cluster predicate |
## Design notes worth reviewing
**Config columns only, and the list is not restated.**
`synthetics_checks::update` rebuilds its `ActiveModel` from the row
already in this region, and `update_mutable_fields` writes only
user-editable columns — so `next_run_at` and the alerting counters keep
local values for free. Hand-listing them in the consumer would duplicate
a list that already exists and let the two drift. A test asserts every
runtime column comes back `Unchanged`.
**`put()` is deliberately unused.** Its insert branch builds from
`build_active_model`, which ends in `..Default::default()` and never
assigns `owner`, `tz_offset` or `next_run_at` — a replicated create
through it would drop the owner and fire a scheduled-later check
immediately.
**`create()` gains `use_given_id`**, matching `table::alerts::create`.
It generated its own uuid, which would have made every region invent a
different primary key for the same check.
**Consumers apply through `infra::table` only.** The publish lives one
layer up in the enterprise service, and `infra` cannot reach the
enterprise crate — that layering is what stops a region re-broadcasting
what it just applied. `source_cluster` does not cover this: it only
stops a region re-consuming its *own* message. Each consumer greps its
own source to keep this true.
**Runtime tables are never replicated** — `synthetics_jobs`,
`synthetics_runs`, `synthetics_agents`, and the locations
`down_notified_at` claim. Asserted at both producer and consumer.
## Known gaps, named not hidden
- If two regions mint a token (or location) with the same name for one
org before either message crosses, the UNIQUE constraint rejects the
arriving row. Local row wins, logged loudly. Erroring would redeliver to
dead-letter, which is worse.
- Pre-existing and untouched:
`write_triggers_stream`/`write_results_stream` in the enterprise reaper
post to `_meta` with the *org's* ingest token, which 401s, and neither
checks response status. Orphan detection fixes this in its own path
only.
- Pre-existing: `create` never sets `tz_offset`, so a new check gets DB
default 0 until its first edit. Replicates identically everywhere, so it
causes no divergence.
## Spec corrections
The spec (`docs/synthetics-lcl/2026-08-11-p1-3-super-cluster-spec.md`,
local notes) is wrong in five places, all found by reading the tree:
§1's invariant evidence (see the enterprise PR), §10's never-run guard,
§4's column list (and it lists six runtime columns — there are seven,
`alerting` is missing), §3's "private rows read-only when remote", and
§7's claim that non-scheduler regions render OFFLINE (they render
`pending`).
## Verification
infra 42, super_cluster_queue 32, jobs 21, web 111/111 vitest +
type-check + design lint, `cargo fmt --check` and clippy clean, full
`--features enterprise` build.
Pre-existing failures on this branch, unrelated and untouched:
`super_cluster_queue::schemas::tests::test_parse_key_{valid,metrics}`,
`infra::table::ratelimit::tests::test_add_batch_missing_rule_id`, and 8
web vitest failures in `buildPayload.spec.ts`/`payloadContract.spec.ts`.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
- enable tower-http zstd request decompression
- add a router middleware regression test using a real zstd-compressed
JSON payload
- update decompression documentation to list zstd support
## Root cause
The Axum router uses `RequestDecompressionLayer`, but the workspace did
not enable tower-http's `decompression-zstd` feature. Requests from
clients such as Vector configured with `compression: "zstd"` were
therefore rejected with `415 Unsupported Media Type` before reaching the
ingestion handler.
## User impact
Vector HTTP sinks can continue using zstd compression when sending
batches to OpenObserve ingestion endpoints.
## Validation
- `cargo fmt --all -- --check`
- `cargo test -p openobserve-api-http
handler::http::router::decompression::tests`
- `git diff --check`
Fixes#13726
Design at: #13702 / inline in PR description
Add the `zstd` feature to tonic and enable `accept_compressed(Zstd)` on
the OTLP logs, metrics and trace gRPC services in both `run_common` and
run_router`.
Closes#13702
## Out of scope
- Response compression on `send_compressed(Zstd)`
- OTLP/HTTP path (tower-http zstd features)
- snappy would be the better choice under CPU pressure, but tonic has no
snappy support.
## Summary
- add ZO_FEATURE_METRICS_PUSHDOWN_FILTER_ENABLED as an opt-in filter
pushdown switch for metrics queries
- group DataFusion and Tantivy feature configuration under the search
config struct
- move inverted-index result cache sizing, footer caches, skip
threshold, TopN group limit, and bloom footer cache settings from limit
to search
- preserve the moved cache settings' environment variable names and
defaults
- apply the metrics-specific setting to object storage, WAL, and Flight
query contexts
## Testing
- cargo fmt --all -- --check
- cargo check for config, infra, search, tantivy_utils, search_service,
and promql-service
- cargo test -p config every_env_config_default_parses --lib
- cargo test -p config inverted_index_config --lib
- cargo test -p search test_create_session_config --lib
- cargo clippy for all affected crates with -D warnings
## Summary
Two coupled alerting features:
**1. Notification Templates v2 ("content" templates)**
- New `kind: content` template type: a structured spec (title, body
markdown, fields, links, rows, per-channel title overrides,
severity-filtered visibility) rendered natively per channel — existing
`custom` (raw body) templates are untouched and byte-identical on the
send path.
- Per-channel renderers: Slack Block Kit, Teams (Adaptive Card +
MessageCard), Discord embeds, PagerDuty Events v2, Opsgenie, ServiceNow,
SNS, HTML/text email, and a stable machine `webhook` envelope.
- Escaping is renderer-owned (mrkdwn entities, HTML encoding, markdown
stripping) with a shared URL-scheme allowlist; golden-fixture tests per
channel.
- Template editor UI: guided content form (fields/links row tables,
compact rows config, channel title overrides), live preview panel with
per-channel preview + payload tab, test-send (rate-limited,
`[TEST]`-marked).
- Migration `m20260802_000001_add_template_kind`; prebuilt default
template + org backfill.
**2. Alert chart images (stateless, storage-free)**
- Alerts can attach a metric-history chart PNG rendered from the
triggers self-reporting stream (`actual_value` per evaluation), with
warn/crit threshold lines and the firing point appended.
- Zero storage: the chart URL *is* the data — deflate+base64url payload
signed with truncated HMAC-SHA256 (org-bound, expiry embedded), rendered
on fetch by the HTTP-terminating node (`GET
/api/v2/{org}/alerts/charts/render`). Signature verified before inflate
(decompression-bomb guard), uniform 404s, per-node render semaphore +
per-minute query budget + small LRU.
- Renderer: plotters + embedded DejaVu Sans (Bitstream Vera license
included), gap-broken line segments so sparse evaluation history doesn't
draw false plateaus.
- Channel delivery: Slack image block, email inline CID (no remote
fetch), Discord multipart attachment, Teams/PagerDuty image URLs;
per-destination opt-in via template chart flag.
- Slack layout live-A/B'd on a real channel: content + buttons/chart
split across two same-colored attachments so the severity stripe spans
everything while staying under Slack's "Show more" fold; clickable link
title; "View in OpenObserve" keeps a reserved primary-button slot.
- Slack image-delivery fallback: Slack validates image URLs server-side
and rejects the ENTIRE message with `400 invalid_attachments` when its
proxy can't fetch the chart (VPN-only/private `ZO_WEB_URL`) —
live-verified. On that rejection the send strips the image blocks and
resends once (alert always delivers), sets a process-wide flag so later
sends pre-strip (reachability is deployment-wide, not per-destination),
and re-probes hourly. Zero config; observable via `slack_image_rejected`
/ `slack_image_pre_stripped` metric events.
## Testing
- 147 notification unit tests (resolve/render/preview/test-send incl.
per-channel goldens and hostile-input/injection coverage), 8 chart
payload tests, renderer PNG tests.
- Live-verified end to end: Slack (Slackbot chart fetches ~100ms,
cache-immutable semantics), SES email inline image, multi-severity
firing scenario (warning→critical) with correct threshold lines.
- FE: AddTemplate/ContentTemplateForm/preview specs; UX audit pass with
fixes.
- `cargo clippy -D warnings` clean in both OSS and enterprise
arrangements.
## Enterprise pairing
Must merge together with openobserve/o2-enterprise#2353 (workspace
twin-manifest mirror for the new OSS deps).
Bumps the `openobserve/vortex` pin from `57ffbf2b` to `1cb48912` in the
workspace manifest.
## Upstream API changes handled
- Arrow interop moved into a dedicated `vortex-arrow` crate, surfaced as
`vortex::arrow`. `vortex::array::arrow::*` and `vortex::dtype::arrow::*`
no longer exist, so `FromArrowArray`, `FromArrowType`, `ToArrowType` and
`ArrowSessionExt` are now imported from `vortex::arrow`.
- `ToArrowType` is no longer in scope implicitly; added the explicit
import where `DType::to_arrow_schema()` is called.
- The `vortex::compressor` facade narrowed to `BtrBlocksCompressor`,
`BtrBlocksCompressorBuilder`, `Scheme`, `SchemeId`. `SchemeExt` and the
scheme list are no longer re-exported there, so `search` takes a direct
`vortex-btrblocks` dependency (same git rev) for `SchemeExt` /
`schemes::integer::IntDictScheme`, which the UTF8 compressor uses to
exclude the int-dict scheme.
No behavior change intended — the compressor configuration and write
strategies are identical.
## Verification
- `cargo check --workspace --all-targets` clean
- `cargo fmt --all -- --check` clean
- `cargo test -p search --lib datafusion::vortex` — 6 passed
- `cargo test -p tantivy_utils --lib vortex` — 3 passed
- `cargo test -p config --lib utils::parquet` — 21 passed
Paired enterprise PR (same branch name):
https://github.com/openobserve/o2-enterprise/pull/2303
## Summary
- decouple stream cleanup from enrichment-data and remove the transitive
compaction/DataFusion dependency from stream
- extract sourcemaps and organization storage services from core into
dedicated crates
- extract GitHub built-in regex access and model pricing into
openobserve-builtins
- rename pipeline and organization-storage persistence modules from
store to db
- remove now-unused direct dependencies from core
- synchronize the enterprise workspace overlay in
openobserve/o2-enterprise#2266
## Validation
OSS:
- cargo check -p openobserve-builtins -p openobserve-core -p
openobserve-jobs -p openobserve-api
- cargo test -p openobserve-builtins --lib
- cargo test -p openobserve-builtins --doc
- cargo clippy -p openobserve-builtins -- -D warnings
- cargo check -p openobserve-org-storage --features enterprise
Enterprise overlay:
- copied ../o2-enterprise/Cargo.toml.openobserve over Cargo.toml
- cargo fmt --all -- --check
- cargo check --all-targets --no-default-features --features enterprise
- cargo clippy --all-targets --no-default-features --features enterprise
-- -D warnings
The default enterprise + vectorscan build was attempted and reached
vectorscan-rs-sys, but the validation host does not have cmake
installed. The enterprise-only build and clippy run completed
successfully.
## Summary
Moves Vortex file-format support out of the enterprise build and makes
it a core, always-compiled capability in open source.
- **Code move**: the enterprise `search/vortex.rs` module
(`Utf8Compressor`, `VORTEX_RUNTIME`, `write_vortex`,
`generate_vortex_access_plan`, plus its unit tests) now lives at
`src/search/src/datafusion/vortex.rs`; the three call sites
(`merge/downsampling.rs`, `merge/mod.rs`, `table_provider/helpers.rs`)
reference the local module.
- **Feature removal**: the `vortex` cargo feature is deleted from the
root and all 11 sub-crate manifests. All `#[cfg(feature = "vortex")]` /
`#[cfg(all(feature = "enterprise", feature = "vortex"))]` gates and
their "requires the vortex feature" error branches are gone —
`FileFormat::Vortex` match arms compile unconditionally in config,
search, and tantivy_utils.
- **Dependencies**: `vortex` / `vortex-datafusion` are now direct git
dependencies (`openobserve/vortex` rev `57ffbf2b`, the same rev the
enterprise workspace pinned); the empty dependency-resolution stub
crates under `src/enterprise/vortex{,-datafusion}` are deleted.
Companion PR in o2-enterprise removes the vortex module/feature there
and syncs `Cargo.toml.openobserve`; that PR should merge together with
(or right after) this one, since its template relies on this repo no
longer declaring the feature.
## Verification
- `cargo check --workspace` (OSS, default features) passes
- `cargo check -p search -p tantivy_utils -p config --all-targets`
passes (vortex tests compile)
- Enterprise combined workspace (`Cargo.toml.openobserve` swapped in)
`cargo check -p openobserve` passes
- `cargo fmt` applied to touched crates
## Summary
This PR begins decomposing `openobserve-core` into focused service
crates while preserving the existing application behavior and dependency
direction.
It extracts six independently owned areas:
- `ingestion-common`: shared ingestion contracts and response types
- `schema`: schema inference, validation, evolution, and persistence
helpers
- `compaction`: merge, retention, and compaction services
- `enrichment-data`: enrichment-table lifecycle and runtime support
- `promql-service`: PromQL service orchestration
- `stream`: stream metadata, settings, retention, and lifecycle
operations
## What changed
- registered the extracted crates in the workspace and propagated
enterprise/cloud/vectorscan/vortex features
- moved implementations out of Core and updated API, Jobs, query, and
queue consumers to use the new crates
- removed Core facade aliases for extracted crates and converted
remaining callers to direct crate paths
- retained the small cross-service stream-delete coordinator in Core for
now because it still coordinates Pipeline and Alert cleanup
- restored required root/Core dependencies after validating the full
enterprise build
- added the missing `anyhow` dependency for enterprise-only
`search_service::search_jobs`
- updated the lockfile for the new crate dependency graph
## Dependency direction
The extracted crates do not depend on `openobserve-core`. Core remains a
temporary upper-level consumer while the remaining domain services are
split incrementally.
```text
ingestion-common -> common/config
schema -> ingestion-common/db/infra
stream -> schema/enrichment-data/db/infra
compaction -> schema/search/search-service/db/infra
openobserve-core -> ingestion-common/schema/stream/compaction/...
api/jobs -> openobserve-core + extracted services
```
## Enterprise compatibility
The private OpenObserve manifest was updated in
openobserve/o2-enterprise#2257 to register the extracted workspace
members and dependencies.
Related Enterprise PR:
https://github.com/openobserve/o2-enterprise/pull/2257
## Validation
OSS manifest:
- `cargo fmt --all -- --check`
- `cargo clippy -- -D warnings`
Using `o2-enterprise/Cargo.toml.openobserve` as the OpenObserve root
manifest:
- `cargo check --no-default-features --features enterprise`
- `cargo check --no-default-features --features cloud`
- `cargo clippy --workspace --no-default-features --features enterprise
-- -D warnings`
- `cargo clippy --workspace --no-default-features --features cloud -- -D
warnings`
Enterprise repository:
- `cargo fmt --all -- --check`
- `cargo clippy -- -A clippy::question-mark -D warnings`
All checks above pass in the development profile.
## Summary
- remove compatibility re-exports introduced during the recent crate
refactors
- migrate callers to the crates that own each API
- delete five pure facade modules and add the required direct crate
dependencies
- keep OSS and enterprise manifests aligned
## Validation
- cargo fmt --all -- --check
- cargo clippy --workspace -- -D warnings
- cargo test --locked --workspace --no-run
- enterprise overlay: cargo test --offline --workspace
--no-default-features --features enterprise --no-run
- enterprise overlay: cargo clippy --offline --workspace
--no-default-features --features enterprise -- -D warnings
- git diff --check
Companion enterprise PR uses the same branch name:
codex/remove-refactor-reexports.
## TODO / Follow-ups
- [ ] **Re-establish the pipeline write-guardrail lost with
`db::pipeline`.** That module was not a pure facade: it re-exported only
the six read-only accessors from `infra::pipeline` and its doc comment
recorded the invariant that *mutations must go through
`service::pipeline::store`* (which keeps the `ExecutablePipeline` caches
and the coordinator/super-cluster watch events in sync). Deleting it
points callers straight at `infra::pipeline`, which also exposes
`put`/`delete`/`drop_table` with no guardrail. Decide the layering and
enforce it:
- **Reads** may hit `infra` directly (raw persistence, no side effects).
- **Writes** must go through the wrapper that persists *and* emits the
coordinator/super-cluster event + refreshes the in-memory cache. That
wrapper is `db::<entity>::{set,delete}` for most entities (e.g.
`db::alerts::alert::set` →
`infra::coordinator::alerts::emit_put_event`), but for pipeline it is
`core::pipeline::store` (not `db`), because the
`STREAM_EXECUTABLE_PIPELINES` cache it maintains is a core-level
construct the `db` crate cannot build. The super-cluster receiver
(`super_cluster_queue::pipelines`) is the one sanctioned caller of
`infra::pipeline::{put,delete}` directly, and it manually pairs each
with `emit_*_event` to avoid a re-broadcast loop.
- Net invariant to protect: never call `infra::pipeline::{put,delete}`
without emitting the coordinator event — i.e. nowhere outside
`core::pipeline::store` and the super-cluster handler. A bare `infra`
write leaves in-memory caches stale until restart (see the warning
comment still live at `src/core/src/org_cleanup.rs:484`).
- [ ] Consider whether the pipeline write-wrapper living in
`core::pipeline::store` while every other entity's lives in
`db::<entity>` is worth unifying, or at least documenting as an
intentional exception.
## Summary
- extract persistence, cache, watcher data access, and DB service
modules from openobserve-core into a standalone workspace db crate
- keep application orchestration that depends on search, organization
provisioning, and other Core services behind the existing Core db facade
- preserve existing call paths through re-exports while making the
dependency direction openobserve-core -> db
- move HTTP error mapping back to the Core/API boundary so db has no
direct axum dependency
- avoid provider/provisioner callback traits and function-pointer hooks
## Validation
- cargo fmt --all
- cargo check --all-targets
- cargo check -p db --all-targets
- Enterprise workspace: cargo check --all-targets --offline
- Enterprise workspace: cargo check -p db --all-targets --features
enterprise --offline
Companion Enterprise manifest PR: openobserve/o2-enterprise#2230.
## Summary
Two crate-split refactors continuing the workspace decomposition:
**1. `transform` crate (renamed from `openobserve-vrl`, `src/vrl` →
`src/transform`)**
- Now the only crate depending on `vrl`, `vector-enrichment` and
`rquickjs` directly; downstream crates use `transform::vrl::` /
`transform::vector_enrichment::` re-export paths and drop their direct
dependencies
- The QuickJS runtime (`common/utils/js.rs`) moved into `transform::js`
(js_security checks gated behind the crate's `enterprise` feature);
`apply_vrl_fn`, `convert_to_vrl`/`convert_from_vrl` and
`TRANSFORM_FAILED` moved from core
- Exception: `src/config` keeps its direct `vrl` dependency (it sits
below `transform` in the dependency graph)
**2. `super_cluster_queue` extracted from the root crate into a
workspace member**
- Only depends on `openobserve-core`, `infra`, `config` and
`o2_enterprise`, so it builds in parallel with `api` and edits no longer
recompile the root crate (which re-embeds `web/dist` on every rebuild)
- Gated behind its own `enterprise` feature — compiles as an empty crate
in OSS builds so `cargo --workspace` keeps working against the
`o2_enterprise` stub; `vectorscan` wired through for `re_pattern.rs`
- Root `lib.rs` re-exports the crate, so
`openobserve::super_cluster_queue::init()` in `main.rs` is unchanged
Pairs with the `Cargo.toml.openobserve` template sync PR in
o2-enterprise (must merge together).
## Test plan
- `cargo check --workspace --all-targets` (OSS) passes
- `cargo check -p openobserve --no-default-features --features
enterprise` with the synced enterprise template passes
- `cargo clippy -p super_cluster_queue --features enterprise,vectorscan`
— zero warnings
- `cargo clippy -- -D warnings` (CI invocation) passes; `cargo fmt
--all` clean
## Summary
- move the MCP protocol, tool discovery, schema simplification,
execution, and response filtering implementation into a new OSS
`openobserve-mcp` workspace crate
- enable the real MCP POST, GET, and DELETE handlers in community builds
- initialize the MCP tool catalog from the OpenAPI document for every
edition
- keep OAuth authorization-server discovery enterprise-only because it
depends on Dex
- add OSS handler and OpenAPI initialization coverage
Closes#13266
Design at: #13266
## Impact
Community users can connect MCP clients directly to OpenObserve and use
the OpenAPI-backed tool catalog. Enterprise builds continue to use the
same MCP implementation from the OSS workspace crate.
Companion PR openobserve/o2-enterprise#2206 removes the old private copy
and updates the enterprise workspace overlay.
## Summary
- add a dedicated promql workspace crate for parsing and evaluation
- keep cluster routing, cache, WAL, file-list, and request orchestration
in openobserve-core
- preserve existing service::promql call sites through a compatibility
facade; the facade re-exports an explicit item list (not a glob), and
the promql search orchestration in core depends on the promql crate
directly
- inject enterprise query cancellation through TableProvider instead of
coupling the evaluator to core search state
- replace the core regex-UDF dependency with DataFusion regexp_like
expressions
## Companion PR
- openobserve/o2-enterprise#2198 updates the enterprise workspace
override for the new promql crate
## Validation
- cargo fmt --all -- --check
- cargo check --workspace
- cargo clippy -p promql --all-targets -- -D warnings
- cargo clippy -p openobserve-core
- cargo test -p promql: 363 passed
- verified the promql dependency tree has no openobserve-core or search
dependency
- cargo check -p o2_enterprise --lib in the paired private workspace
## Notes
The evaluator moves atomically in this PR. The existing
service::promql::search subtree remains in core because it owns
OpenObserve-specific orchestration rather than PromQL evaluation.
The two web/ file changes only update source-path references in comments
that pointed at the moved Rust files (src/service/promql/... ->
src/promql/src/...).
## Summary
- add `src/vrl` as a leaf package named `openobserve-vrl` with Rust path
`openobserve_vrl`
- move `QUERY_FUNCTIONS`, enrichment-table ownership, GeoIP/MaxMind
table registration, VRL compiler configuration, and shared runtime
helpers into `openobserve-vrl`
- let ingestion, jobs, core, and search use the shared VRL integration
without depending on one another
- keep DataFusion UDF registration direct through `Sql.org_id`; no
`QueryExecutionContext`, `PreparedQueryTransform`, or
`prepare_query_transforms`
- move the cipher registry, query parameters, and file-cache helpers to
`search`
- move the generic file downloader implementation to `infra`
- add feature forwarding for the `search` crate
- keep compatibility re-exports for existing API, ingestion, gRPC, jobs,
and Enterprise call sites
Enterprise companion:
https://github.com/openobserve/o2-enterprise/pull/2196
## Summary
- add the minimal `openobserve-search` workspace crate and move shared
search/cache types into it
- move the concrete enrichment table type and cache ownership to the
search crate while retaining normal compatibility re-exports
- move trace schema constants to `config` so the SQL engine no longer
reaches into the traces service
- expose concrete enrichment read functions from `infra` and make
DataFusion call them directly
- construct the stateless VRL runtime directly in the transform UDF
- keep search execution and orchestration in `openobserve-core` for the
later physical moves
- do not introduce `SearchRuntime`, provider traits, callbacks,
`#[path]`, or a search-to-core dependency
This combines the original crate-boundary step with the dependency
cleanup needed before SQL/DataFusion files can move safely.
## Validation
- `cargo test -p openobserve-search --lib` (11 passed)
- `cargo test -p openobserve-core service::enrichment::tests --lib` (19
passed)
- `cargo check -p openobserve --features mimalloc`
- enterprise + mimalloc check using the o2-enterprise overlay
- cloud + mimalloc check using the o2-enterprise overlay
- `cargo fmt --all -- --check`
- strict all-target clippy for `openobserve-search` and
`openobserve-core`
- strict library clippy for `infra`
- dependency check confirms `openobserve-search` does not depend on
`openobserve-core`
`infra --all-targets` currently reports three pre-existing inconsistent
digit-grouping warnings in
`src/infra/src/table/entity/alert_dedup_state.rs`; they are unrelated to
this change and are not included here.
## Problem
Every rebuild of `openobserve-api` re-runs the `RustEmbed` macro
expansion, which embeds `web/dist` (53MB, 1035 files) and compresses
every file with gzip + brotli(q11) — single-threaded, non-incremental,
and it peaks at 8GB RSS. `-Ztime-passes` attributes ~540s of the 769s
api compile to `expand_crate`. Net effect: touching one file in
`openobserve-core` costs a **16m45s** rebuild on an M-series laptop, and
neither incremental compilation nor codegen-unit tuning can help,
because macro expansion happens before both.
## Fix
Move the `#[derive(RustEmbed)]` into a new `src/web` leaf crate with no
workspace dependencies. The embed + compression now only recompiles when
`web/dist` changes; edits anywhere in the backend no longer pay for it.
**Measured (same machine, default profile settings, `--profile
release-profiling`):**
| | before | after |
|---|---|---|
| `openobserve-api` compile | 769s | 231s |
| touch `src/core` → binary | 16m45s | **6m44s** |
## Also in this PR
- **`src/web/build.rs`**: the derive has no file tracking, so a rebuilt
`web/dist` silently shipped stale embedded assets until something else
dirtied the api crate. `rerun-if-changed` on the embed folder fixes
that. The folder is parsed from the `#[folder]` attribute rather than
hardcoded, so it stays correct when the e2e workflows rewrite it. A
missing `index.html` now prints a build warning instead of silently
embedding nothing (the #13197 failure mode).
- **Playwright workflows**: the embed-folder `sed` now targets
`src/web/src/lib.rs` and is guarded with `grep -q`, so a future path
drift fails at the sed step instead of silently shipping an empty UI and
failing at the login page (see openobserve/o2-enterprise#2175 for that
incident).
- **`deny.toml`**: allow the new crate's AGPL-3.0 license (CI runs
`cargo deny check licenses`, which requires a per-crate exception).
## Validation
- `cargo deny check licenses` passes
- Smoke-tested the binary: `/healthz` ok, `/web/` serves `index.html`,
JS assets return 200 from the embedded bundle
- Enterprise overlay resolves: `cargo metadata` passes with the
companion `Cargo.toml.openobserve` change
## Companion
openobserve/o2-enterprise#2184 carries the overlay + e2e workflow sync
on the **same branch name** (`perf/web-embed-crate`); the two should
merge together.
## Summary
- add the `openobserve-api-management` crate under `src/api_management`
- physically move management handlers, models, and authentication
helpers out of `openobserve-api`
- preserve the existing API module paths through re-exports
- move shared stream permission checking into `openobserve-core` to
avoid a management-to-query dependency
- allow the new crate's `AGPL-3.0` license in `deny.toml` (fixes the
cargo-deny `check` job)
- keep router, OpenAPI generation, and rate-limit path mapping in
`openobserve-api`
## Dependency boundary
`openobserve-api-management` depends on `openobserve-api-common` and
`openobserve-core`; it does not depend on `openobserve-api-query`.
## Note on `split-debuginfo`
Removing `split-debuginfo = "packed"` from `[profile.release-profiling]`
is intentional (not a formatting artifact): profiling builds now keep
DWARF debuginfo embedded in the binary, so profiling artifacts can be
symbolized at runtime without shipping a separate debuginfo file. The
o2-enterprise overlay `Cargo.toml.openobserve` is kept in sync.
## Companion PR
- openobserve/o2-enterprise#2175
## Validation
- OSS check with `mimalloc`
- OSS strict clippy with all targets
- enterprise check and strict clippy
- cloud check and strict clippy
- `cargo test -p openobserve-api-management --lib`: 300 passed
- `cargo deny check licenses`
- `cargo fmt --all`
- `git diff --check`
Build-time benchmarking is intentionally deferred for this stage.
## Summary
- add `openobserve-api-common` for HTTP extractors and request DTOs
shared across API domain crates
- add `openobserve-api-query` and physically move the search, PromQL,
and traces handlers (about 16.4K lines)
- preserve the existing `handler::http::request::{search,promql,traces}`
paths through re-exports so routing, Swagger generation, and rate-limit
metadata remain compatible
- forward enterprise, cloud, vectorscan, and vortex features through the
new crate boundary
- avoid `#[path]` source inclusion
## Enterprise overlay
Requires openobserve/o2-enterprise#2174, which registers the two new
workspace crates in `Cargo.toml.openobserve`.
## Validation
- `cargo check -p openobserve --features mimalloc`
- enterprise check with `--no-default-features --features
enterprise,mimalloc`
- cloud check with `--no-default-features --features cloud,mimalloc`
- OSS and cloud/enterprise all-targets Clippy with `-D warnings`
- `openobserve-api-common`: 9 tests passed
- `openobserve-api-query`: 136 independent tests passed single-threaded;
the unchanged DB-dependent saved-view test was excluded because
standalone crate tests do not initialize the infra database
- `cargo fmt --all -- --check`
Build-time benchmarking is intentionally deferred until the crate-split
series is ready for a controlled `release-profiling` comparison.
Ashish Kolhe
Huaijin
IluckySiandHengfei Yang
Pinguladora