mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-05 16:10:43 +00:00
630ecf41f35e3c19a7fec4d00a595f2c3fde89c9
426
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Hengfei Yang
|
2f53bf2f37 | fix: keep trace columns in quick mode projections (#13376) (#14169) | ||
  Ashish Kolheandgithub-actions[bot]
|
07807995e4 |
feat: public status pages built on synthetics checks (#13974)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
 Huaijin
|
732f721d04 | fix: vortex Utf8-typed string statistics broke Utf8View interval analysis (#14011) | ||
 
|
b09b56e25a |
feat(llm): add Phase 3.0 experiment workflows (#13778)
## Summary - allow dataset rows without an expected output across management APIs and UI flows - add immutable experiment preview, creation, listing, persistence, RBAC, and super-cluster synchronization - add Experiment as an evaluation target with slot hydration, scoring, and result indexing - harden idempotent creation, scorer-version pinning, score-index setup, and upgrade migrations ## Tickets - Related to [openobserve/o2-enterprise#2398](https://github.com/openobserve/o2-enterprise/issues/2398) - Related to [openobserve/o2-enterprise#2399](https://github.com/openobserve/o2-enterprise/issues/2399) - Related to [openobserve/o2-enterprise#2400](https://github.com/openobserve/o2-enterprise/issues/2400) ## Paired Enterprise PR - [openobserve/o2-enterprise#2413](https://github.com/openobserve/o2-enterprise/pull/2413) ## Validation - focused Rust and frontend tests for the affected dataset and experiment seams - experiment migration test passes - upgraded local environment applies schema version 68 and returns 200 from GET /api/team3/experiments --------- Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Ashish Kolhe
|
b2639c4f9f |
feat(dbm): Database Monitoring (#13918)
## What this is Database Monitoring: an aggregation and presentation layer over telemetry OpenObserve already ingests. It reads three vantages — client spans, server-vantage logs, and collector metrics — and turns them into per-statement, per-instance and per-database views. No new storage engine and no new ingest path. Statements are normalized and rolled up into a stats stream, and the read API queries that stream through the existing search path. ## Shape of the change | | | |---|---| | Read endpoints | 17 | | UI pages | 8 (plus a shell) | | Commits | 141 | **Backend** — `src/core/src/db_monitoring/` holds the non-HTTP half (normalization, rollup, server-vantage parsing). The read API lives in `src/api/management/src/request/db_monitoring/` as a handler / models / service trio; the service layer is split one file per feature. **Frontend** — `web/src/views/DatabaseMonitoring/`, with shared logic in `web/src/composables/dbm/`. ## Feature availability Databases, queries, query detail and history, endpoints, samples, activity, insights, plans, server metrics, server queries, server samples, instances, instance metrics, badges — is available in OSS. ## Reviewing this It is a large diff, and most of it is additive: 327 files, and the two largest non-test files are the shared service module and the deadlocks service. Suggested order: 1. `src/api/management/src/request/db_monitoring/handler.rs` — the 17 entry points and their gating 2. `src/api/management/src/request/db_monitoring/service/common.rs` — auth, SQL building, the shared read path 3. `src/core/src/db_monitoring/rollup.rs` — how the stats stream is produced 4. `web/src/views/DatabaseMonitoring/` — the pages Time semantics are the subtlest part. Rollup rows are stamped at the window **end**, so reads are exclusive at both edges; `stats_read_range` carries the reasoning and a test pins it. ## Known follow-ups - **The server-vantage stream named as `_o2_dbm_server`.** ## Testing - Rust: unit tests across the DBM core and read service, including SQL-building, auth-gating and normalizer cases - Web: unit tests across the DBM pages and composables - Playwright: DBM suite covering the pages and cross-page navigation Verified before opening: OSS and enterprise builds both compile warning-free, `cargo fmt` clean, `vue-tsc` clean, and the full web unit suite passes (44,704 tests). |
||
|
Huaijin
|
37e5f55684 |
feat(metrics): .midx series index for TSID-major files and PromQL pruning through it (#13899)
The query half of the TSID-major layout: #13882 wrote `tsid-major-v3-*` files ordered by `(__hash__, _timestamp)`; this PR adds the `.midx` series index next to them and makes PromQL prune through it. Supersedes the remaining scope of #13839. ## Write side - Every `tsid-major-v3-*` file written by the hour-end merge gets a `.midx` series index (Arrow IPC + zstd), stored like the Tantivy index under a derived root distinct from the data tree: `files/{org}/metrics/{stream}/{date}/{hour}/tsid-major-v3-{id}.parquet` → `files/{org}/midx/{stream}/{date}/{hour}/tsid-major-v3-{id}.midx` (Tantivy: `files/{org}/index/{stream}_{type}/…/{id}.ttv`): **one row per contiguous TSID run** — its `row_count: u32` plus the label columns of the run's first row (source types preserved). Runs tile the data file, so each run's starting row is the prefix sum of the preceding counts and is not stored; `__hash__` is excluded (matchers never reference it). The whole index is one ZSTD-compressed IPC batch. Runs crossing an input RecordBatch boundary appear as two adjacent entries; query-time range coalescing makes that representation equivalent without buffering samples. File and row-group boundaries stay independent of TSID boundaries. - `TsidSeriesIndexWriter` lives inside `search/datafusion/merge/tsid_major.rs` and collects run entries alongside the Parquet writer (one buffered row per run), encoding and compressing once at finish. `MergedFile` carries `series_index: Option<Vec<u8>>`; the compactor uploads it to the `midx/` tree (compliance mode included). - `.midx` is intentionally **not** tracked in `file_list`; its path is derived from the data file's key (`MetricsFileLayout::series_index_path`) and it is deleted together with the parent in the delayed-deletion pass (`deleted.rs`); like the `index/` tree, the `midx/` day directories are covered by retention's local-disk deletion and the gc tool. ## Query side (PromQL) - `promql_service::…::tsid_series_index::search` runs where Tantivy used to, before the metrics table is registered: for each TSID-major file it loads the `.midx` (concurrent, `2×target_partitions ≤ 64`), projects the row-range columns plus the matched labels **by name against each series index's own schema** (label sets / column orders differ between files), builds the matcher filter **per series index** with `physical_expr::create_physical_expr` — the exact same predicates as `apply_matchers` (`matcher_predicates`) — evaluates it over the run rows, coalesces adjacent/overlapping ranges and attaches them as `FileSelection::RowRanges` (converted into a Parquet access plan by the generic helper from #13861, no per-row bitmap). - **Applied per file**: files of any other layout in the same query (still-open hours, legacy data, late arrivals) are left in place and scanned in full. A matched label missing from a series index over-selects that file; a query whose matchers cannot be evaluated at all selects the whole file; any series index failure logs and falls back to a full scan. The final PromQL matchers are always applied, so results are exact in every case. - Evaluated selections are cached (LRU, 256 MiB) keyed by account + series index path + the full matcher set. - The Tantivy path in the PromQL storage scan is removed (`convert_matchers_to_index_condition`, index-field plumbing, `is_add_filter_back` bookkeeping). SQL search is untouched. ## Benchmark (from the #13839 POC, same layout + series index) 478,144,896 rows, 7 CPUs, one-hour PromQL range, step 15s, result cache off, release + mimalloc: | Layout | Filter steady median | No-filter steady median | Data files | Total storage | |---|---:|---:|---:|---:| | main, 2 GiB | 695.709 ms | 4.638 s | 88 | 4,925.81 MiB | | TSID-major + `.midx` | 269.496 ms | 2.793 s | 80 | 1,294.61 MiB | ## Validation - `cargo fmt --check`; `cargo check` / `clippy` on config, search, compaction, promql-service, jobs, search_service (`--tests`) clean - `config tsid_layout` 3 passed (incl. `series_index_path`); `search tsid_major mode:: merge::` 10 passed (rotation test now also asserts the series index, exact-run and coalescing series index tests); `compaction merge deleted` 33 passed; `promql-service` 20 passed + 1 pre-existing wall-clock flake (`cache::test_promql_cache_set_and_get`, fails intermittently on main too) - End-to-end on the #13839 branch (same code paths): hour-end merge writes N `tsid-major-v3-*` + `.midx`; PromQL takes the series index path (`selected 39 ranges across 4/4 files`), counts/sums match the synthetic data, cold-cache identical after restart; a mixed hour (3 major + 1 late `tsid-sorted`) reports `1 files without a series index left to tantivy/full scan` with unchanged results; delayed deletion removes `.parquet` **and** `.midx` |
||
|
Huaijin
|
08d31371cc |
refactor(metrics): prepare indexed metrics groundwork (#13907)
## Summary - rename the existing TSID-major config, modules, types, merge modes, and persisted file prefixes to indexed-metrics terminology - add the promql-service dependency declarations needed by the follow-up metrics-index work - carry the exact Cargo.lock content from #13899, including h2 0.4.16 and the promql-service dependency entries ## Scope This is no-new-feature groundwork split from #13899. It retains the existing metrics layout mechanics under the new names and does not add .midx sidecar writing, index lookup/query pruning, or index lifecycle handling. The old environment variable and file prefixes are intentionally not kept as compatibility aliases because this experimental path has not been used in a deployment. ## Validation - cargo fmt --all -- --check - cargo metadata --locked --no-deps --format-version 1 - cargo check -p promql-service - cargo test -p config --lib -- layout (5 passed) - cargo test -p search --lib -- datafusion::merge (8 passed) - cargo test -p compaction --lib -- merge (32 passed) - cargo check -p openobserve-jobs |
||
|
Huaijin
|
789eee463f |
feat(metrics): TSID-major Parquet layout behind ZO_METRICS_TSID_MAJOR_ENABLED (#13882)
The storage-layout half of #13839, without the `.sidx` series index and its PromQL query path (those stay in #13839, which will rebase on this). ## What Behind `ZO_METRICS_TSID_MAJOR_ENABLED` (default off), Parquet metrics files of a stream whose `__hash__` column is `UInt64` are written ordered by `(__hash__, _timestamp)` so every series is contiguous: - **`config::meta::promql::tsid_layout`** — `metrics_tsid_major_enabled(stream_type)` (flag + configured Parquet), `metrics_tsid_major_stream(stream_type, schema)` (also `__hash__: UInt64`; JSON-ingested metrics store `__hash__` as a string and keep the classic layout), and `MetricsFileLayout { Legacy, HashSorted, TsidMajor }` encoded in the file-name prefix (`tsid-sorted-*`, `tsid-major-v3-*`) so readers and later merges know the physical order without opening the file. - **`MergeMode::HashSorted`** (open hour): the ingester movers and incremental compactor merges write one hash-ordered file named `tsid-sorted-*`. In pack mode the single-segment upload-as-is fast path is skipped for such streams (segments are in arrival order). - **`MergeMode::TsidMajor`** (closed hour): `merges_whole_batch()` — every file of the hour, regardless of size — into size-split `tsid-major-v3-*` files written at RecordBatch boundaries using `ZO_COMPACT_MAX_FILE_SIZE` as logical original size (`search/datafusion/merge/tsid_major.rs`). `MergeMode::for_compactor / for_ingester` take the schema and pick the mode via `metrics_tsid_major_stream`; downsampling keeps precedence. The compactor skips a closed hour that is already all `tsid-major-v3-*`. - **`MergedFile.layout`** decides the output file name (`layout.file_name` / `mark_file_key`) in the compactor and the ingester movers. - **Merging pre-sorted inputs**: `MergeMode::input_sort_order(&files)` declares `FileSortOrder::HashTimestampAsc` when every input is hash-ordered (`tsid-sorted-*` / `tsid-major-v3-*`). Overlapping hash ranges make DataFusion drop the order for its size-based groups; the listing adapter regroups one file per partition (single-file groups are trivially ordered) and keeps the declared `output_ordering`, so `EnforceSorting` turns the `SortExec` into a `SortPreservingMergeExec` over N pre-sorted streams (memory N × batch instead of the whole hour). A batch that mixes layouts declares no order and falls back to the full sort. Two file states only: `tsid-sorted-*` (ordered, not finalized) and `tsid-major-v3-*` (finalized). Nothing changes for streams without the flag, non-Parquet metrics, or a non-UInt64 `__hash__`. ## Known limitation The SQL query path still derives `sorted_by_time` from the SQL shape only, so with the flag on a `SELECT … FROM <metrics> ORDER BY _timestamp DESC LIMIT n` may assume a timestamp order that hash-sorted files do not have. PromQL is unaffected (no ordering declared). To be addressed with the query-side work. ## Validation - `cargo fmt --check`; `cargo check` / `cargo clippy` on config, search, compaction, jobs, promql-service, search_service (`--tests`) clean - `cargo test -p config --lib -- tsid_layout` 2 passed; `cargo test -p search --lib -- merge:: mode:: tsid_major listing_adapter` 9 passed (mode selection, per-layout input order, hash `ORDER BY`, size-split rotation, SPM plan shape); `cargo test -p compaction --lib -- merge` 31 passed - End-to-end on #13839 (debug build, local disk, OTLP gauge 300 series): ingester uploads `tsid-sorted-*` verified `(__hash__, _timestamp)` ordered; hour-end `merge [tsid_major] input sort order: hash_timestamp_asc` → plan `SortPreservingMergeExec` ← `DataSourceExec{N single-file groups}` (no `SortExec`) → `tsid-major-v3-*`; incremental `merge [hash_sorted]` 2 → 1 `tsid-sorted-*`; JSON-ingested stream (Utf8 `__hash__`) keeps the classic layout with no compactor errors. ## TODO (follow-up, not in this PR) **Sort TSID-major streams by `(__hash__, _timestamp)` already at WAL dump time.** `Partition::persist` full-sorts every dumped chunk anyway (today by `_timestamp DESC` in `merge_record_batches`); switching the sort key for TSID-major streams is near-free at dump time and removes the biggest ingester-side cost of this layout — the full `SortExec` over the merge input: 1. **persist**: lexsort the chunk by `(__hash__ ASC, _timestamp ASC)` (same gate as the writers, `metrics_tsid_major_stream`) and stamp a `sort_order: hash_timestamp_asc` key-value into the Parquet footer. A file-name marker is not an option for WAL files — `parse_time_range_from_filename` splits on `.` and a prefix would break WAL time pruning. 2. **ingester mover**: it already reads every input's footer for the schema; declare `HashTimestampAsc` on the table only when *all* inputs carry the footer marker (else `FileSortOrder::None`, full sort — handles mixed files during rolling upgrades / flag flips, same role `MetricsFileLayout::of` plays for the compactor). The merge then becomes a `SortPreservingMergeExec` over pre-sorted files instead of a full sort. 3. **pack, single segment**: the as-is fast path (skipped for TSID-major streams in this PR because segments were time-sorted) can come back — the segment is already hash-sorted, upload it unchanged and mark it `tsid-sorted-*`. 4. **pack, multiple segments**: one `MemTable` partition per segment with a declared sort order, so the in-memory merge is also an SPM instead of a full sort. Query-side is already safe for this change: metrics SQL never declares `sorted_by_time` and PromQL declares no order, so hash-sorted WAL files cannot mislead a reader. The cost is slightly coarser time clustering inside WAL files, negligible at WAL file sizes. Steps 1–2 carry most of the win; 3–4 are incremental. |
||
|
Huaijin
|
4e805038fe |
refactor: merge result / access-plan / deletion helpers groundwork for TSID-major metrics (#13861)
Behavior-neutral refactors split out of #13839 so that PR only carries the TSID-major feature. Nothing here changes what is written, read or deleted; every commit is independently reviewable. ## Commits - `cba75d244b` **promql**: `apply_matchers` split into `matcher_predicates(schema, matchers) -> Vec<Expr>` + `apply_matchers` (unchanged behavior; lets other inputs evaluate the same matcher semantics). - `b5ff6fed21` **parquet**: `new_parquet_writer` / `append_metadata` generic over `W: AsyncFileWriter` instead of `&mut Vec<u8>`; existing callers unchanged. - `b07d84216a` **merge**: `MergeParquetResult::{Single, Multiple}` → `MergeParquetResult { files: Vec<MergedFile { buf, meta }>, file_format }` + `into_single()`. The compactor's two copies of the upload / index / push sequence become one loop (`compressed_size` is set from the buffer for every file, as the `Multiple` arm already did); downsampling maps its `(bufs, metas)` into it; ingester movers use `into_single()` instead of panicking/erroring on `Multiple`. - `014b6c5721` **compactor**: `deleted.rs` `derived_files()` / `delete_from_storage()` replace three copies of collect / `storage::del` / ignore-not-found (data, `.ttv`, flattened). - `7163786362` **search**: `generate_parquet_access_plan_from_ranges` extracted from the bitmap path (`set_slices` → ranges); an out-of-range, empty or unsorted range is rejected with `assert!` (`56de386522`; a producer that violates the sorted/non-overlapping contract is a bug, not something to degrade around) instead of the previous `unreachable!()` on the bitmap path. Adds `FileSelection::RowRanges(Arc<Vec<Range<usize>>>)` as the matching selection kind (no producer yet; Parquet only, Vortex full scan like `RowGroups`). - `754014ccf3` **test**: `ListingTableAdapter` plan-shape test for `FileSortOrder::HashTimestampAsc` (from #13853): three overlapping hash-sorted files → `SortPreservingMergeExec` over three single-file groups, no `SortExec`, globally ordered rows. `tempfile` as `search` dev-dependency. ## Validation - `cargo fmt --check` - `cargo clippy -p config -p promql -p search -p compaction -p openobserve-jobs -p search_service -p promql-service --tests` clean - `cargo test -p promql --lib -- utils` 10, `-p search --lib -- merge:: helpers listing_adapter` 14, `-p compaction --lib -- merge deleted` 31, `-p config --lib -- test_file_key` 4 passed - `merge/downsampling.rs` is enterprise-only and was updated by hand; needs the enterprise CI compile. |
||
 Harsh Mahajan
|
0ed5336bf1 | chore: adding HA mode for o2-ai agent (#13525) | ||
|
Hengfei Yang
|
c0d49810b3 |
perf: re-add process-wide promql series label cache (#13748)
## Summary Restores the process-wide PromQL series label cache that was added in #13109 and removed in #13148, ported onto the partitioned label-loading pipeline introduced by #13148. Labels are immutable per series hash, so a warm cache lets repeated queries skip the second (label-column) scan entirely — the biggest wins are on `by()`-grouped aggregations, where the label phase is a fixed cost that the cache removes. ## Changes - **`src/promql/src/label_cache.rs`** (ported from #13109): sharded LRU keyed by `(context fingerprint, series hash)`, bounded by **memory size** — `ZO_METRICS_LABEL_CACHE_MAX_SIZE` in MB, `0` (default) = auto: 5% of total memory clamped to [100 MB, 1 GiB]. - **Reads are always served; admission gates writes only.** A lookup cannot grow the cache, so even a query too big to admit benefits from entries other queries warmed. Admission is sized by the query's actual **misses** (the only entries it could insert) and bypasses **writes** when they exceed 50% of the budget, so one oversized query cannot evict the shared cache. Hit/miss metrics therefore reflect real lookup outcomes. - **`load_series_labels`** attaches cached labels partition-parallel (rayon) before the scan and collects per-partition missing hashes: - when every series hits, the DataFusion label scan is **skipped entirely** - the miss scan is narrowed to missing series: a hash in-list filter for up to 8192 misses, plus a per-missing-series max-timestamp filter (falls back to the sample scan's timestamp set when everything misses) - Partition tasks only attach/cache series in their missing set; extracted label sets are stored back into the cache, sharing the label interner's `Arc<Label>` values (hot label values are deduped inside the cache too). - Cached entries exclude the synthetic `__hash__` label; it is prepended per query, so raw-data and regular queries share entries. - Restores the `query_metrics_label_cache_hit_count` / `query_metrics_label_cache_miss_count` metrics (per organization). ## Performance Re-validated on the current partitioned pipeline: warm queries are **~20% faster than main** on the high-cardinality benchmark dataset below. The detailed tables are from #13109 (same cache design and dataset, measured on the pre-#13148 pipeline): single node (M-series Mac, 32GB), 1.34B rows / 3.17M series of a histogram bucket stream, result cache disabled, warm = median of 3 runs after first. **What the cache itself contributes** (cache on vs off on the same binary, `ZO_METRICS_LABEL_CACHE_MAX_SIZE=1` as the off-switch): | Query | cache on | cache off | cache contribution | |---|---|---|---| | `sum by (region) (rate(m[5m]))` | **3.62s** | 5.40s | **1.49x** | | `histogram_quantile(0.9, sum by (le) (rate(m[5m])))` | **3.53s** | 4.93s | **1.40x** | | `count by (path) (rate(m[5m]))` | **3.73s** | 5.56s | **1.49x** | | no-modifier aggregations (skip path), `topk` (write-bypassed), small filters | — | — | none (by design) | The cache is precisely a **`by()`-grouped-query optimization**: 1.4–1.5x there, zero effect (and zero overhead) everywhere else. Cold runs with the cache are on par with cache-off (`sum by (region)` cold: 5.33s cached vs 5.30s uncached). **Admission control** (`topk(5, rate(m[5m]))` needs full labels for all 3.17M series ≈ 3GB working set, past the 1 GiB auto budget): | Configuration | time | |---|---| | cache disabled (baseline behavior) | 10.4s | | cache without admission, 1 GiB budget (thrash) | 12.0s ⚠️ | | **cache with admission, 1 GiB budget (bypasses)** | **10.6s — parity restored** | | cache with `ZO_METRICS_LABEL_CACHE_MAX_SIZE=8192` (fits, admitted) | **4.5s — 2.3x faster** | The thrash row is what admission prevents — and it was caused by insert + eviction churn, which is why the current revision gates only writes and keeps reads always on. Notes on reproducing: this dataset is deliberately extreme, so its `by()` working sets (~0.6–0.9GB) exceed the default budget's 50% single-query share — set `ZO_METRICS_LABEL_CACHE_MAX_SIZE=2048` or higher to admit them. At typical production cardinality (≤1M series), `by()` working sets are admitted at defaults. Within a single query, selectors share the cache (the second operand of a binary op loads labels in 315ms vs the 2.57s the first paid). ## Operational notes Re-introduces `ZO_METRICS_LABEL_CACHE_MAX_SIZE` and the two cache metrics removed in #13148. Sizing guide: a typical series (~10 labels) costs ~1KB, so the 1 GiB auto cap holds roughly 1M series; column-pruned entries from `by()` queries are far smaller (~100B), so grouped queries fit ~10M series. A tiny budget (e.g. `1`) acts as a de-facto off switch for writes. ## Testing - 4 label-cache unit tests (get/put, byte-bounded eviction, admission control, fingerprint) ported from #13109 - New tests: partial-miss scan attaches only missing series and populates the cache; cached entries exclude `__hash__`; cached-attach hit/miss partitioning - All 374 `promql` crate tests pass; full workspace `cargo check` and clippy clean - Output verified numerically identical (#13109 methodology, max relative diff 1e-13 from float summation order) |
||
|
Huaijin
|
d5fa24711c |
perf(metrics): choose label timestamp filter automatically (#13756)
## Summary - replace the `ZO_METRICS_INLIST_FILTER_ENABLED` switch with automatic timestamp filter selection - use an exact `IN` filter for up to 1024 timestamps and a bounded `BETWEEN` predicate above that limit - sort exact timestamp literals for stable query plans - add boundary coverage for empty, threshold, and over-threshold timestamp sets ## Rationale Large timestamp `IN` lists add planning and evaluation overhead, while always using `BETWEEN` can admit many unrelated label rows for sparse timestamp sets. A bounded exact-list strategy keeps the selective path for manageable sets without allowing query plans to grow indefinitely. Release benchmarks used 4.2M rows, five label columns, four hash partitions, and repeated median measurements: - with a representative sparse range returning about 262K rows through `BETWEEN`, 1024 timestamps favored `IN` (9.1 ms vs 10.1 ms), while 2048 favored `BETWEEN` (11.7 ms vs 10.2 ms) - when unique maximum timestamps equaled the series count and both predicates returned exactly one row per series, `BETWEEN` consistently won and the gap grew with list size, confirming the need for an upper bound ## Testing - `cargo test -p promql series_labels::tests` - `cargo clippy -p promql --tests --no-deps -- -D warnings` - `git diff origin/main...HEAD --check` ## Operational note `ZO_METRICS_INLIST_FILTER_ENABLED` is removed and no longer needs to be configured. |
||
|
Hengfei Yang
|
238b0d5262 |
fix: support zstd-compressed HTTP requests (#13728)
## Summary - enable tower-http zstd request decompression - add a router middleware regression test using a real zstd-compressed JSON payload - update decompression documentation to list zstd support ## Root cause The Axum router uses `RequestDecompressionLayer`, but the workspace did not enable tower-http's `decompression-zstd` feature. Requests from clients such as Vector configured with `compression: "zstd"` were therefore rejected with `415 Unsupported Media Type` before reaching the ingestion handler. ## User impact Vector HTTP sinks can continue using zstd compression when sending batches to OpenObserve ingestion endpoints. ## Validation - `cargo fmt --all -- --check` - `cargo test -p openobserve-api-http handler::http::router::decompression::tests` - `git diff --check` Fixes #13726 |
||
 Pinguladora
|
575e8ea4d5 |
feat: add zstd compression on OTLP gRPC ingest (#13703)
Design at: #13702 / inline in PR description Add the `zstd` feature to tonic and enable `accept_compressed(Zstd)` on the OTLP logs, metrics and trace gRPC services in both `run_common` and run_router`. Closes #13702 ## Out of scope - Response compression on `send_compressed(Zstd)` - OTLP/HTTP path (tower-http zstd features) - snappy would be the better choice under CPU pressure, but tonic has no snappy support. |
||
|
Huaijin
|
aaed33543c |
refactor(search): group DataFusion and Tantivy feature configs (#13664)
## Summary - add ZO_FEATURE_METRICS_PUSHDOWN_FILTER_ENABLED as an opt-in filter pushdown switch for metrics queries - group DataFusion and Tantivy feature configuration under the search config struct - move inverted-index result cache sizing, footer caches, skip threshold, TopN group limit, and bloom footer cache settings from limit to search - preserve the moved cache settings' environment variable names and defaults - apply the metrics-specific setting to object storage, WAL, and Flight query contexts ## Testing - cargo fmt --all -- --check - cargo check for config, infra, search, tantivy_utils, search_service, and promql-service - cargo test -p config every_env_config_default_parses --lib - cargo test -p config inverted_index_config --lib - cargo test -p search test_create_session_config --lib - cargo clippy for all affected crates with -D warnings |
||
|
Ashish Kolhe
|
58c7dafa55 |
fix: notification templates v2 — rich content templates + alert chart images (#13640)
## Summary
Two coupled alerting features:
**1. Notification Templates v2 ("content" templates)**
- New `kind: content` template type: a structured spec (title, body
markdown, fields, links, rows, per-channel title overrides,
severity-filtered visibility) rendered natively per channel — existing
`custom` (raw body) templates are untouched and byte-identical on the
send path.
- Per-channel renderers: Slack Block Kit, Teams (Adaptive Card +
MessageCard), Discord embeds, PagerDuty Events v2, Opsgenie, ServiceNow,
SNS, HTML/text email, and a stable machine `webhook` envelope.
- Escaping is renderer-owned (mrkdwn entities, HTML encoding, markdown
stripping) with a shared URL-scheme allowlist; golden-fixture tests per
channel.
- Template editor UI: guided content form (fields/links row tables,
compact rows config, channel title overrides), live preview panel with
per-channel preview + payload tab, test-send (rate-limited,
`[TEST]`-marked).
- Migration `m20260802_000001_add_template_kind`; prebuilt default
template + org backfill.
**2. Alert chart images (stateless, storage-free)**
- Alerts can attach a metric-history chart PNG rendered from the
triggers self-reporting stream (`actual_value` per evaluation), with
warn/crit threshold lines and the firing point appended.
- Zero storage: the chart URL *is* the data — deflate+base64url payload
signed with truncated HMAC-SHA256 (org-bound, expiry embedded), rendered
on fetch by the HTTP-terminating node (`GET
/api/v2/{org}/alerts/charts/render`). Signature verified before inflate
(decompression-bomb guard), uniform 404s, per-node render semaphore +
per-minute query budget + small LRU.
- Renderer: plotters + embedded DejaVu Sans (Bitstream Vera license
included), gap-broken line segments so sparse evaluation history doesn't
draw false plateaus.
- Channel delivery: Slack image block, email inline CID (no remote
fetch), Discord multipart attachment, Teams/PagerDuty image URLs;
per-destination opt-in via template chart flag.
- Slack layout live-A/B'd on a real channel: content + buttons/chart
split across two same-colored attachments so the severity stripe spans
everything while staying under Slack's "Show more" fold; clickable link
title; "View in OpenObserve" keeps a reserved primary-button slot.
- Slack image-delivery fallback: Slack validates image URLs server-side
and rejects the ENTIRE message with `400 invalid_attachments` when its
proxy can't fetch the chart (VPN-only/private `ZO_WEB_URL`) —
live-verified. On that rejection the send strips the image blocks and
resends once (alert always delivers), sets a process-wide flag so later
sends pre-strip (reachability is deployment-wide, not per-destination),
and re-probes hourly. Zero config; observable via `slack_image_rejected`
/ `slack_image_pre_stripped` metric events.
## Testing
- 147 notification unit tests (resolve/render/preview/test-send incl.
per-channel goldens and hostile-input/injection coverage), 8 chart
payload tests, renderer PNG tests.
- Live-verified end to end: Slack (Slackbot chart fetches ~100ms,
cache-immutable semantics), SES email inline image, multi-severity
firing scenario (warning→critical) with correct threshold lines.
- FE: AddTemplate/ContentTemplateForm/preview specs; UX audit pass with
fixes.
- `cargo clippy -D warnings` clean in both OSS and enterprise
arrangements.
## Enterprise pairing
Must merge together with openobserve/o2-enterprise#2353 (workspace
twin-manifest mirror for the new OSS deps).
|
||
|
Huaijin
|
e1e3091c06 |
refactor: move streaming aggregates to OSS (#13589)
## Summary - move streaming aggregation cache, aggregate execution, and optimizer code into the OSS search crate - move cache aggregation SQL analysis into the SQL visitor and enable streaming aggregation in search_service without enterprise gating - move MetadataCountExec into OSS and use it for metadata count optimization in all builds - remove migration-only dead code while keeping the cache module under search Companion cleanup: openobserve/o2-enterprise#2327. ## Testing - cargo fmt --all -- --check - cargo test -p search metadata_count --lib - cargo test -p search aggregate_optimize_rewrite --lib - cargo test -p search streaming_aggregate --lib - cargo test -p search cache::streaming_agg --lib - cargo test -p search datafusion::optimizer::stream_aggregate --lib - cargo test -p search_service partition --lib - cargo check -p search_service |
||
|
Huaijin
|
872ceb724a |
refactor: move search and compaction optimizations to OSS (#13583)
## Summary - move optimized Tantivy simple and multi histogram collectors into the open-source config crate - move general broadcast join eligibility, rewrite, execution, TmpExec codec, and shared async state into open-source search - enable enrichment-table broadcast join in OSS, including join reordering, remote-scan rewriting, EnrichmentExec, and its physical-plan codec - move AggregateTopkExec, its heap/sort streams, physical optimizer rule, and codec into open-source search - replace the enterprise-only Aggregate TopK settings with OSS-owned `ZO_DF_USE_AGG_TOPK_HEAP` and `ZO_DF_TOPK_HEAP_MAX_LIMIT` settings - move `approx_topk` and `approx_topk_distinct` UDAFs into open-source search and register them in both OSS and enterprise builds - omit the Aggregate TopK benchmark, benchmark documentation, and data-generator example from OSS - move Bloom term extraction, post-merge build, `.bf` writing, and orphan cleanup into the OSS compaction crate - run the Bloom build path after compaction in both OSS and enterprise builds, with non-fatal fallback to regular Tantivy search - move the direct `tantivy` and `object_store` dependency ownership required by Bloom building into OSS compaction - enable sorted timestamp, block, RANK, broadcast join, Aggregate TopK, approximate TopK, and Bloom paths in both OSS and enterprise builds - retain cross-super-cluster TmpExec fetching behind the enterprise feature - remove feature-gated enterprise imports and use the shared OSS implementations - add fastdivide for histogram bucket computation - resolve the duplicated SLO test attribute warning and remove an unused test-only alert-state migration helper ## Tests - `cargo fmt --all -- --check` (OSS and enterprise) - `cargo clippy --locked -p search --lib -- -D warnings` - `cargo clippy --locked -p compaction --lib -- -D warnings` - `cargo clippy --locked -p o2_enterprise --lib -- -D warnings` - `cargo test -p config tantivy::query::histogram_collector --lib` (21 passed) - `cargo test -p search tantivy::search::tests --lib` (9 passed) - `cargo test -p search enrichment --lib` (2 passed) - `cargo test -p search broadcast_join --lib` (7 passed) - `cargo test -p search join_reorder --lib` (8 passed) - `cargo test -p search datafusion::distributed_plan::codec::tmp_exec --lib` (1 passed) - `cargo test -p search aggregate_topk --lib` (7 passed) - `cargo test --locked -p search approx_topk --lib` (8 passed) - `cargo test --locked -p config meta::slo --lib` (371 passed) - `cargo test --locked -p infra bloom --lib` (30 passed) - `cargo check --locked -p config --tests` - `cargo check --locked -p infra --tests` - `cargo check --locked -p search --lib` - `cargo check --locked -p compaction --lib` - `cargo check --locked` (OSS root package) - `cargo check --locked --workspace` (enterprise workspace) - `cargo check --offline -p compaction --lib --features enterprise` with the paired repositories - `cargo check --offline -p openobserve --no-default-features --features enterprise` with the paired repositories Companion enterprise cleanup PR: https://github.com/openobserve/o2-enterprise/pull/2326 |
||
|
Huaijin
|
015a20a184 |
refactor: remove HTTP API compatibility re-exports (#13560)
## Summary - remove the HTTP request/model compatibility re-exports introduced by the domain crate split - migrate router, OpenAPI, enterprise action-server, and integration-test callers to the crates that own each API - remove the ingest Content-Type re-export and declare the direct root/test dependencies - delete the pure `handler::http::models` facade module ## Validation - `cargo fmt --all -- --check` - `cargo clippy -p openobserve-api-ingest -p openobserve-api-http -- -D warnings` - `cargo test --test integration_test --no-run` - `cargo check -p openobserve --no-default-features` - enterprise overlay: `cargo test --offline --workspace --no-default-features --features enterprise --no-run` - cloud overlay: `cargo test --offline --workspace --no-default-features --features cloud --no-run` - `git diff --check` |
||
|
Huaijin
|
6c029f7a2c |
chore(deps): bump vortex to 1cb48912 (#13540)
Bumps the `openobserve/vortex` pin from `57ffbf2b` to `1cb48912` in the workspace manifest. ## Upstream API changes handled - Arrow interop moved into a dedicated `vortex-arrow` crate, surfaced as `vortex::arrow`. `vortex::array::arrow::*` and `vortex::dtype::arrow::*` no longer exist, so `FromArrowArray`, `FromArrowType`, `ToArrowType` and `ArrowSessionExt` are now imported from `vortex::arrow`. - `ToArrowType` is no longer in scope implicitly; added the explicit import where `DType::to_arrow_schema()` is called. - The `vortex::compressor` facade narrowed to `BtrBlocksCompressor`, `BtrBlocksCompressorBuilder`, `Scheme`, `SchemeId`. `SchemeExt` and the scheme list are no longer re-exported there, so `search` takes a direct `vortex-btrblocks` dependency (same git rev) for `SchemeExt` / `schemes::integer::IntDictScheme`, which the UTF8 compressor uses to exclude the int-dict scheme. No behavior change intended — the compressor configuration and write strategies are identical. ## Verification - `cargo check --workspace --all-targets` clean - `cargo fmt --all -- --check` clean - `cargo test -p search --lib datafusion::vortex` — 6 passed - `cargo test -p tantivy_utils --lib vortex` — 3 passed - `cargo test -p config --lib utils::parquet` — 21 passed Paired enterprise PR (same branch name): https://github.com/openobserve/o2-enterprise/pull/2303 |
||
|
Huaijin
|
17ef03b8d6 |
refactor: cut release build time in the web and telemetry paths (#13528)
Two changes that cut build time, both measured on the same machine (M2
Max,
12 cores) with `cargo build --features mimalloc --profile release`.
## 1. The `web` crate no longer depends on `config`
`src/web` is a leaf crate so the RustEmbed expansion over `web/dist`
only
re-runs when the assets change — that expansion is 149.8s, of which
144.1s is
macro expansion (the crate already sets `#[br = false]` / `#[gzip =
false]`, so
this is not compression).
It also depended on `config`, for exactly one value:
`cfg.common.base_uri`. That
put a 50MB embed downstream of a 724-crate subgraph, so **every `config`
edit
paid the 148.5s embed again** — which is precisely what the crate's own
doc
comment says it exists to avoid.
`ui_routes()` now takes the `<base href>` value from the caller and
`base_href_middleware` carries it as middleware state.
`create_app_router()`
already computed that exact string (`"{base_uri}/web/"`) for its own
redirect,
so it is now resolved once and passed in rather than recomputed per
request
from a second `get_config()` call. `base_uri` is env-derived, normalised
once at
config init and never touched by `refresh_config`, so hoisting it out of
the
request path cannot drift.
The web crate's dependency tree goes **724 → 92 crates**, and `cargo
tree -i
config` no longer lists it.
## 2. Logging and tracing setup moved out of `main.rs`
`setup_logs`, `enable_tracing` and their span-processor / exporter types
compose
very large generic trees. Everything `main.rs` monomorphizes is emitted
into the
binary's own codegen units — the last unit on the critical path,
overlapping
with nothing.
`-Ztime-passes` on the binary unit (91.1s total):
```
LLVM_thinlto 39.8s
LLVM_passes 37.5s
codegen_to_LLVM_IR 5.3s
link_binary 5.3s (run_linker 3.9s)
type_check_crate 0.4s
```
The `openobserve` library does not depend on `openobserve-api-http`, so
its unit
compiles in parallel with the API crates — in the cold build it finished
23.9s
before the binary needed it. Moving the setup into `src/telemetry.rs`
(563
lines) puts that work in the slack instead of at the tail.
Two tests moved with the code and were dropped rather than relocated:
`test_setup_logs` and `test_enable_tracing_error_handling` both install
a global
tracing subscriber, and `setup_logs()` calls `.init()` unguarded, so
they can
only ever pass depending on ordering. They used to race just with each
other
inside the bin test binary; keeping them would have moved that race into
the lib
test binary next to the `cli` and `migration` tests. Neither asserted
anything —
the second wraps the call in `catch_unwind` and discards the result.
`main.rs` 1304 → 712 lines. **Binary unit 88.8s → 80.7s**; the library
unit grows
30.9s → 62.3s and is fully absorbed by the existing slack.
## 3. Unrelated: a dead-code warning in `openobserve-core`
Not a build-time change, just a warning fix noticed while checking the
above.
`COMPARE_ROLLUP_LOOKBACK_MICROS` was gated on `any(feature =
"enterprise",
test)`, but its only two readers (`fetch_arm_aggregate`,
`fetch_arm_failures`)
are `enterprise`-only and the unit tests below it exercise only the pure
aggregation helpers. So in an OSS build `cargo check -p openobserve-core
--all-targets` warned `constant COMPARE_ROLLUP_LOOKBACK_MICROS is never
used`.
Gate narrowed to `enterprise`, and the doc comment — which claimed the
opposite
— corrected. Happy to split this out if you'd rather it not ride along.
## 4. Review follow-ups
- `enable_tracing` became a public library API in §2 without a doc
comment,
unlike `setup_logs` beside it. Documented: which span processors attach
under
which env vars, why the returned provider must be held rather than
dropped,
and that it and `setup_logs` are mutually exclusive since both call
`.init()`.
- Added the `<base href>` test described under Verification below.
Not taken: the suggestion to re-add `test_setup_logs` /
`test_enable_tracing_error_handling` to `src/telemetry.rs`. Both install
a
global subscriber against `.init()`, so they cannot be made
order-independent,
and neither asserted anything — see §2.
`test_resource_key_value_creation` was
deleted for the same reason; it asserted that
`opentelemetry::KeyValue::new`
stores the key it was given.
## Measured and rejected
Recording these so nobody repeats them.
**Moving route registration into the crates that own the handlers.** 317
of the
371 routes were registered in one 622-line function in
`openobserve-api-http`.
Splitting them out to api-management (234), api-search (46),
api-pipelines (23)
and api-ingest (9) worked as designed — api-http went 145.0s → 63.0s —
but the
cold build only improved 654.4s → 637.7s. Splitting across a crate
boundary
**copies** work rather than relocating it: without LTO each crate
instantiates
the axum/tower machinery for itself, so the four receiving crates grew
222.1s
combined and total CPU rose 190.8s. It also just relocated the
bottleneck —
api-management became the new tail at 135.1s, with api-search finishing
only
6.9s behind it, so the tail is a plateau rather than a peak. Not carried
in this
branch.
| Tried | Result |
|---|---|
| sccache | 97.4% hit rate, **0** wall-clock gain; first fill +54% |
| Remove aws-lc-sys from the critical path | **0** — its `dur=72.0s` in
`--timings` is contention, it needs 15.6s alone |
| `codegen-units = 16` on the glue crates | **31.4s slower**, CPU
+297.8s — `lto` defaults to `false`, so thin-local LTO is on and its
cost scales with CGU count |
| Trim DataFusion features | `datafusion-core` depends on
`datafusion-functions` without `default-features = false`, so the
expression features cannot be turned off from the facade at all; what
remains is worth 13.8s of CPU (0.25%) |
A caveat on reading `cargo --timings`: a unit's `duration` is **not**
its
intrinsic cost — it is inflated by CPU contention when the machine is
saturated.
The aws-lc entry above is the clearest example.
## Verification
At this revision:
- `cargo check --all-targets` clean on OSS — 0 errors, 0 warnings — and
each
commit checks clean standalone, so `git bisect` works
- `cargo test -p web` passes
Carried over from the revision these changes were first written against
(the
edits since are doc comments, the `ui_routes` parameter rename, two
deleted
tests, a `Cargo.lock` revert, and the `cfg` narrowing in §3 — which is a
no-op
when `enterprise` is on):
- `cargo check` clean on the enterprise feature combination
- Cold release build succeeds
- Smoke-tested against a running binary: `/healthz` 200, UI and API
routes
resolve, unknown paths 404
The `<base href>` rewrite — the one runtime behaviour §1 touches — is
now
pinned by `base_href_is_rewritten_to_the_mount_point`, and checked end
to end
against the real embedded `index.html`: `ui_routes("/abc/web/")` serves
`<base href="/abc/web/" />`, matching what the old per-request
`format!("{base_uri}/web/")` produced for `ZO_BASE_URI=/abc`. The
committed test
drives a synthetic HTML response instead, so it still asserts something
in
builds where `web/dist` is absent. Verified by mutation: breaking the
rewrite
fails it.
|
||
|
Hengfei Yang
|
2bf19b2b44 |
feat(mcp): support MCP spec revision 2026-07-28 (stateless, dual-era server) (#13527)
Design at: #13526 ## Summary Implements MCP specification revision **2026-07-28** (stateless protocol) alongside the existing 2025-11-25 behavior on the same endpoint, following the spec's dual-era server model: requests carrying modern per-request `_meta` are served per 2026-07-28, while `initialize` selects legacy semantics. Legacy clients are unaffected. ## Changes - **Version negotiation** (`src/api/http/.../mcp/mod.rs`, `src/mcp/src/protocol.rs`): the `MCP-Protocol-Version` header is now validated against a supported-version set (`2026-07-28`, `2025-11-25`) instead of strict equality — the old check would have rejected every client that upgrades to the new revision. Unsupported versions (header or body `_meta`) get `400` + JSON-RPC `-32022` (`UnsupportedProtocolVersionError`) listing the supported versions so clients can retry. - **`server/discover`** (`src/mcp/src/protocol.rs`): new RPC required by 2026-07-28, returning `supportedVersions`, `capabilities`, server identity in `_meta`, `instructions`, and `ttlMs`/`cacheScope` cache fields. - **Modern result decoration** (`src/mcp/src/protocol.rs`): requests declaring `io.modelcontextprotocol/protocolVersion: 2026-07-28` in `_meta` get `resultType: "complete"` and `_meta` server identity on results, plus `ttlMs`/`cacheScope` on `tools/list`. Legacy responses carry none of these fields — byte-for-byte unchanged. - **Request metadata headers** (`src/api/http/.../mcp/mod.rs`): modern requests must mirror body fields into `Mcp-Method` / `Mcp-Name` headers (including the Base64 sentinel encoding for `Mcp-Name`); missing or mismatched headers are rejected with `400` + JSON-RPC `-32020` (`HeaderMismatch`), per the transport's server-validation rules. - **HTTP status mapping**: unknown methods on modern requests return `404` + `-32601` so dual-era clients can distinguish a modern server from a legacy one; version errors return `400`. Legacy requests keep errors in-band with `200` as before. - **Legacy paths untouched**: `initialize`, `ping`, the GET SSE keepalive channel, DELETE no-op, and notification → `202` handling are unchanged. Tool ordering was already deterministic (sorted at init), satisfying the new caching guidance. ## Testing - `cargo test -p openobserve-mcp` — 94 tests pass (new coverage: `server/discover` shape, unsupported-version error payload, modern decoration, legacy non-decoration, sentinel decoding, version set). - `cargo test -p openobserve-api-http --lib handler::http::request::mcp` — 13 tests pass (new coverage: modern happy path, missing/mismatched `Mcp-Method`/`Mcp-Name`, header↔body version mismatch, missing version header on modern requests, modern unknown method → 404, legacy header accepted). - `cargo clippy` clean on both crates. |
||
|
Hengfei Yang
|
94bddd7a7b |
feat: pack ingester wal persist into large files to avoid massive small files (#13497)
Design at: https://github.com/openobserve/openobserve/issues/13509 Closes #13509 ## Problem On clusters with 100k+ metrics streams, every memtable rotation persists **one parquet file per stream × hour**, producing thousands of tiny files per second on the ingester. The legacy file mover cannot keep up: it re-scans the whole `wal/files` tree every `file_push_interval` (capped at `file_push_limit` = 10000 files per cycle ≈ a hard ceiling of ~1000 files/s), repeatedly stats not-yet-expired files, and pays a full DataFusion plan + S3 PUT per stream×hour for a few KB of data. Generation outpaces consumption and wal files pile up unboundedly. ## Design Behind `ZO_FEATURE_WAL_PACK_ENABLED` (bool, default **off**), applying to **metrics streams only** — the stream-cardinality problem is metrics-specific; other stream types keep the legacy per-stream persist path. With the flag on, the "split by stream" step moves from persist time (disk) to upload time (memory): - **Persist**: all streams of a memtable are appended into a few large **pack** files (`wal/pack/{idx}/{memtable_id}.{seq}.pack`, rolled at `ZO_MAX_FILE_SIZE_ON_DISK`, default 512MB). Each segment inside a pack is a **complete, self-contained parquet file**; the pack footer (JSON + gxhash trailer, versioned) indexes segments with stream, partition key, offset/length and FileMeta. One rotation now writes O(1) sequential files instead of O(streams) tiny ones. - **Query**: an in-memory segment index (rebuilt from footers on startup) resolves stream → byte ranges; segments are materialized into record batches and reuse the memtable search table-building path. Dedup against in-memory data uses the existing memtable_ids mechanism. - **Mover**: pending streams are snapshotted from the segment index (no readdir/stat storm, no `file_push_limit` cap). A stream is flushed when its segments are big enough or the oldest exceeds `max_file_retention_time`. A **single segment is uploaded as-is (zero decode/re-encode)**; multiple segments are merged in memory via DataFusion MemTable + the existing `merge_parquet_files`. Object keys, file_list, incremental compaction trigger, tantivy index, cache_latest_files and enterprise service-streams all follow the legacy mover. **S3 layout, file_list schema, compactor and querier are completely unchanged.** - **Crash recovery**: packs follow the same `.lock` five-step flow as legacy files (`.pack.tmp` → `.lock` → delete wal → rename → unlock), keeping exactly-once semantics. Lock recovery stays synchronous on startup (it must run before the pack index rebuild), while the orphan `.par` sweep over the potentially millions-of-files `wal/files` tree moved to a background task so startup no longer blocks. - **Pack recycling**: a registry tracks per-pack `{total, consumed, readers}`; a pack file is deleted once all segments are consumed and no in-flight query read holds it (read guards replace the legacy `lock_files`/pending-delete mechanism for packs). Also included: shared-memtable hash changed from `{thread_id}_{org_id}` to `{org_id}_{stream_name}` so one stream maps to exactly one bucket, removing the per-request round-robin fan-out that multiplied file/segment counts. ## Testing - Unit tests: pack format roundtrip, size-based rollover, corruption/truncation rejection, real-parquet segment read through the index with time-range/memtable-id filtering, consume lifecycle (pack deleted when fully consumed), reader-guard deferred deletion, flush conditions. - End-to-end with the real binary (4 phases): 1. wal binary format is byte-identical to main (`git diff main... -- src/wal src/ingester/src/entry.rs` is empty); 2. old-format wal written with the flag off **replays into packs** with the flag on — no migration needed; 3. mixed-source search returns exact counts with legacy wal parquet + pack segments + memtable coexisting (300/400/500 across three batches, filters and row fetches correct); 4. pack mover merges multiple segments into one storage file, generates the tantivy index, deletes consumed packs; final storage-side queries and `match_all` are exact. ## Unrelated fixes included Two startup paths shared the same "treat any db error as absence" bug: - `init_db` used to treat ANY error from `get_db_schema_version` as version 0 ("assuming default of 0 and trying upgrade") and kicked off the full db upgrade — so a merely unreachable or overloaded database got migration DDL piled on top of its existing load. Now `get_db_schema_version` returns 0 only when the version is genuinely absent (missing key, or the meta table itself not created yet on a fresh install); transient errors are retried 5 times with backoff and then abort the startup so the orchestrator restarts the node once the database recovers. - bootstrap did the same with the **instance id**: `instance::get()` reported a missing key as an error, so "not stored yet" and "db unreachable" landed in one arm that generated a fresh instance id — and could overwrite the stored one once the db recovered. The instance id must stay stable for the lifetime of a deployment, so a new one is now generated only when it is genuinely absent; read errors are retried with backoff and then abort the startup, and a failed write of a newly generated id is no longer ignored. ## Notes for reviewers - Flag is default **off**; with the flag off, behavior is unchanged except the startup fixes and the shared-memtable hash change. The flag only gates the persist path — pack search always runs (no-op on an empty index), so packs already on disk stay searchable after the flag is turned off while the mover drains them. - Verified with the real binary that with the flag on, a metrics stream is packed while a logs stream keeps the legacy per-stream parquet, both queryable (SQL + PromQL). - Pairs with o2-enterprise branch `feat/ingester-wal-pack-drain` (drain completion check also scans `wal/pack`); that change must land before/with enabling the flag in production, otherwise a draining node could shut down with pack segments not yet uploaded. - Known follow-ups (not in this PR): pack-specific metrics, extending pack to other stream types, large-scale load test for segment-index memory sizing, `wal-reader`-style debugging tool for packs. |
||
|
Huaijin
|
ef3617f157 |
refactor: split HTTP APIs by domain (#13479)
## Summary
- organize API crates under
`src/api/{common,grpc,http,ingest,management,pipelines,search}`
- keep ingest, management, pipelines, and search independent from one
another; compose them only in the HTTP frontend
- move pattern APIs into search and simplify `main.rs` wiring
- update workspace manifests, lockfile, integration tests, and
cargo-deny entries
- keep `web/` unchanged
## Validation
- `cargo fmt --all -- --check`
- `cargo clippy --all-targets -- -D warnings`
- `cargo deny check licenses`
- enterprise overlay: `cargo build --no-default-features --features
enterprise`
- enterprise overlay: `cargo build --no-default-features --features
cloud`
- enterprise overlay: `cargo clippy --all-targets -- -D warnings`
The matching `o2-enterprise` overlay PR will be linked after creation.
|
||
|
Huaijin
|
d165299bcd |
refactor: extract TLS and node services from core (#13449)
## Summary - extract TLS certificate loading and client configuration from openobserve-core into openobserve-tls - extract node discovery, cluster information, ingester gRPC channel selection, and Tokio runtime metrics into openobserve-node - update Core, API, API Management, Jobs, and the root binary to depend on the new crates directly - preserve one-way dependencies: openobserve-core depends on openobserve-node, while openobserve-node does not depend on Core - register both crates in the enterprise workspace overlay ## Validation - cargo test -p openobserve-node --lib - CARGO_INCREMENTAL=0 cargo clippy -p openobserve-node -p openobserve-core -p openobserve-jobs -p openobserve-api-management -p openobserve-api -p openobserve -- -D warnings - RUSTFLAGS="--cfg tokio_unstable" cargo check -p openobserve-node - enterprise overlay: CARGO_INCREMENTAL=0 cargo clippy -p openobserve --no-default-features --features enterprise -- -D warnings - cargo deny check licenses --hide-inclusion-graph - cargo fmt --all -- --check - git diff --check Companion enterprise PR: openobserve/o2-enterprise#2267 |
||
|
Huaijin
|
3f9ce27542 |
refactor: split service dependencies out of core (#13446)
## Summary - decouple stream cleanup from enrichment-data and remove the transitive compaction/DataFusion dependency from stream - extract sourcemaps and organization storage services from core into dedicated crates - extract GitHub built-in regex access and model pricing into openobserve-builtins - rename pipeline and organization-storage persistence modules from store to db - remove now-unused direct dependencies from core - synchronize the enterprise workspace overlay in openobserve/o2-enterprise#2266 ## Validation OSS: - cargo check -p openobserve-builtins -p openobserve-core -p openobserve-jobs -p openobserve-api - cargo test -p openobserve-builtins --lib - cargo test -p openobserve-builtins --doc - cargo clippy -p openobserve-builtins -- -D warnings - cargo check -p openobserve-org-storage --features enterprise Enterprise overlay: - copied ../o2-enterprise/Cargo.toml.openobserve over Cargo.toml - cargo fmt --all -- --check - cargo check --all-targets --no-default-features --features enterprise - cargo clippy --all-targets --no-default-features --features enterprise -- -D warnings The default enterprise + vectorscan build was attempted and reached vectorscan-rs-sys, but the validation host does not have cmake installed. The enterprise-only build and clippy run completed successfully. |
||
|
Huaijin
|
6123240a64 |
refactor: clean up core service dependencies (#13436)
## Summary
- remove the remaining openobserve-core Tantivy facade and let Jobs
depend on tantivy_utils directly
- inject enrichment-table cleanup from Core so the stream crate no
longer depends on enrichment-data (and transitively compaction)
- define a StreamDataCleanup port for organization deletion and move the
compaction implementation to Jobs
- remove openobserve-core's direct compaction dependency and enterprise
feature forwarding
- allow OSS builds to preserve ZO_FILE_FORMAT=vortex now that Vortex
support is always compiled
- add regression coverage for Vortex configuration and injected stream
cleanup failures
## Dependency direction
jobs -> tantivy_utils
jobs -> compaction
jobs -> openobserve-core::StreamDataCleanup
core -> stream + enrichment-data
core -X-> compaction (direct)
stream -X-> enrichment-data -> compaction
## Verification
- cargo fmt --all -- --check
- cargo check -p stream -p openobserve-core -p openobserve-jobs
- cargo test -p stream --lib (24 passed)
- cargo test -p config --lib (2027 passed, 2 ignored)
- cargo test -p openobserve-core org_cleanup --lib (14 passed)
- cargo clippy -p config -p stream -p openobserve-core -p
openobserve-jobs -- -D warnings
Enterprise-only check is currently blocked by the existing
openobserve-cipher build error: o2_enterprise::enterprise is unavailable
in this checkout.
|
||
|
Huaijin
|
df9d35c3d3 |
feat(search): move vortex support from enterprise into open source (#13434)
## Summary
Moves Vortex file-format support out of the enterprise build and makes
it a core, always-compiled capability in open source.
- **Code move**: the enterprise `search/vortex.rs` module
(`Utf8Compressor`, `VORTEX_RUNTIME`, `write_vortex`,
`generate_vortex_access_plan`, plus its unit tests) now lives at
`src/search/src/datafusion/vortex.rs`; the three call sites
(`merge/downsampling.rs`, `merge/mod.rs`, `table_provider/helpers.rs`)
reference the local module.
- **Feature removal**: the `vortex` cargo feature is deleted from the
root and all 11 sub-crate manifests. All `#[cfg(feature = "vortex")]` /
`#[cfg(all(feature = "enterprise", feature = "vortex"))]` gates and
their "requires the vortex feature" error branches are gone —
`FileFormat::Vortex` match arms compile unconditionally in config,
search, and tantivy_utils.
- **Dependencies**: `vortex` / `vortex-datafusion` are now direct git
dependencies (`openobserve/vortex` rev `57ffbf2b`, the same rev the
enterprise workspace pinned); the empty dependency-resolution stub
crates under `src/enterprise/vortex{,-datafusion}` are deleted.
Companion PR in o2-enterprise removes the vortex module/feature there
and syncs `Cargo.toml.openobserve`; that PR should merge together with
(or right after) this one, since its template relies on this repo no
longer declaring the feature.
## Verification
- `cargo check --workspace` (OSS, default features) passes
- `cargo check -p search -p tantivy_utils -p config --all-targets`
passes (vortex tests compile)
- Enterprise combined workspace (`Cargo.toml.openobserve` swapped in)
`cargo check -p openobserve` passes
- `cargo fmt` applied to touched crates
|
||
|
Huaijin
|
1624a6ff24 |
fix(schema): restore persistence guards on auto-UDS settings write (#13431)
## Summary #13405 rewired `handle_diff_schema`'s auto-UDS persistence from `stream::save_stream_settings` to a direct `db::schema::update_setting` call (required by the new crate dependency direction: `schema` cannot depend on `stream`). The direct write dropped the persistence guards that path provided. This PR restores them inside the schema crate, without reintroducing a dependency on the stream service: - skip persisting when the stream is being deleted (`db::compact::retention::is_deleting_stream`), matching the old `save_stream_settings` early-return - enforce `cfg.limit.user_defined_schema_max_fields` on the persisted field list (previously, with `ZO_SCHEMA_MAX_FIELDS_TO_ENABLE_UDS` set above the cap, settings that were never persisted before #13405 would now be written); skipped writes are logged at warn level instead of silently dropped - sort `defined_schema_fields` before persisting so the stored order is deterministic (was `HashSet` iteration order, nondeterministic across nodes/restarts) Also restores the `calculate_files_size` unit tests that were dropped when `src/core/src/file_list.rs` was deleted in #13405 — they now live next to the function in `infra::file_list`. ## Validation - `cargo check -p schema -p infra` - `cargo test -p schema --lib` (7 passed) - `cargo test -p infra --lib file_list::tests::test_calculate_files_size` (5 passed) - `cargo clippy -p schema -p infra` - `cargo fmt -p schema -p infra -- --check` |
||
|
Huaijin
|
15275ae7cf |
refactor: split core foundation services (#13405)
## Summary This PR begins decomposing `openobserve-core` into focused service crates while preserving the existing application behavior and dependency direction. It extracts six independently owned areas: - `ingestion-common`: shared ingestion contracts and response types - `schema`: schema inference, validation, evolution, and persistence helpers - `compaction`: merge, retention, and compaction services - `enrichment-data`: enrichment-table lifecycle and runtime support - `promql-service`: PromQL service orchestration - `stream`: stream metadata, settings, retention, and lifecycle operations ## What changed - registered the extracted crates in the workspace and propagated enterprise/cloud/vectorscan/vortex features - moved implementations out of Core and updated API, Jobs, query, and queue consumers to use the new crates - removed Core facade aliases for extracted crates and converted remaining callers to direct crate paths - retained the small cross-service stream-delete coordinator in Core for now because it still coordinates Pipeline and Alert cleanup - restored required root/Core dependencies after validating the full enterprise build - added the missing `anyhow` dependency for enterprise-only `search_service::search_jobs` - updated the lockfile for the new crate dependency graph ## Dependency direction The extracted crates do not depend on `openobserve-core`. Core remains a temporary upper-level consumer while the remaining domain services are split incrementally. ```text ingestion-common -> common/config schema -> ingestion-common/db/infra stream -> schema/enrichment-data/db/infra compaction -> schema/search/search-service/db/infra openobserve-core -> ingestion-common/schema/stream/compaction/... api/jobs -> openobserve-core + extracted services ``` ## Enterprise compatibility The private OpenObserve manifest was updated in openobserve/o2-enterprise#2257 to register the extracted workspace members and dependencies. Related Enterprise PR: https://github.com/openobserve/o2-enterprise/pull/2257 ## Validation OSS manifest: - `cargo fmt --all -- --check` - `cargo clippy -- -D warnings` Using `o2-enterprise/Cargo.toml.openobserve` as the OpenObserve root manifest: - `cargo check --no-default-features --features enterprise` - `cargo check --no-default-features --features cloud` - `cargo clippy --workspace --no-default-features --features enterprise -- -D warnings` - `cargo clippy --workspace --no-default-features --features cloud -- -D warnings` Enterprise repository: - `cargo fmt --all -- --check` - `cargo clippy -- -A clippy::question-mark -D warnings` All checks above pass in the development profile. |
||
|
Huaijin
|
eec3207c24 |
refactor: decouple API compilation from web assets (#13403)
## Summary - move embedded UI routing from `openobserve-api` into the existing `web` crate - inject `web::ui_routes` into the API router while preserving the existing `ui_enabled` and missing-`dist` behavior - make `openobserve-api` and `web` sibling dependencies so they can compile independently ## Validation - `cargo fmt --all --check` - Enterprise manifest: `cargo clippy --workspace --no-default-features --features enterprise,mimalloc -- -D warnings` - `cargo clippy --workspace -- -D warnings` - `cargo clippy -p web --all-targets -- -D warnings` - `cargo test -p web` - `cargo deny check licenses` |
||
|
Huaijin
|
6adcae3a1f |
refactor: remove crate compatibility re-exports (#13381)
## Summary
- remove compatibility re-exports introduced during the recent crate
refactors
- migrate callers to the crates that own each API
- delete five pure facade modules and add the required direct crate
dependencies
- keep OSS and enterprise manifests aligned
## Validation
- cargo fmt --all -- --check
- cargo clippy --workspace -- -D warnings
- cargo test --locked --workspace --no-run
- enterprise overlay: cargo test --offline --workspace
--no-default-features --features enterprise --no-run
- enterprise overlay: cargo clippy --offline --workspace
--no-default-features --features enterprise -- -D warnings
- git diff --check
Companion enterprise PR uses the same branch name:
codex/remove-refactor-reexports.
## TODO / Follow-ups
- [ ] **Re-establish the pipeline write-guardrail lost with
`db::pipeline`.** That module was not a pure facade: it re-exported only
the six read-only accessors from `infra::pipeline` and its doc comment
recorded the invariant that *mutations must go through
`service::pipeline::store`* (which keeps the `ExecutablePipeline` caches
and the coordinator/super-cluster watch events in sync). Deleting it
points callers straight at `infra::pipeline`, which also exposes
`put`/`delete`/`drop_table` with no guardrail. Decide the layering and
enforce it:
- **Reads** may hit `infra` directly (raw persistence, no side effects).
- **Writes** must go through the wrapper that persists *and* emits the
coordinator/super-cluster event + refreshes the in-memory cache. That
wrapper is `db::<entity>::{set,delete}` for most entities (e.g.
`db::alerts::alert::set` →
`infra::coordinator::alerts::emit_put_event`), but for pipeline it is
`core::pipeline::store` (not `db`), because the
`STREAM_EXECUTABLE_PIPELINES` cache it maintains is a core-level
construct the `db` crate cannot build. The super-cluster receiver
(`super_cluster_queue::pipelines`) is the one sanctioned caller of
`infra::pipeline::{put,delete}` directly, and it manually pairs each
with `emit_*_event` to avoid a re-broadcast loop.
- Net invariant to protect: never call `infra::pipeline::{put,delete}`
without emitting the coordinator event — i.e. nowhere outside
`core::pipeline::store` and the super-cluster handler. A bare `infra`
write leaves in-memory caches stale until restart (see the warning
comment still live at `src/core/src/org_cleanup.rs:484`).
- [ ] Consider whether the pipeline write-wrapper living in
`core::pipeline::store` while every other entity's lives in
`db::<entity>` is worth unifying, or at least documenting as an
intentional exception.
|
||
|
Huaijin
|
f6560d7d02 |
refactor: decouple search from core (#13338)
## Summary - extract search orchestration from core into the `search_service` crate - keep the existing low-level `search` crate and directory names unchanged - extract usage reporting and audit services into dedicated crates - replace global async function pointers with narrow publisher traits - keep compatibility re-exports while updating workspace dependencies and feature propagation Companion enterprise PR: https://github.com/openobserve/o2-enterprise/pull/2229 ## Validation - `cargo check --workspace` - `cargo check -p search --tests` - `cargo test -p search_service --lib` (184 passed) - enterprise non-release build with the paired `o2-enterprise` manifest - `cargo fmt --all -- --check` - `git diff --check` |
||
|
Huaijin
|
e236c7e653 |
refactor: extract database layer into db crate (#13340)
## Summary - extract persistence, cache, watcher data access, and DB service modules from openobserve-core into a standalone workspace db crate - keep application orchestration that depends on search, organization provisioning, and other Core services behind the existing Core db facade - preserve existing call paths through re-exports while making the dependency direction openobserve-core -> db - move HTTP error mapping back to the Core/API boundary so db has no direct axum dependency - avoid provider/provisioner callback traits and function-pointer hooks ## Validation - cargo fmt --all - cargo check --all-targets - cargo check -p db --all-targets - Enterprise workspace: cargo check --all-targets --offline - Enterprise workspace: cargo check -p db --all-targets --features enterprise --offline Companion Enterprise manifest PR: openobserve/o2-enterprise#2230. |
||
|
Huaijin
|
b835ff2f86 |
refactor: consolidate vrl/js into transform crate and extract super_cluster_queue crate (#13324)
## Summary Two crate-split refactors continuing the workspace decomposition: **1. `transform` crate (renamed from `openobserve-vrl`, `src/vrl` → `src/transform`)** - Now the only crate depending on `vrl`, `vector-enrichment` and `rquickjs` directly; downstream crates use `transform::vrl::` / `transform::vector_enrichment::` re-export paths and drop their direct dependencies - The QuickJS runtime (`common/utils/js.rs`) moved into `transform::js` (js_security checks gated behind the crate's `enterprise` feature); `apply_vrl_fn`, `convert_to_vrl`/`convert_from_vrl` and `TRANSFORM_FAILED` moved from core - Exception: `src/config` keeps its direct `vrl` dependency (it sits below `transform` in the dependency graph) **2. `super_cluster_queue` extracted from the root crate into a workspace member** - Only depends on `openobserve-core`, `infra`, `config` and `o2_enterprise`, so it builds in parallel with `api` and edits no longer recompile the root crate (which re-embeds `web/dist` on every rebuild) - Gated behind its own `enterprise` feature — compiles as an empty crate in OSS builds so `cargo --workspace` keeps working against the `o2_enterprise` stub; `vectorscan` wired through for `re_pattern.rs` - Root `lib.rs` re-exports the crate, so `openobserve::super_cluster_queue::init()` in `main.rs` is unchanged Pairs with the `Cargo.toml.openobserve` template sync PR in o2-enterprise (must merge together). ## Test plan - `cargo check --workspace --all-targets` (OSS) passes - `cargo check -p openobserve --no-default-features --features enterprise` with the synced enterprise template passes - `cargo clippy -p super_cluster_queue --features enterprise,vectorscan` — zero warnings - `cargo clippy -- -D warnings` (CI invocation) passes; `cargo fmt --all` clean |
||
|
Hengfei Yang
|
33d1da9b6e |
feat: move MCP server to OSS (#13272)
## Summary - move the MCP protocol, tool discovery, schema simplification, execution, and response filtering implementation into a new OSS `openobserve-mcp` workspace crate - enable the real MCP POST, GET, and DELETE handlers in community builds - initialize the MCP tool catalog from the OpenAPI document for every edition - keep OAuth authorization-server discovery enterprise-only because it depends on Dex - add OSS handler and OpenAPI initialization coverage Closes #13266 Design at: #13266 ## Impact Community users can connect MCP clients directly to OpenObserve and use the OpenAPI-backed tool catalog. Enterprise builds continue to use the same MCP implementation from the OSS workspace crate. Companion PR openobserve/o2-enterprise#2206 removes the old private copy and updates the enterprise workspace overlay. |
||
|
Huaijin
|
3b44da3eee |
refactor: extract shared common crate (#13252)
## Summary - introduce a `common` workspace crate for shared metadata, infrastructure helpers, and utilities - keep service-backed code in `openobserve-core` while preserving the existing `common` compatibility namespace - flatten `src/core/src/service/` into the standard `src/core/src/` module layout without using `#[path]` - retain `service::*` as compatibility re-exports from `service.rs` - move cipher request/response types from core into the search crate while preserving `openobserve_core::cipher` - remove unused Cargo dependencies from workspace crates Enterprise companion: https://github.com/openobserve/o2-enterprise/pull/2202 ## Validation - `cargo check --workspace --all-targets` - `cargo clippy -p common -p openobserve-core -p search --lib -- -D warnings` - common crate unit tests: 271 passed, 4 ignored - enterprise overlay: `cargo check -p openobserve-core --features cloud` - migrated cipher tests: 8 passed - `cargo deny check licenses` - `cargo fmt --all -- --check` - `git diff --check` |
||
    
|
4031f76426 |
feat: synthetic monitoring (#12849)
Design at : https://github.com/openobserve/designs/tree/main/synthetics ref: https://github.com/openobserve/o2-enterprise/issues/2095 --------- Co-authored-by: omkark06 <omkar@zinclabs.io> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Claude <claude@users.noreply.github.com> |
||
|
Huaijin
|
66ca3d2439 |
refactor: extract PromQL evaluator into dedicated crate (#13239)
## Summary - add a dedicated promql workspace crate for parsing and evaluation - keep cluster routing, cache, WAL, file-list, and request orchestration in openobserve-core - preserve existing service::promql call sites through a compatibility facade; the facade re-exports an explicit item list (not a glob), and the promql search orchestration in core depends on the promql crate directly - inject enterprise query cancellation through TableProvider instead of coupling the evaluator to core search state - replace the core regex-UDF dependency with DataFusion regexp_like expressions ## Companion PR - openobserve/o2-enterprise#2198 updates the enterprise workspace override for the new promql crate ## Validation - cargo fmt --all -- --check - cargo check --workspace - cargo clippy -p promql --all-targets -- -D warnings - cargo clippy -p openobserve-core - cargo test -p promql: 363 passed - verified the promql dependency tree has no openobserve-core or search dependency - cargo check -p o2_enterprise --lib in the paired private workspace ## Notes The evaluator moves atomically in this PR. The existing service::promql::search subtree remains in core because it owns OpenObserve-specific orchestration rather than PromQL evaluation. The two web/ file changes only update source-path references in comments that pointed at the moved Rust files (src/service/promql/... -> src/promql/src/...). |
||
|
Huaijin
|
f36c9bd5f1 |
refactor: move search engine into search crate (#13238)
## Summary This is PR 2 of the atomic search-crate split, following #13234. - move the complete query-engine SCC into `src/search`: DataFusion, SQL, index, Tantivy, bloom pruning, inspector, and engine utilities - move 130 Rust files / 43,935 lines as a single atomic change - keep search orchestration, result-cache policy, cluster/gRPC/streaming, partitioning, usage, and audit responsibilities in `openobserve-core` - preserve existing `openobserve_core::service::search::*` paths through ordinary facade re-exports - move direct engine dependencies and feature ownership into the `search` manifest - remove core dependencies used only by the moved engine No query behavior change is intended. Source changes inside the moved engine are limited to crate-relative import paths and formatting. ## Dependency boundary ```text openobserve / API / jobs | v openobserve-core (orchestration) | v search (query engine) | v config / infra / proto / flight / openobserve-vrl ``` - `search` does not depend on `openobserve-core` - `openobserve-vrl` remains independent of both `search` and `openobserve-core` - the old core DataFusion, SQL, and Tantivy implementation directories no longer exist ## Validation - `cargo check --workspace` - `cargo check -p openobserve --features mimalloc` - `cargo clippy -- -D warnings` - `cargo clippy -p search -p openobserve-core --lib -- -D warnings` - `cargo fmt --all -- --check` - Cargo metadata dependency-boundary checks - `cargo test -p openobserve-vrl --lib` — 7 passed - `cargo test -p search --lib` — 876 passed; 12 existing histogram static-configuration tests fail in this local environment - `cargo test -p openobserve-core --lib` — 2407 passed, 6 existing global-state/filesystem/DB-isolation tests failed, 18 ignored - filtered reruns excluding those existing environment-sensitive tests passed ## Not run - private Enterprise / Vectorscan / Vortex overlay builds - local/distributed runtime smoke tests - cold/incremental build and query runtime performance benchmarks No performance improvement is claimed by this PR until those benchmarks are completed. |
||
|
Huaijin
|
059058519e |
refactor: make search engine sources crate-independent (#13234)
## Summary - add `src/vrl` as a leaf package named `openobserve-vrl` with Rust path `openobserve_vrl` - move `QUERY_FUNCTIONS`, enrichment-table ownership, GeoIP/MaxMind table registration, VRL compiler configuration, and shared runtime helpers into `openobserve-vrl` - let ingestion, jobs, core, and search use the shared VRL integration without depending on one another - keep DataFusion UDF registration direct through `Sql.org_id`; no `QueryExecutionContext`, `PreparedQueryTransform`, or `prepare_query_transforms` - move the cipher registry, query parameters, and file-cache helpers to `search` - move the generic file downloader implementation to `infra` - add feature forwarding for the `search` crate - keep compatibility re-exports for existing API, ingestion, gRPC, jobs, and Enterprise call sites Enterprise companion: https://github.com/openobserve/o2-enterprise/pull/2196 |
||
|
Huaijin
|
d6166aac49 |
perf: speed up embedded web asset compilation (#13233)
## Summary - upgrade rust-embed-for-web from 11.3 to 11.4.1 - disable unused compile-time gzip and Brotli generation for embedded UI assets - keep browser response compression through the global tower-http CompressionLayer Related enterprise PR: https://github.com/openobserve/o2-enterprise/pull/2192 ## Benchmark Clean release-profiling build with mimalloc: - before: 21m 13s - after: 15m 45s - improvement: 5m 28s, or 25.8% Focused web crate compilation improved from 546.66s to 98.55s. ## Testing - cargo metadata --locked --no-deps --format-version 1 - cargo clean - cargo build --features mimalloc --profile release-profiling |
||
|
Huaijin
|
5dc22a0ee8 |
refactor: establish search crate boundary (#13217)
## Summary - add the minimal `openobserve-search` workspace crate and move shared search/cache types into it - move the concrete enrichment table type and cache ownership to the search crate while retaining normal compatibility re-exports - move trace schema constants to `config` so the SQL engine no longer reaches into the traces service - expose concrete enrichment read functions from `infra` and make DataFusion call them directly - construct the stateless VRL runtime directly in the transform UDF - keep search execution and orchestration in `openobserve-core` for the later physical moves - do not introduce `SearchRuntime`, provider traits, callbacks, `#[path]`, or a search-to-core dependency This combines the original crate-boundary step with the dependency cleanup needed before SQL/DataFusion files can move safely. ## Validation - `cargo test -p openobserve-search --lib` (11 passed) - `cargo test -p openobserve-core service::enrichment::tests --lib` (19 passed) - `cargo check -p openobserve --features mimalloc` - enterprise + mimalloc check using the o2-enterprise overlay - cloud + mimalloc check using the o2-enterprise overlay - `cargo fmt --all -- --check` - strict all-target clippy for `openobserve-search` and `openobserve-core` - strict library clippy for `infra` - dependency check confirms `openobserve-search` does not depend on `openobserve-core` `infra --all-targets` currently reports three pre-existing inconsistent digit-grouping warnings in `src/infra/src/table/entity/alert_dedup_state.rs`; they are unrelated to this change and are not included here. |
||
|
Huaijin
|
3559d483ae |
refactor: move web UI asset embedding into a dedicated leaf crate (#13211)
## Problem Every rebuild of `openobserve-api` re-runs the `RustEmbed` macro expansion, which embeds `web/dist` (53MB, 1035 files) and compresses every file with gzip + brotli(q11) — single-threaded, non-incremental, and it peaks at 8GB RSS. `-Ztime-passes` attributes ~540s of the 769s api compile to `expand_crate`. Net effect: touching one file in `openobserve-core` costs a **16m45s** rebuild on an M-series laptop, and neither incremental compilation nor codegen-unit tuning can help, because macro expansion happens before both. ## Fix Move the `#[derive(RustEmbed)]` into a new `src/web` leaf crate with no workspace dependencies. The embed + compression now only recompiles when `web/dist` changes; edits anywhere in the backend no longer pay for it. **Measured (same machine, default profile settings, `--profile release-profiling`):** | | before | after | |---|---|---| | `openobserve-api` compile | 769s | 231s | | touch `src/core` → binary | 16m45s | **6m44s** | ## Also in this PR - **`src/web/build.rs`**: the derive has no file tracking, so a rebuilt `web/dist` silently shipped stale embedded assets until something else dirtied the api crate. `rerun-if-changed` on the embed folder fixes that. The folder is parsed from the `#[folder]` attribute rather than hardcoded, so it stays correct when the e2e workflows rewrite it. A missing `index.html` now prints a build warning instead of silently embedding nothing (the #13197 failure mode). - **Playwright workflows**: the embed-folder `sed` now targets `src/web/src/lib.rs` and is guarded with `grep -q`, so a future path drift fails at the sed step instead of silently shipping an empty UI and failing at the login page (see openobserve/o2-enterprise#2175 for that incident). - **`deny.toml`**: allow the new crate's AGPL-3.0 license (CI runs `cargo deny check licenses`, which requires a per-crate exception). ## Validation - `cargo deny check licenses` passes - Smoke-tested the binary: `/healthz` ok, `/web/` serves `index.html`, JS assets return 200 from the embedded bundle - Enterprise overlay resolves: `cargo metadata` passes with the companion `Cargo.toml.openobserve` change ## Companion openobserve/o2-enterprise#2184 carries the overlay + e2e workflow sync on the **same branch name** (`perf/web-embed-crate`); the two should merge together. |
||
|
Huaijin
|
bcea47f644 |
refactor: extract management API crate (#13202)
## Summary - add the `openobserve-api-management` crate under `src/api_management` - physically move management handlers, models, and authentication helpers out of `openobserve-api` - preserve the existing API module paths through re-exports - move shared stream permission checking into `openobserve-core` to avoid a management-to-query dependency - allow the new crate's `AGPL-3.0` license in `deny.toml` (fixes the cargo-deny `check` job) - keep router, OpenAPI generation, and rate-limit path mapping in `openobserve-api` ## Dependency boundary `openobserve-api-management` depends on `openobserve-api-common` and `openobserve-core`; it does not depend on `openobserve-api-query`. ## Note on `split-debuginfo` Removing `split-debuginfo = "packed"` from `[profile.release-profiling]` is intentional (not a formatting artifact): profiling builds now keep DWARF debuginfo embedded in the binary, so profiling artifacts can be symbolized at runtime without shipping a separate debuginfo file. The o2-enterprise overlay `Cargo.toml.openobserve` is kept in sync. ## Companion PR - openobserve/o2-enterprise#2175 ## Validation - OSS check with `mimalloc` - OSS strict clippy with all targets - enterprise check and strict clippy - cloud check and strict clippy - `cargo test -p openobserve-api-management --lib`: 300 passed - `cargo deny check licenses` - `cargo fmt --all` - `git diff --check` Build-time benchmarking is intentionally deferred for this stage. |
||
|
Huaijin
|
cd8610cafa |
refactor: extract shared and query API crates (#13199)
## Summary
- add `openobserve-api-common` for HTTP extractors and request DTOs
shared across API domain crates
- add `openobserve-api-query` and physically move the search, PromQL,
and traces handlers (about 16.4K lines)
- preserve the existing `handler::http::request::{search,promql,traces}`
paths through re-exports so routing, Swagger generation, and rate-limit
metadata remain compatible
- forward enterprise, cloud, vectorscan, and vortex features through the
new crate boundary
- avoid `#[path]` source inclusion
## Enterprise overlay
Requires openobserve/o2-enterprise#2174, which registers the two new
workspace crates in `Cargo.toml.openobserve`.
## Validation
- `cargo check -p openobserve --features mimalloc`
- enterprise check with `--no-default-features --features
enterprise,mimalloc`
- cloud check with `--no-default-features --features cloud,mimalloc`
- OSS and cloud/enterprise all-targets Clippy with `-D warnings`
- `openobserve-api-common`: 9 tests passed
- `openobserve-api-query`: 136 independent tests passed single-threaded;
the unchanged DB-dependent saved-view test was excluded because
standalone crate tests do not initialize the infra database
- `cargo fmt --all -- --check`
Build-time benchmarking is intentionally deferred until the crate-split
series is ready for a controlled `release-profiling` comparison.
|
||
|
Hengfei Yang
|
17dbff89b3 |
feat: skip file download cache for data older than configurable max age (#13201)
Design at: #13200 ## What changed Adds a max-age admission policy for the file download cache: - New configs `ZO_DISK_CACHE_MAX_AGE_DAYS` and `ZO_MEMORY_CACHE_MAX_AGE_DAYS` (default `0` = disabled, current behavior unchanged). - `queue_download()` now skips files whose `max_ts` is older than the configured max age — they never enter the download queue; queries read them directly from object storage (the existing cache-miss fallback). - The `cache_latest_files` gRPC event handler applies the same filter when collecting files, covering the batch `download_from_node` path that writes to the cache via `file_data::set` without going through the queue (e.g. merged files produced by compaction of old partitions). - The check is exposed as `file_downloader::exceeds_cache_max_age(ts, cache_type)`; `ts <= 0` (unknown timestamp) is never skipped. ## Why `time_lru` has an eviction policy but no admission policy. With a cache sized for the recent hot window (e.g. 3 days), a query on old data (e.g. 30 days ago): 1. Triggers gc **before** insert: the first old-file insert evicts `max(release_size, data_size * 100)` — by default ~10% of the cache — from the oldest hours of the *hot* window. 2. Once inserted, the old files themselves become the oldest `time_lru` bucket and are evicted first, often within the same query — S3 download → disk write → immediate delete, pure wasted work with zero retention for repeat queries. With a max age matching the cache window, deep-history queries no longer disturb the hot cache or waste S3 bandwidth and disk IO. ## Coverage All query-side download paths funnel through `queue_download()` with the file's real `max_ts` and are covered: search (parquet), tantivy index (`.ttv`), promql/metrics, and file_list dump. Deliberately **not** gated: compactor working downloads (`cache_remote_files` and merge-output writes — required reads/writes, not speculative caching) and the results/aggregations/metrics_results caches (separate concern). ## Tests Unit tests for the age check (disabled, unknown ts, recent file, old file); `cargo test -p openobserve-core --lib file_downloader` passes. |
||
|
Huaijin
|
5f65481821 |
refactor: extract background jobs crate (#13189)
## Summary - extract background scheduling code into the src/jobs workspace crate - physically move job sources without path-based module indirection - move file downloading, MMDB initialization signaling, and grouped-alert delivery behind core service APIs to keep the dependency direction one-way - preserve the existing root job API through re-export and forward enterprise, cloud, vectorscan, and vortex features Companion o2-enterprise PR: https://github.com/openobserve/o2-enterprise/pull/2173 ## Validation - cargo check -p openobserve --features mimalloc - cargo clippy -p openobserve -p openobserve-api -p openobserve-core -p openobserve-jobs --all-targets --features mimalloc -- -D warnings - enterprise cargo check with the o2-enterprise overlay - cloud cargo check and strict Clippy with the o2-enterprise overlay - cargo fmt --all -- --check Build-time benchmarking is intentionally deferred until the crate-splitting series is complete. |
||
|
|
b92d5765ed |
refactor: extract API frontend crate (#13181)
## Summary Second step of the crate split (after #13168 which extracted `openobserve-core`): move `src/handler` and `src/router` into a new `openobserve-api` crate at `src/api`, so the HTTP/gRPC transport frontend (routing, auth, extractors, OpenAPI generation) builds as its own compilation unit. All 155 moved source files are byte-identical moves (`git mv` only, zero content changes). The real changes are limited to manifests and scaffolding: - **`src/api/src/lib.rs`** re-exports `openobserve_core::{common, job, service}` (plus `cipher` under `enterprise`) at the crate root, so the existing `crate::common::...` / `crate::service::...` paths inside the moved files keep resolving without touching them. - **`src/lib.rs`** re-exports `openobserve_api::{handler, router}`, so external paths like `openobserve::handler::...` used by `main.rs` and `tests/integration_test.rs` are unchanged. - **Feature forwarding**: `enterprise` / `cloud` / `vectorscan` / `vortex` / `profiling` on the root crate now forward to `openobserve-api` as well. - **`src/enterprise-stubs` → `src/enterprise`**: directory rename, plus a new `o2_ratelimit` dependency-resolution stub (the api crate depends on it under the `enterprise` feature; there was no stub for it before). - `#![feature(variant_count)]` moves with its only use site (`models/dashboards.rs`) into the api crate. - **CI**: added `deny.toml` license exceptions for the two new AGPL-3.0 workspace crates (`openobserve-api`, `o2_ratelimit`); updated the `ui.rs` path patched by the playwright workflows. `src/api/Cargo.toml` is shared verbatim between OSS and enterprise builds — only the root manifest is replaced by the enterprise overlay; the `o2_*` deps resolve via `workspace = true` to either the stubs (OSS) or the real crates (enterprise). Companion PR: openobserve/o2-enterprise#2168 ## Testing - `cargo check --workspace` passes (OSS). - Verified all moved files are 0-diff renames (`git diff -M90% --stat`). - Verified every `crate::` reference in the moved code resolves through the re-exports (`service`/`common`/`handler`/`job`/`router`/`cipher`), with the single `crate::cipher` use correctly gated behind `enterprise`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
|
HuaijinandClaude Fable 5
|
81cc0f2255 |
refactor: extract application core crate (#13168)
## Summary Extract the application core (`src/common`, `src/service`, `src/job`, `src/cipher`) out of the root binary crate into a new workspace crate `src/core` (package `openobserve-core`). The root crate keeps the binary entrypoints and edges: `cli`, `handler`, `router`, `migration`, and `super_cluster_queue`. This builds on #13162 (remove service → handler dependencies), which made the extraction possible. > Naming note: the directory is `src/core` but the package stays `openobserve-core` — a Cargo dependency literally named `core` shadows Rust's built-in `core` crate and breaks the `::core::` paths emitted by derive macros (thiserror, serde, ...). ## What moved - `src/common`, `src/service`, `src/job`, `src/cipher` → `src/core/src/...` (467 files moved; 385 of them byte-identical, the rest only needed `pub(crate)` → `pub` widening and clippy fixes required by the new crate boundary) - `ua_regex/regexes.yaml` → `src/core/ua_regex/` (same content, consumer moved with it) - `src/handler/grpc/request/search/mod.rs` → `src/core/src/service/search/grpc_server.rs` (byte-identical; old path kept via `pub use ... as search`) - `impl From<MetricsQueryRequest> for cluster_rpc::MetricsQueryRequest` moved from `handler/grpc/mod.rs` into `service/promql` (orphan rule; field-for-field identical) - The `impl From<XError> for Response` blocks scattered across HTTP handler files (alerts, dashboards, reports, folders, pipeline, eval_jobs, providers, ratelimit) are consolidated into `src/core/src/service/http.rs` with unchanged status-code mappings ## Compatibility `src/lib.rs` re-exports the moved modules (`pub use openobserve_core::{common, job, service}`, and `cipher` under the `enterprise` feature), so all existing `crate::service::...` / `openobserve::common::...` paths keep working — handlers, tests, and downstream code need no changes. ## Enterprise dependency resolution `openobserve-core` optionally depends on `o2_enterprise` / `o2_dex` / `o2_openfga` / `vortex` behind features. So that the OSS repo resolves on its own, workspace deps point at empty stub packages under `src/enterprise-stubs/` (never compiled into default builds). Enterprise builds replace the workspace manifest with `Cargo.toml.openobserve`, which points these at the real crates. ⚠️ Must be merged together with openobserve/o2-enterprise#2162 (same branch name) — merging either PR alone breaks the enterprise build. ## Feature forwarding Root `enterprise` / `cloud` / `vectorscan` / `vortex` features forward to the corresponding `openobserve-core` features; the core crate forwards to `config` / `infra` / `o2_enterprise` exactly as the root crate did before. ## Unit-test fix The `service::sourcemaps` tests relied on the workspace root being the test cwd (shared zip fixture via a relative path, sqlite db dir/tables created by `init-db` from `coverage.sh`). Unit tests of a sub-crate run with the crate dir as cwd, so they now read the fixture via `CARGO_MANIFEST_DIR` and create the db dir + the tables they touch themselves; `src/core/data` is cleaned between the postgres and sqlite CI passes and gitignored, mirroring `src/infra/data`. ## Cleanup The extracted crate's manifest is pruned to what core actually uses (45 dependency entries copied from the root crate plus a dead `profiling` feature removed), and the stale `roaring` dependency (unused since the tantivy result cache rewrite in #7890) is dropped from the root manifest. All new files carry the standard AGPL header. ## Testing - `cargo check --workspace --all-targets`, `cargo clippy -D warnings`, and `cargo fmt --check` pass - `service::sourcemaps` unit tests pass under `--test-threads=1` (the CI setting; they share one db group and are not parallel-safe, same as before) - No logic changes anywhere: the diff is moves + visibility widening + clippy appeasement; error-to-HTTP-response mappings verified variant-by-variant - e2e label kept so Playwright runs --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |