## What this is
Database Monitoring: an aggregation and presentation layer over
telemetry OpenObserve already ingests. It reads three vantages — client
spans, server-vantage logs, and collector metrics — and turns them into
per-statement, per-instance and per-database views.
No new storage engine and no new ingest path. Statements are normalized
and rolled up into a stats stream, and the read API queries that stream
through the existing search path.
## Shape of the change
| | |
|---|---|
| Read endpoints | 17 |
| UI pages | 8 (plus a shell) |
| Commits | 141 |
**Backend** — `src/core/src/db_monitoring/` holds the non-HTTP half
(normalization, rollup, server-vantage parsing). The read API lives in
`src/api/management/src/request/db_monitoring/` as a handler / models /
service trio; the service layer is split one file per feature.
**Frontend** — `web/src/views/DatabaseMonitoring/`, with shared logic in
`web/src/composables/dbm/`.
## Feature availability
Databases, queries, query detail and history, endpoints, samples,
activity, insights, plans, server metrics, server queries, server
samples, instances, instance metrics, badges — is available in OSS.
## Reviewing this
It is a large diff, and most of it is additive: 327 files, and the two
largest non-test files are the shared service module and the deadlocks
service. Suggested order:
1. `src/api/management/src/request/db_monitoring/handler.rs` — the 17
entry points and their gating
2. `src/api/management/src/request/db_monitoring/service/common.rs` —
auth, SQL building, the shared read path
3. `src/core/src/db_monitoring/rollup.rs` — how the stats stream is
produced
4. `web/src/views/DatabaseMonitoring/` — the pages
Time semantics are the subtlest part. Rollup rows are stamped at the
window **end**, so reads are exclusive at both edges; `stats_read_range`
carries the reasoning and a test pins it.
## Known follow-ups
- **The server-vantage stream named as `_o2_dbm_server`.**
## Testing
- Rust: unit tests across the DBM core and read service, including
SQL-building, auth-gating and normalizer cases
- Web: unit tests across the DBM pages and composables
- Playwright: DBM suite covering the pages and cross-page navigation
Verified before opening: OSS and enterprise builds both compile
warning-free, `cargo fmt` clean, `vue-tsc` clean, and the full web unit
suite passes (44,704 tests).
The query half of the TSID-major layout: #13882 wrote `tsid-major-v3-*`
files ordered by `(__hash__, _timestamp)`; this PR adds the `.midx`
series index next to them and makes PromQL prune through it. Supersedes
the remaining scope of #13839.
## Write side
- Every `tsid-major-v3-*` file written by the hour-end merge gets a
`.midx` series index (Arrow IPC + zstd), stored like the Tantivy index
under a derived root distinct from the data tree:
`files/{org}/metrics/{stream}/{date}/{hour}/tsid-major-v3-{id}.parquet`
→ `files/{org}/midx/{stream}/{date}/{hour}/tsid-major-v3-{id}.midx`
(Tantivy: `files/{org}/index/{stream}_{type}/…/{id}.ttv`): **one row per
contiguous TSID run** — its `row_count: u32` plus the label columns of
the run's first row (source types preserved). Runs tile the data file,
so each run's starting row is the prefix sum of the preceding counts and
is not stored; `__hash__` is excluded (matchers never reference it). The
whole index is one ZSTD-compressed IPC batch. Runs crossing an input
RecordBatch boundary appear as two adjacent entries; query-time range
coalescing makes that representation equivalent without buffering
samples. File and row-group boundaries stay independent of TSID
boundaries.
- `TsidSeriesIndexWriter` lives inside
`search/datafusion/merge/tsid_major.rs` and collects run entries
alongside the Parquet writer (one buffered row per run), encoding and
compressing once at finish. `MergedFile` carries `series_index:
Option<Vec<u8>>`; the compactor uploads it to the `midx/` tree
(compliance mode included).
- `.midx` is intentionally **not** tracked in `file_list`; its path is
derived from the data file's key
(`MetricsFileLayout::series_index_path`) and it is deleted together with
the parent in the delayed-deletion pass (`deleted.rs`); like the
`index/` tree, the `midx/` day directories are covered by retention's
local-disk deletion and the gc tool.
## Query side (PromQL)
- `promql_service::…::tsid_series_index::search` runs where Tantivy used
to, before the metrics table is registered: for each TSID-major file it
loads the `.midx` (concurrent, `2×target_partitions ≤ 64`), projects the
row-range columns plus the matched labels **by name against each series
index's own schema** (label sets / column orders differ between files),
builds the matcher filter **per series index** with
`physical_expr::create_physical_expr` — the exact same predicates as
`apply_matchers` (`matcher_predicates`) — evaluates it over the run
rows, coalesces adjacent/overlapping ranges and attaches them as
`FileSelection::RowRanges` (converted into a Parquet access plan by the
generic helper from #13861, no per-row bitmap).
- **Applied per file**: files of any other layout in the same query
(still-open hours, legacy data, late arrivals) are left in place and
scanned in full. A matched label missing from a series index
over-selects that file; a query whose matchers cannot be evaluated at
all selects the whole file; any series index failure logs and falls back
to a full scan. The final PromQL matchers are always applied, so results
are exact in every case.
- Evaluated selections are cached (LRU, 256 MiB) keyed by account +
series index path + the full matcher set.
- The Tantivy path in the PromQL storage scan is removed
(`convert_matchers_to_index_condition`, index-field plumbing,
`is_add_filter_back` bookkeeping). SQL search is untouched.
## Benchmark (from the #13839 POC, same layout + series index)
478,144,896 rows, 7 CPUs, one-hour PromQL range, step 15s, result cache
off, release + mimalloc:
| Layout | Filter steady median | No-filter steady median | Data files |
Total storage |
|---|---:|---:|---:|---:|
| main, 2 GiB | 695.709 ms | 4.638 s | 88 | 4,925.81 MiB |
| TSID-major + `.midx` | 269.496 ms | 2.793 s | 80 | 1,294.61 MiB |
## Validation
- `cargo fmt --check`; `cargo check` / `clippy` on config, search,
compaction, promql-service, jobs, search_service (`--tests`) clean
- `config tsid_layout` 3 passed (incl. `series_index_path`); `search
tsid_major mode:: merge::` 10 passed (rotation test now also asserts the
series index, exact-run and coalescing series index tests); `compaction
merge deleted` 33 passed; `promql-service` 20 passed + 1 pre-existing
wall-clock flake (`cache::test_promql_cache_set_and_get`, fails
intermittently on main too)
- End-to-end on the #13839 branch (same code paths): hour-end merge
writes N `tsid-major-v3-*` + `.midx`; PromQL takes the series index path
(`selected 39 ranges across 4/4 files`), counts/sums match the synthetic
data, cold-cache identical after restart; a mixed hour (3 major + 1 late
`tsid-sorted`) reports `1 files without a series index left to
tantivy/full scan` with unchanged results; delayed deletion removes
`.parquet` **and** `.midx`
## Summary
- rename the existing TSID-major config, modules, types, merge modes,
and persisted file prefixes to indexed-metrics terminology
- add the promql-service dependency declarations needed by the follow-up
metrics-index work
- carry the exact Cargo.lock content from #13899, including h2 0.4.16
and the promql-service dependency entries
## Scope
This is no-new-feature groundwork split from #13899. It retains the
existing metrics layout mechanics under the new names and does not add
.midx sidecar writing, index lookup/query pruning, or index lifecycle
handling.
The old environment variable and file prefixes are intentionally not
kept as compatibility aliases because this experimental path has not
been used in a deployment.
## Validation
- cargo fmt --all -- --check
- cargo metadata --locked --no-deps --format-version 1
- cargo check -p promql-service
- cargo test -p config --lib -- layout (5 passed)
- cargo test -p search --lib -- datafusion::merge (8 passed)
- cargo test -p compaction --lib -- merge (32 passed)
- cargo check -p openobserve-jobs
The storage-layout half of #13839, without the `.sidx` series index and
its PromQL query path (those stay in #13839, which will rebase on this).
## What
Behind `ZO_METRICS_TSID_MAJOR_ENABLED` (default off), Parquet metrics
files of a stream whose `__hash__` column is `UInt64` are written
ordered by `(__hash__, _timestamp)` so every series is contiguous:
- **`config::meta::promql::tsid_layout`** —
`metrics_tsid_major_enabled(stream_type)` (flag + configured Parquet),
`metrics_tsid_major_stream(stream_type, schema)` (also `__hash__:
UInt64`; JSON-ingested metrics store `__hash__` as a string and keep the
classic layout), and `MetricsFileLayout { Legacy, HashSorted, TsidMajor
}` encoded in the file-name prefix (`tsid-sorted-*`, `tsid-major-v3-*`)
so readers and later merges know the physical order without opening the
file.
- **`MergeMode::HashSorted`** (open hour): the ingester movers and
incremental compactor merges write one hash-ordered file named
`tsid-sorted-*`. In pack mode the single-segment upload-as-is fast path
is skipped for such streams (segments are in arrival order).
- **`MergeMode::TsidMajor`** (closed hour): `merges_whole_batch()` —
every file of the hour, regardless of size — into size-split
`tsid-major-v3-*` files written at RecordBatch boundaries using
`ZO_COMPACT_MAX_FILE_SIZE` as logical original size
(`search/datafusion/merge/tsid_major.rs`). `MergeMode::for_compactor /
for_ingester` take the schema and pick the mode via
`metrics_tsid_major_stream`; downsampling keeps precedence. The
compactor skips a closed hour that is already all `tsid-major-v3-*`.
- **`MergedFile.layout`** decides the output file name
(`layout.file_name` / `mark_file_key`) in the compactor and the ingester
movers.
- **Merging pre-sorted inputs**: `MergeMode::input_sort_order(&files)`
declares `FileSortOrder::HashTimestampAsc` when every input is
hash-ordered (`tsid-sorted-*` / `tsid-major-v3-*`). Overlapping hash
ranges make DataFusion drop the order for its size-based groups; the
listing adapter regroups one file per partition (single-file groups are
trivially ordered) and keeps the declared `output_ordering`, so
`EnforceSorting` turns the `SortExec` into a `SortPreservingMergeExec`
over N pre-sorted streams (memory N × batch instead of the whole hour).
A batch that mixes layouts declares no order and falls back to the full
sort.
Two file states only: `tsid-sorted-*` (ordered, not finalized) and
`tsid-major-v3-*` (finalized). Nothing changes for streams without the
flag, non-Parquet metrics, or a non-UInt64 `__hash__`.
## Known limitation
The SQL query path still derives `sorted_by_time` from the SQL shape
only, so with the flag on a `SELECT … FROM <metrics> ORDER BY _timestamp
DESC LIMIT n` may assume a timestamp order that hash-sorted files do not
have. PromQL is unaffected (no ordering declared). To be addressed with
the query-side work.
## Validation
- `cargo fmt --check`; `cargo check` / `cargo clippy` on config, search,
compaction, jobs, promql-service, search_service (`--tests`) clean
- `cargo test -p config --lib -- tsid_layout` 2 passed; `cargo test -p
search --lib -- merge:: mode:: tsid_major listing_adapter` 9 passed
(mode selection, per-layout input order, hash `ORDER BY`, size-split
rotation, SPM plan shape); `cargo test -p compaction --lib -- merge` 31
passed
- End-to-end on #13839 (debug build, local disk, OTLP gauge 300 series):
ingester uploads `tsid-sorted-*` verified `(__hash__, _timestamp)`
ordered; hour-end `merge [tsid_major] input sort order:
hash_timestamp_asc` → plan `SortPreservingMergeExec` ← `DataSourceExec{N
single-file groups}` (no `SortExec`) → `tsid-major-v3-*`; incremental
`merge [hash_sorted]` 2 → 1 `tsid-sorted-*`; JSON-ingested stream (Utf8
`__hash__`) keeps the classic layout with no compactor errors.
## TODO (follow-up, not in this PR)
**Sort TSID-major streams by `(__hash__, _timestamp)` already at WAL
dump time.** `Partition::persist` full-sorts every dumped chunk anyway
(today by `_timestamp DESC` in `merge_record_batches`); switching the
sort key for TSID-major streams is near-free at dump time and removes
the biggest ingester-side cost of this layout — the full `SortExec` over
the merge input:
1. **persist**: lexsort the chunk by `(__hash__ ASC, _timestamp ASC)`
(same gate as the writers, `metrics_tsid_major_stream`) and stamp a
`sort_order: hash_timestamp_asc` key-value into the Parquet footer. A
file-name marker is not an option for WAL files —
`parse_time_range_from_filename` splits on `.` and a prefix would break
WAL time pruning.
2. **ingester mover**: it already reads every input's footer for the
schema; declare `HashTimestampAsc` on the table only when *all* inputs
carry the footer marker (else `FileSortOrder::None`, full sort — handles
mixed files during rolling upgrades / flag flips, same role
`MetricsFileLayout::of` plays for the compactor). The merge then becomes
a `SortPreservingMergeExec` over pre-sorted files instead of a full
sort.
3. **pack, single segment**: the as-is fast path (skipped for TSID-major
streams in this PR because segments were time-sorted) can come back —
the segment is already hash-sorted, upload it unchanged and mark it
`tsid-sorted-*`.
4. **pack, multiple segments**: one `MemTable` partition per segment
with a declared sort order, so the in-memory merge is also an SPM
instead of a full sort.
Query-side is already safe for this change: metrics SQL never declares
`sorted_by_time` and PromQL declares no order, so hash-sorted WAL files
cannot mislead a reader. The cost is slightly coarser time clustering
inside WAL files, negligible at WAL file sizes. Steps 1–2 carry most of
the win; 3–4 are incremental.
Behavior-neutral refactors split out of #13839 so that PR only carries
the TSID-major feature. Nothing here changes what is written, read or
deleted; every commit is independently reviewable.
## Commits
- `cba75d244b` **promql**: `apply_matchers` split into
`matcher_predicates(schema, matchers) -> Vec<Expr>` + `apply_matchers`
(unchanged behavior; lets other inputs evaluate the same matcher
semantics).
- `b5ff6fed21` **parquet**: `new_parquet_writer` / `append_metadata`
generic over `W: AsyncFileWriter` instead of `&mut Vec<u8>`; existing
callers unchanged.
- `b07d84216a` **merge**: `MergeParquetResult::{Single, Multiple}` →
`MergeParquetResult { files: Vec<MergedFile { buf, meta }>, file_format
}` + `into_single()`. The compactor's two copies of the upload / index /
push sequence become one loop (`compressed_size` is set from the buffer
for every file, as the `Multiple` arm already did); downsampling maps
its `(bufs, metas)` into it; ingester movers use `into_single()` instead
of panicking/erroring on `Multiple`.
- `014b6c5721` **compactor**: `deleted.rs` `derived_files()` /
`delete_from_storage()` replace three copies of collect / `storage::del`
/ ignore-not-found (data, `.ttv`, flattened).
- `7163786362` **search**: `generate_parquet_access_plan_from_ranges`
extracted from the bitmap path (`set_slices` → ranges); an out-of-range,
empty or unsorted range is rejected with `assert!` (`56de386522`; a
producer that violates the sorted/non-overlapping contract is a bug, not
something to degrade around) instead of the previous `unreachable!()` on
the bitmap path. Adds `FileSelection::RowRanges(Arc<Vec<Range<usize>>>)`
as the matching selection kind (no producer yet; Parquet only, Vortex
full scan like `RowGroups`).
- `754014ccf3` **test**: `ListingTableAdapter` plan-shape test for
`FileSortOrder::HashTimestampAsc` (from #13853): three overlapping
hash-sorted files → `SortPreservingMergeExec` over three single-file
groups, no `SortExec`, globally ordered rows. `tempfile` as `search`
dev-dependency.
## Validation
- `cargo fmt --check`
- `cargo clippy -p config -p promql -p search -p compaction -p
openobserve-jobs -p search_service -p promql-service --tests` clean
- `cargo test -p promql --lib -- utils` 10, `-p search --lib -- merge::
helpers listing_adapter` 14, `-p compaction --lib -- merge deleted` 31,
`-p config --lib -- test_file_key` 4 passed
- `merge/downsampling.rs` is enterprise-only and was updated by hand;
needs the enterprise CI compile.
## Summary
Restores the process-wide PromQL series label cache that was added in
#13109 and removed in #13148, ported onto the partitioned label-loading
pipeline introduced by #13148.
Labels are immutable per series hash, so a warm cache lets repeated
queries skip the second (label-column) scan entirely — the biggest wins
are on `by()`-grouped aggregations, where the label phase is a fixed
cost that the cache removes.
## Changes
- **`src/promql/src/label_cache.rs`** (ported from #13109): sharded LRU
keyed by `(context fingerprint, series hash)`, bounded by **memory
size** — `ZO_METRICS_LABEL_CACHE_MAX_SIZE` in MB, `0` (default) = auto:
5% of total memory clamped to [100 MB, 1 GiB].
- **Reads are always served; admission gates writes only.** A lookup
cannot grow the cache, so even a query too big to admit benefits from
entries other queries warmed. Admission is sized by the query's actual
**misses** (the only entries it could insert) and bypasses **writes**
when they exceed 50% of the budget, so one oversized query cannot evict
the shared cache. Hit/miss metrics therefore reflect real lookup
outcomes.
- **`load_series_labels`** attaches cached labels partition-parallel
(rayon) before the scan and collects per-partition missing hashes:
- when every series hits, the DataFusion label scan is **skipped
entirely**
- the miss scan is narrowed to missing series: a hash in-list filter for
up to 8192 misses, plus a per-missing-series max-timestamp filter (falls
back to the sample scan's timestamp set when everything misses)
- Partition tasks only attach/cache series in their missing set;
extracted label sets are stored back into the cache, sharing the label
interner's `Arc<Label>` values (hot label values are deduped inside the
cache too).
- Cached entries exclude the synthetic `__hash__` label; it is prepended
per query, so raw-data and regular queries share entries.
- Restores the `query_metrics_label_cache_hit_count` /
`query_metrics_label_cache_miss_count` metrics (per organization).
## Performance
Re-validated on the current partitioned pipeline: warm queries are
**~20% faster than main** on the high-cardinality benchmark dataset
below.
The detailed tables are from #13109 (same cache design and dataset,
measured on the pre-#13148 pipeline): single node (M-series Mac, 32GB),
1.34B rows / 3.17M series of a histogram bucket stream, result cache
disabled, warm = median of 3 runs after first.
**What the cache itself contributes** (cache on vs off on the same
binary, `ZO_METRICS_LABEL_CACHE_MAX_SIZE=1` as the off-switch):
| Query | cache on | cache off | cache contribution |
|---|---|---|---|
| `sum by (region) (rate(m[5m]))` | **3.62s** | 5.40s | **1.49x** |
| `histogram_quantile(0.9, sum by (le) (rate(m[5m])))` | **3.53s** |
4.93s | **1.40x** |
| `count by (path) (rate(m[5m]))` | **3.73s** | 5.56s | **1.49x** |
| no-modifier aggregations (skip path), `topk` (write-bypassed), small
filters | — | — | none (by design) |
The cache is precisely a **`by()`-grouped-query optimization**: 1.4–1.5x
there, zero effect (and zero overhead) everywhere else. Cold runs with
the cache are on par with cache-off (`sum by (region)` cold: 5.33s
cached vs 5.30s uncached).
**Admission control** (`topk(5, rate(m[5m]))` needs full labels for all
3.17M series ≈ 3GB working set, past the 1 GiB auto budget):
| Configuration | time |
|---|---|
| cache disabled (baseline behavior) | 10.4s |
| cache without admission, 1 GiB budget (thrash) | 12.0s ⚠️ |
| **cache with admission, 1 GiB budget (bypasses)** | **10.6s — parity
restored** |
| cache with `ZO_METRICS_LABEL_CACHE_MAX_SIZE=8192` (fits, admitted) |
**4.5s — 2.3x faster** |
The thrash row is what admission prevents — and it was caused by insert
+ eviction churn, which is why the current revision gates only writes
and keeps reads always on.
Notes on reproducing: this dataset is deliberately extreme, so its
`by()` working sets (~0.6–0.9GB) exceed the default budget's 50%
single-query share — set `ZO_METRICS_LABEL_CACHE_MAX_SIZE=2048` or
higher to admit them. At typical production cardinality (≤1M series),
`by()` working sets are admitted at defaults. Within a single query,
selectors share the cache (the second operand of a binary op loads
labels in 315ms vs the 2.57s the first paid).
## Operational notes
Re-introduces `ZO_METRICS_LABEL_CACHE_MAX_SIZE` and the two cache
metrics removed in #13148. Sizing guide: a typical series (~10 labels)
costs ~1KB, so the 1 GiB auto cap holds roughly 1M series; column-pruned
entries from `by()` queries are far smaller (~100B), so grouped queries
fit ~10M series. A tiny budget (e.g. `1`) acts as a de-facto off switch
for writes.
## Testing
- 4 label-cache unit tests (get/put, byte-bounded eviction, admission
control, fingerprint) ported from #13109
- New tests: partial-miss scan attaches only missing series and
populates the cache; cached entries exclude `__hash__`; cached-attach
hit/miss partitioning
- All 374 `promql` crate tests pass; full workspace `cargo check` and
clippy clean
- Output verified numerically identical (#13109 methodology, max
relative diff 1e-13 from float summation order)
## Summary
- replace the `ZO_METRICS_INLIST_FILTER_ENABLED` switch with automatic
timestamp filter selection
- use an exact `IN` filter for up to 1024 timestamps and a bounded
`BETWEEN` predicate above that limit
- sort exact timestamp literals for stable query plans
- add boundary coverage for empty, threshold, and over-threshold
timestamp sets
## Rationale
Large timestamp `IN` lists add planning and evaluation overhead, while
always using `BETWEEN` can admit many unrelated label rows for sparse
timestamp sets. A bounded exact-list strategy keeps the selective path
for manageable sets without allowing query plans to grow indefinitely.
Release benchmarks used 4.2M rows, five label columns, four hash
partitions, and repeated median measurements:
- with a representative sparse range returning about 262K rows through
`BETWEEN`, 1024 timestamps favored `IN` (9.1 ms vs 10.1 ms), while 2048
favored `BETWEEN` (11.7 ms vs 10.2 ms)
- when unique maximum timestamps equaled the series count and both
predicates returned exactly one row per series, `BETWEEN` consistently
won and the gap grew with list size, confirming the need for an upper
bound
## Testing
- `cargo test -p promql series_labels::tests`
- `cargo clippy -p promql --tests --no-deps -- -D warnings`
- `git diff origin/main...HEAD --check`
## Operational note
`ZO_METRICS_INLIST_FILTER_ENABLED` is removed and no longer needs to be
configured.
## Summary
- enable tower-http zstd request decompression
- add a router middleware regression test using a real zstd-compressed
JSON payload
- update decompression documentation to list zstd support
## Root cause
The Axum router uses `RequestDecompressionLayer`, but the workspace did
not enable tower-http's `decompression-zstd` feature. Requests from
clients such as Vector configured with `compression: "zstd"` were
therefore rejected with `415 Unsupported Media Type` before reaching the
ingestion handler.
## User impact
Vector HTTP sinks can continue using zstd compression when sending
batches to OpenObserve ingestion endpoints.
## Validation
- `cargo fmt --all -- --check`
- `cargo test -p openobserve-api-http
handler::http::router::decompression::tests`
- `git diff --check`
Fixes#13726
Design at: #13702 / inline in PR description
Add the `zstd` feature to tonic and enable `accept_compressed(Zstd)` on
the OTLP logs, metrics and trace gRPC services in both `run_common` and
run_router`.
Closes#13702
## Out of scope
- Response compression on `send_compressed(Zstd)`
- OTLP/HTTP path (tower-http zstd features)
- snappy would be the better choice under CPU pressure, but tonic has no
snappy support.
## Summary
- add ZO_FEATURE_METRICS_PUSHDOWN_FILTER_ENABLED as an opt-in filter
pushdown switch for metrics queries
- group DataFusion and Tantivy feature configuration under the search
config struct
- move inverted-index result cache sizing, footer caches, skip
threshold, TopN group limit, and bloom footer cache settings from limit
to search
- preserve the moved cache settings' environment variable names and
defaults
- apply the metrics-specific setting to object storage, WAL, and Flight
query contexts
## Testing
- cargo fmt --all -- --check
- cargo check for config, infra, search, tantivy_utils, search_service,
and promql-service
- cargo test -p config every_env_config_default_parses --lib
- cargo test -p config inverted_index_config --lib
- cargo test -p search test_create_session_config --lib
- cargo clippy for all affected crates with -D warnings
## Summary
Two coupled alerting features:
**1. Notification Templates v2 ("content" templates)**
- New `kind: content` template type: a structured spec (title, body
markdown, fields, links, rows, per-channel title overrides,
severity-filtered visibility) rendered natively per channel — existing
`custom` (raw body) templates are untouched and byte-identical on the
send path.
- Per-channel renderers: Slack Block Kit, Teams (Adaptive Card +
MessageCard), Discord embeds, PagerDuty Events v2, Opsgenie, ServiceNow,
SNS, HTML/text email, and a stable machine `webhook` envelope.
- Escaping is renderer-owned (mrkdwn entities, HTML encoding, markdown
stripping) with a shared URL-scheme allowlist; golden-fixture tests per
channel.
- Template editor UI: guided content form (fields/links row tables,
compact rows config, channel title overrides), live preview panel with
per-channel preview + payload tab, test-send (rate-limited,
`[TEST]`-marked).
- Migration `m20260802_000001_add_template_kind`; prebuilt default
template + org backfill.
**2. Alert chart images (stateless, storage-free)**
- Alerts can attach a metric-history chart PNG rendered from the
triggers self-reporting stream (`actual_value` per evaluation), with
warn/crit threshold lines and the firing point appended.
- Zero storage: the chart URL *is* the data — deflate+base64url payload
signed with truncated HMAC-SHA256 (org-bound, expiry embedded), rendered
on fetch by the HTTP-terminating node (`GET
/api/v2/{org}/alerts/charts/render`). Signature verified before inflate
(decompression-bomb guard), uniform 404s, per-node render semaphore +
per-minute query budget + small LRU.
- Renderer: plotters + embedded DejaVu Sans (Bitstream Vera license
included), gap-broken line segments so sparse evaluation history doesn't
draw false plateaus.
- Channel delivery: Slack image block, email inline CID (no remote
fetch), Discord multipart attachment, Teams/PagerDuty image URLs;
per-destination opt-in via template chart flag.
- Slack layout live-A/B'd on a real channel: content + buttons/chart
split across two same-colored attachments so the severity stripe spans
everything while staying under Slack's "Show more" fold; clickable link
title; "View in OpenObserve" keeps a reserved primary-button slot.
- Slack image-delivery fallback: Slack validates image URLs server-side
and rejects the ENTIRE message with `400 invalid_attachments` when its
proxy can't fetch the chart (VPN-only/private `ZO_WEB_URL`) —
live-verified. On that rejection the send strips the image blocks and
resends once (alert always delivers), sets a process-wide flag so later
sends pre-strip (reachability is deployment-wide, not per-destination),
and re-probes hourly. Zero config; observable via `slack_image_rejected`
/ `slack_image_pre_stripped` metric events.
## Testing
- 147 notification unit tests (resolve/render/preview/test-send incl.
per-channel goldens and hostile-input/injection coverage), 8 chart
payload tests, renderer PNG tests.
- Live-verified end to end: Slack (Slackbot chart fetches ~100ms,
cache-immutable semantics), SES email inline image, multi-severity
firing scenario (warning→critical) with correct threshold lines.
- FE: AddTemplate/ContentTemplateForm/preview specs; UX audit pass with
fixes.
- `cargo clippy -D warnings` clean in both OSS and enterprise
arrangements.
## Enterprise pairing
Must merge together with openobserve/o2-enterprise#2353 (workspace
twin-manifest mirror for the new OSS deps).
## Summary
- move streaming aggregation cache, aggregate execution, and optimizer
code into the OSS search crate
- move cache aggregation SQL analysis into the SQL visitor and enable
streaming aggregation in search_service without enterprise gating
- move MetadataCountExec into OSS and use it for metadata count
optimization in all builds
- remove migration-only dead code while keeping the cache module under
search
Companion cleanup: openobserve/o2-enterprise#2327.
## Testing
- cargo fmt --all -- --check
- cargo test -p search metadata_count --lib
- cargo test -p search aggregate_optimize_rewrite --lib
- cargo test -p search streaming_aggregate --lib
- cargo test -p search cache::streaming_agg --lib
- cargo test -p search datafusion::optimizer::stream_aggregate --lib
- cargo test -p search_service partition --lib
- cargo check -p search_service
Bumps the `openobserve/vortex` pin from `57ffbf2b` to `1cb48912` in the
workspace manifest.
## Upstream API changes handled
- Arrow interop moved into a dedicated `vortex-arrow` crate, surfaced as
`vortex::arrow`. `vortex::array::arrow::*` and `vortex::dtype::arrow::*`
no longer exist, so `FromArrowArray`, `FromArrowType`, `ToArrowType` and
`ArrowSessionExt` are now imported from `vortex::arrow`.
- `ToArrowType` is no longer in scope implicitly; added the explicit
import where `DType::to_arrow_schema()` is called.
- The `vortex::compressor` facade narrowed to `BtrBlocksCompressor`,
`BtrBlocksCompressorBuilder`, `Scheme`, `SchemeId`. `SchemeExt` and the
scheme list are no longer re-exported there, so `search` takes a direct
`vortex-btrblocks` dependency (same git rev) for `SchemeExt` /
`schemes::integer::IntDictScheme`, which the UTF8 compressor uses to
exclude the int-dict scheme.
No behavior change intended — the compressor configuration and write
strategies are identical.
## Verification
- `cargo check --workspace --all-targets` clean
- `cargo fmt --all -- --check` clean
- `cargo test -p search --lib datafusion::vortex` — 6 passed
- `cargo test -p tantivy_utils --lib vortex` — 3 passed
- `cargo test -p config --lib utils::parquet` — 21 passed
Paired enterprise PR (same branch name):
https://github.com/openobserve/o2-enterprise/pull/2303
Two changes that cut build time, both measured on the same machine (M2
Max,
12 cores) with `cargo build --features mimalloc --profile release`.
## 1. The `web` crate no longer depends on `config`
`src/web` is a leaf crate so the RustEmbed expansion over `web/dist`
only
re-runs when the assets change — that expansion is 149.8s, of which
144.1s is
macro expansion (the crate already sets `#[br = false]` / `#[gzip =
false]`, so
this is not compression).
It also depended on `config`, for exactly one value:
`cfg.common.base_uri`. That
put a 50MB embed downstream of a 724-crate subgraph, so **every `config`
edit
paid the 148.5s embed again** — which is precisely what the crate's own
doc
comment says it exists to avoid.
`ui_routes()` now takes the `<base href>` value from the caller and
`base_href_middleware` carries it as middleware state.
`create_app_router()`
already computed that exact string (`"{base_uri}/web/"`) for its own
redirect,
so it is now resolved once and passed in rather than recomputed per
request
from a second `get_config()` call. `base_uri` is env-derived, normalised
once at
config init and never touched by `refresh_config`, so hoisting it out of
the
request path cannot drift.
The web crate's dependency tree goes **724 → 92 crates**, and `cargo
tree -i
config` no longer lists it.
## 2. Logging and tracing setup moved out of `main.rs`
`setup_logs`, `enable_tracing` and their span-processor / exporter types
compose
very large generic trees. Everything `main.rs` monomorphizes is emitted
into the
binary's own codegen units — the last unit on the critical path,
overlapping
with nothing.
`-Ztime-passes` on the binary unit (91.1s total):
```
LLVM_thinlto 39.8s
LLVM_passes 37.5s
codegen_to_LLVM_IR 5.3s
link_binary 5.3s (run_linker 3.9s)
type_check_crate 0.4s
```
The `openobserve` library does not depend on `openobserve-api-http`, so
its unit
compiles in parallel with the API crates — in the cold build it finished
23.9s
before the binary needed it. Moving the setup into `src/telemetry.rs`
(563
lines) puts that work in the slack instead of at the tail.
Two tests moved with the code and were dropped rather than relocated:
`test_setup_logs` and `test_enable_tracing_error_handling` both install
a global
tracing subscriber, and `setup_logs()` calls `.init()` unguarded, so
they can
only ever pass depending on ordering. They used to race just with each
other
inside the bin test binary; keeping them would have moved that race into
the lib
test binary next to the `cli` and `migration` tests. Neither asserted
anything —
the second wraps the call in `catch_unwind` and discards the result.
`main.rs` 1304 → 712 lines. **Binary unit 88.8s → 80.7s**; the library
unit grows
30.9s → 62.3s and is fully absorbed by the existing slack.
## 3. Unrelated: a dead-code warning in `openobserve-core`
Not a build-time change, just a warning fix noticed while checking the
above.
`COMPARE_ROLLUP_LOOKBACK_MICROS` was gated on `any(feature =
"enterprise",
test)`, but its only two readers (`fetch_arm_aggregate`,
`fetch_arm_failures`)
are `enterprise`-only and the unit tests below it exercise only the pure
aggregation helpers. So in an OSS build `cargo check -p openobserve-core
--all-targets` warned `constant COMPARE_ROLLUP_LOOKBACK_MICROS is never
used`.
Gate narrowed to `enterprise`, and the doc comment — which claimed the
opposite
— corrected. Happy to split this out if you'd rather it not ride along.
## 4. Review follow-ups
- `enable_tracing` became a public library API in §2 without a doc
comment,
unlike `setup_logs` beside it. Documented: which span processors attach
under
which env vars, why the returned provider must be held rather than
dropped,
and that it and `setup_logs` are mutually exclusive since both call
`.init()`.
- Added the `<base href>` test described under Verification below.
Not taken: the suggestion to re-add `test_setup_logs` /
`test_enable_tracing_error_handling` to `src/telemetry.rs`. Both install
a
global subscriber against `.init()`, so they cannot be made
order-independent,
and neither asserted anything — see §2.
`test_resource_key_value_creation` was
deleted for the same reason; it asserted that
`opentelemetry::KeyValue::new`
stores the key it was given.
## Measured and rejected
Recording these so nobody repeats them.
**Moving route registration into the crates that own the handlers.** 317
of the
371 routes were registered in one 622-line function in
`openobserve-api-http`.
Splitting them out to api-management (234), api-search (46),
api-pipelines (23)
and api-ingest (9) worked as designed — api-http went 145.0s → 63.0s —
but the
cold build only improved 654.4s → 637.7s. Splitting across a crate
boundary
**copies** work rather than relocating it: without LTO each crate
instantiates
the axum/tower machinery for itself, so the four receiving crates grew
222.1s
combined and total CPU rose 190.8s. It also just relocated the
bottleneck —
api-management became the new tail at 135.1s, with api-search finishing
only
6.9s behind it, so the tail is a plateau rather than a peak. Not carried
in this
branch.
| Tried | Result |
|---|---|
| sccache | 97.4% hit rate, **0** wall-clock gain; first fill +54% |
| Remove aws-lc-sys from the critical path | **0** — its `dur=72.0s` in
`--timings` is contention, it needs 15.6s alone |
| `codegen-units = 16` on the glue crates | **31.4s slower**, CPU
+297.8s — `lto` defaults to `false`, so thin-local LTO is on and its
cost scales with CGU count |
| Trim DataFusion features | `datafusion-core` depends on
`datafusion-functions` without `default-features = false`, so the
expression features cannot be turned off from the facade at all; what
remains is worth 13.8s of CPU (0.25%) |
A caveat on reading `cargo --timings`: a unit's `duration` is **not**
its
intrinsic cost — it is inflated by CPU contention when the machine is
saturated.
The aws-lc entry above is the clearest example.
## Verification
At this revision:
- `cargo check --all-targets` clean on OSS — 0 errors, 0 warnings — and
each
commit checks clean standalone, so `git bisect` works
- `cargo test -p web` passes
Carried over from the revision these changes were first written against
(the
edits since are doc comments, the `ui_routes` parameter rename, two
deleted
tests, a `Cargo.lock` revert, and the `cfg` narrowing in §3 — which is a
no-op
when `enterprise` is on):
- `cargo check` clean on the enterprise feature combination
- Cold release build succeeds
- Smoke-tested against a running binary: `/healthz` 200, UI and API
routes
resolve, unknown paths 404
The `<base href>` rewrite — the one runtime behaviour §1 touches — is
now
pinned by `base_href_is_rewritten_to_the_mount_point`, and checked end
to end
against the real embedded `index.html`: `ui_routes("/abc/web/")` serves
`<base href="/abc/web/" />`, matching what the old per-request
`format!("{base_uri}/web/")` produced for `ZO_BASE_URI=/abc`. The
committed test
drives a synthetic HTML response instead, so it still asserts something
in
builds where `web/dist` is absent. Verified by mutation: breaking the
rewrite
fails it.
Huaijin
Hengfei Yang
Ashish Kolhe
Prabhat Sharmaandgithub-actions[bot]
IluckySi
Harsh Mahajan
Pinguladora