## What this is
Database Monitoring: an aggregation and presentation layer over
telemetry OpenObserve already ingests. It reads three vantages — client
spans, server-vantage logs, and collector metrics — and turns them into
per-statement, per-instance and per-database views.
No new storage engine and no new ingest path. Statements are normalized
and rolled up into a stats stream, and the read API queries that stream
through the existing search path.
## Shape of the change
| | |
|---|---|
| Read endpoints | 17 |
| UI pages | 8 (plus a shell) |
| Commits | 141 |
**Backend** — `src/core/src/db_monitoring/` holds the non-HTTP half
(normalization, rollup, server-vantage parsing). The read API lives in
`src/api/management/src/request/db_monitoring/` as a handler / models /
service trio; the service layer is split one file per feature.
**Frontend** — `web/src/views/DatabaseMonitoring/`, with shared logic in
`web/src/composables/dbm/`.
## Feature availability
Databases, queries, query detail and history, endpoints, samples,
activity, insights, plans, server metrics, server queries, server
samples, instances, instance metrics, badges — is available in OSS.
## Reviewing this
It is a large diff, and most of it is additive: 327 files, and the two
largest non-test files are the shared service module and the deadlocks
service. Suggested order:
1. `src/api/management/src/request/db_monitoring/handler.rs` — the 17
entry points and their gating
2. `src/api/management/src/request/db_monitoring/service/common.rs` —
auth, SQL building, the shared read path
3. `src/core/src/db_monitoring/rollup.rs` — how the stats stream is
produced
4. `web/src/views/DatabaseMonitoring/` — the pages
Time semantics are the subtlest part. Rollup rows are stamped at the
window **end**, so reads are exclusive at both edges; `stats_read_range`
carries the reasoning and a test pins it.
## Known follow-ups
- **The server-vantage stream named as `_o2_dbm_server`.**
## Testing
- Rust: unit tests across the DBM core and read service, including
SQL-building, auth-gating and normalizer cases
- Web: unit tests across the DBM pages and composables
- Playwright: DBM suite covering the pages and cross-page navigation
Verified before opening: OSS and enterprise builds both compile
warning-free, `cargo fmt` clean, `vue-tsc` clean, and the full web unit
suite passes (44,704 tests).
The query half of the TSID-major layout: #13882 wrote `tsid-major-v3-*`
files ordered by `(__hash__, _timestamp)`; this PR adds the `.midx`
series index next to them and makes PromQL prune through it. Supersedes
the remaining scope of #13839.
## Write side
- Every `tsid-major-v3-*` file written by the hour-end merge gets a
`.midx` series index (Arrow IPC + zstd), stored like the Tantivy index
under a derived root distinct from the data tree:
`files/{org}/metrics/{stream}/{date}/{hour}/tsid-major-v3-{id}.parquet`
→ `files/{org}/midx/{stream}/{date}/{hour}/tsid-major-v3-{id}.midx`
(Tantivy: `files/{org}/index/{stream}_{type}/…/{id}.ttv`): **one row per
contiguous TSID run** — its `row_count: u32` plus the label columns of
the run's first row (source types preserved). Runs tile the data file,
so each run's starting row is the prefix sum of the preceding counts and
is not stored; `__hash__` is excluded (matchers never reference it). The
whole index is one ZSTD-compressed IPC batch. Runs crossing an input
RecordBatch boundary appear as two adjacent entries; query-time range
coalescing makes that representation equivalent without buffering
samples. File and row-group boundaries stay independent of TSID
boundaries.
- `TsidSeriesIndexWriter` lives inside
`search/datafusion/merge/tsid_major.rs` and collects run entries
alongside the Parquet writer (one buffered row per run), encoding and
compressing once at finish. `MergedFile` carries `series_index:
Option<Vec<u8>>`; the compactor uploads it to the `midx/` tree
(compliance mode included).
- `.midx` is intentionally **not** tracked in `file_list`; its path is
derived from the data file's key
(`MetricsFileLayout::series_index_path`) and it is deleted together with
the parent in the delayed-deletion pass (`deleted.rs`); like the
`index/` tree, the `midx/` day directories are covered by retention's
local-disk deletion and the gc tool.
## Query side (PromQL)
- `promql_service::…::tsid_series_index::search` runs where Tantivy used
to, before the metrics table is registered: for each TSID-major file it
loads the `.midx` (concurrent, `2×target_partitions ≤ 64`), projects the
row-range columns plus the matched labels **by name against each series
index's own schema** (label sets / column orders differ between files),
builds the matcher filter **per series index** with
`physical_expr::create_physical_expr` — the exact same predicates as
`apply_matchers` (`matcher_predicates`) — evaluates it over the run
rows, coalesces adjacent/overlapping ranges and attaches them as
`FileSelection::RowRanges` (converted into a Parquet access plan by the
generic helper from #13861, no per-row bitmap).
- **Applied per file**: files of any other layout in the same query
(still-open hours, legacy data, late arrivals) are left in place and
scanned in full. A matched label missing from a series index
over-selects that file; a query whose matchers cannot be evaluated at
all selects the whole file; any series index failure logs and falls back
to a full scan. The final PromQL matchers are always applied, so results
are exact in every case.
- Evaluated selections are cached (LRU, 256 MiB) keyed by account +
series index path + the full matcher set.
- The Tantivy path in the PromQL storage scan is removed
(`convert_matchers_to_index_condition`, index-field plumbing,
`is_add_filter_back` bookkeeping). SQL search is untouched.
## Benchmark (from the #13839 POC, same layout + series index)
478,144,896 rows, 7 CPUs, one-hour PromQL range, step 15s, result cache
off, release + mimalloc:
| Layout | Filter steady median | No-filter steady median | Data files |
Total storage |
|---|---:|---:|---:|---:|
| main, 2 GiB | 695.709 ms | 4.638 s | 88 | 4,925.81 MiB |
| TSID-major + `.midx` | 269.496 ms | 2.793 s | 80 | 1,294.61 MiB |
## Validation
- `cargo fmt --check`; `cargo check` / `clippy` on config, search,
compaction, promql-service, jobs, search_service (`--tests`) clean
- `config tsid_layout` 3 passed (incl. `series_index_path`); `search
tsid_major mode:: merge::` 10 passed (rotation test now also asserts the
series index, exact-run and coalescing series index tests); `compaction
merge deleted` 33 passed; `promql-service` 20 passed + 1 pre-existing
wall-clock flake (`cache::test_promql_cache_set_and_get`, fails
intermittently on main too)
- End-to-end on the #13839 branch (same code paths): hour-end merge
writes N `tsid-major-v3-*` + `.midx`; PromQL takes the series index path
(`selected 39 ranges across 4/4 files`), counts/sums match the synthetic
data, cold-cache identical after restart; a mixed hour (3 major + 1 late
`tsid-sorted`) reports `1 files without a series index left to
tantivy/full scan` with unchanged results; delayed deletion removes
`.parquet` **and** `.midx`
Ashish Kolhe
Harsh Mahajan
dependabot[bot]
HuaijinandHengfei Yang
Prabhat Sharmaandgithub-actions[bot]
IluckySi
