## What this is
Database Monitoring: an aggregation and presentation layer over
telemetry OpenObserve already ingests. It reads three vantages — client
spans, server-vantage logs, and collector metrics — and turns them into
per-statement, per-instance and per-database views.
No new storage engine and no new ingest path. Statements are normalized
and rolled up into a stats stream, and the read API queries that stream
through the existing search path.
## Shape of the change
| | |
|---|---|
| Read endpoints | 17 |
| UI pages | 8 (plus a shell) |
| Commits | 141 |
**Backend** — `src/core/src/db_monitoring/` holds the non-HTTP half
(normalization, rollup, server-vantage parsing). The read API lives in
`src/api/management/src/request/db_monitoring/` as a handler / models /
service trio; the service layer is split one file per feature.
**Frontend** — `web/src/views/DatabaseMonitoring/`, with shared logic in
`web/src/composables/dbm/`.
## Feature availability
Databases, queries, query detail and history, endpoints, samples,
activity, insights, plans, server metrics, server queries, server
samples, instances, instance metrics, badges — is available in OSS.
## Reviewing this
It is a large diff, and most of it is additive: 327 files, and the two
largest non-test files are the shared service module and the deadlocks
service. Suggested order:
1. `src/api/management/src/request/db_monitoring/handler.rs` — the 17
entry points and their gating
2. `src/api/management/src/request/db_monitoring/service/common.rs` —
auth, SQL building, the shared read path
3. `src/core/src/db_monitoring/rollup.rs` — how the stats stream is
produced
4. `web/src/views/DatabaseMonitoring/` — the pages
Time semantics are the subtlest part. Rollup rows are stamped at the
window **end**, so reads are exclusive at both edges; `stats_read_range`
carries the reasoning and a test pins it.
## Known follow-ups
- **The server-vantage stream named as `_o2_dbm_server`.**
## Testing
- Rust: unit tests across the DBM core and read service, including
SQL-building, auth-gating and normalizer cases
- Web: unit tests across the DBM pages and composables
- Playwright: DBM suite covering the pages and cross-page navigation
Verified before opening: OSS and enterprise builds both compile
warning-free, `cargo fmt` clean, `vue-tsc` clean, and the full web unit
suite passes (44,704 tests).
The query half of the TSID-major layout: #13882 wrote `tsid-major-v3-*`
files ordered by `(__hash__, _timestamp)`; this PR adds the `.midx`
series index next to them and makes PromQL prune through it. Supersedes
the remaining scope of #13839.
## Write side
- Every `tsid-major-v3-*` file written by the hour-end merge gets a
`.midx` series index (Arrow IPC + zstd), stored like the Tantivy index
under a derived root distinct from the data tree:
`files/{org}/metrics/{stream}/{date}/{hour}/tsid-major-v3-{id}.parquet`
→ `files/{org}/midx/{stream}/{date}/{hour}/tsid-major-v3-{id}.midx`
(Tantivy: `files/{org}/index/{stream}_{type}/…/{id}.ttv`): **one row per
contiguous TSID run** — its `row_count: u32` plus the label columns of
the run's first row (source types preserved). Runs tile the data file,
so each run's starting row is the prefix sum of the preceding counts and
is not stored; `__hash__` is excluded (matchers never reference it). The
whole index is one ZSTD-compressed IPC batch. Runs crossing an input
RecordBatch boundary appear as two adjacent entries; query-time range
coalescing makes that representation equivalent without buffering
samples. File and row-group boundaries stay independent of TSID
boundaries.
- `TsidSeriesIndexWriter` lives inside
`search/datafusion/merge/tsid_major.rs` and collects run entries
alongside the Parquet writer (one buffered row per run), encoding and
compressing once at finish. `MergedFile` carries `series_index:
Option<Vec<u8>>`; the compactor uploads it to the `midx/` tree
(compliance mode included).
- `.midx` is intentionally **not** tracked in `file_list`; its path is
derived from the data file's key
(`MetricsFileLayout::series_index_path`) and it is deleted together with
the parent in the delayed-deletion pass (`deleted.rs`); like the
`index/` tree, the `midx/` day directories are covered by retention's
local-disk deletion and the gc tool.
## Query side (PromQL)
- `promql_service::…::tsid_series_index::search` runs where Tantivy used
to, before the metrics table is registered: for each TSID-major file it
loads the `.midx` (concurrent, `2×target_partitions ≤ 64`), projects the
row-range columns plus the matched labels **by name against each series
index's own schema** (label sets / column orders differ between files),
builds the matcher filter **per series index** with
`physical_expr::create_physical_expr` — the exact same predicates as
`apply_matchers` (`matcher_predicates`) — evaluates it over the run
rows, coalesces adjacent/overlapping ranges and attaches them as
`FileSelection::RowRanges` (converted into a Parquet access plan by the
generic helper from #13861, no per-row bitmap).
- **Applied per file**: files of any other layout in the same query
(still-open hours, legacy data, late arrivals) are left in place and
scanned in full. A matched label missing from a series index
over-selects that file; a query whose matchers cannot be evaluated at
all selects the whole file; any series index failure logs and falls back
to a full scan. The final PromQL matchers are always applied, so results
are exact in every case.
- Evaluated selections are cached (LRU, 256 MiB) keyed by account +
series index path + the full matcher set.
- The Tantivy path in the PromQL storage scan is removed
(`convert_matchers_to_index_condition`, index-field plumbing,
`is_add_filter_back` bookkeeping). SQL search is untouched.
## Benchmark (from the #13839 POC, same layout + series index)
478,144,896 rows, 7 CPUs, one-hour PromQL range, step 15s, result cache
off, release + mimalloc:
| Layout | Filter steady median | No-filter steady median | Data files |
Total storage |
|---|---:|---:|---:|---:|
| main, 2 GiB | 695.709 ms | 4.638 s | 88 | 4,925.81 MiB |
| TSID-major + `.midx` | 269.496 ms | 2.793 s | 80 | 1,294.61 MiB |
## Validation
- `cargo fmt --check`; `cargo check` / `clippy` on config, search,
compaction, promql-service, jobs, search_service (`--tests`) clean
- `config tsid_layout` 3 passed (incl. `series_index_path`); `search
tsid_major mode:: merge::` 10 passed (rotation test now also asserts the
series index, exact-run and coalescing series index tests); `compaction
merge deleted` 33 passed; `promql-service` 20 passed + 1 pre-existing
wall-clock flake (`cache::test_promql_cache_set_and_get`, fails
intermittently on main too)
- End-to-end on the #13839 branch (same code paths): hour-end merge
writes N `tsid-major-v3-*` + `.midx`; PromQL takes the series index path
(`selected 39 ranges across 4/4 files`), counts/sums match the synthetic
data, cold-cache identical after restart; a mixed hour (3 major + 1 late
`tsid-sorted`) reports `1 files without a series index left to
tantivy/full scan` with unchanged results; delayed deletion removes
`.parquet` **and** `.midx`
## Summary
- rename the existing TSID-major config, modules, types, merge modes,
and persisted file prefixes to indexed-metrics terminology
- add the promql-service dependency declarations needed by the follow-up
metrics-index work
- carry the exact Cargo.lock content from #13899, including h2 0.4.16
and the promql-service dependency entries
## Scope
This is no-new-feature groundwork split from #13899. It retains the
existing metrics layout mechanics under the new names and does not add
.midx sidecar writing, index lookup/query pruning, or index lifecycle
handling.
The old environment variable and file prefixes are intentionally not
kept as compatibility aliases because this experimental path has not
been used in a deployment.
## Validation
- cargo fmt --all -- --check
- cargo metadata --locked --no-deps --format-version 1
- cargo check -p promql-service
- cargo test -p config --lib -- layout (5 passed)
- cargo test -p search --lib -- datafusion::merge (8 passed)
- cargo test -p compaction --lib -- merge (32 passed)
- cargo check -p openobserve-jobs
The storage-layout half of #13839, without the `.sidx` series index and
its PromQL query path (those stay in #13839, which will rebase on this).
## What
Behind `ZO_METRICS_TSID_MAJOR_ENABLED` (default off), Parquet metrics
files of a stream whose `__hash__` column is `UInt64` are written
ordered by `(__hash__, _timestamp)` so every series is contiguous:
- **`config::meta::promql::tsid_layout`** —
`metrics_tsid_major_enabled(stream_type)` (flag + configured Parquet),
`metrics_tsid_major_stream(stream_type, schema)` (also `__hash__:
UInt64`; JSON-ingested metrics store `__hash__` as a string and keep the
classic layout), and `MetricsFileLayout { Legacy, HashSorted, TsidMajor
}` encoded in the file-name prefix (`tsid-sorted-*`, `tsid-major-v3-*`)
so readers and later merges know the physical order without opening the
file.
- **`MergeMode::HashSorted`** (open hour): the ingester movers and
incremental compactor merges write one hash-ordered file named
`tsid-sorted-*`. In pack mode the single-segment upload-as-is fast path
is skipped for such streams (segments are in arrival order).
- **`MergeMode::TsidMajor`** (closed hour): `merges_whole_batch()` —
every file of the hour, regardless of size — into size-split
`tsid-major-v3-*` files written at RecordBatch boundaries using
`ZO_COMPACT_MAX_FILE_SIZE` as logical original size
(`search/datafusion/merge/tsid_major.rs`). `MergeMode::for_compactor /
for_ingester` take the schema and pick the mode via
`metrics_tsid_major_stream`; downsampling keeps precedence. The
compactor skips a closed hour that is already all `tsid-major-v3-*`.
- **`MergedFile.layout`** decides the output file name
(`layout.file_name` / `mark_file_key`) in the compactor and the ingester
movers.
- **Merging pre-sorted inputs**: `MergeMode::input_sort_order(&files)`
declares `FileSortOrder::HashTimestampAsc` when every input is
hash-ordered (`tsid-sorted-*` / `tsid-major-v3-*`). Overlapping hash
ranges make DataFusion drop the order for its size-based groups; the
listing adapter regroups one file per partition (single-file groups are
trivially ordered) and keeps the declared `output_ordering`, so
`EnforceSorting` turns the `SortExec` into a `SortPreservingMergeExec`
over N pre-sorted streams (memory N × batch instead of the whole hour).
A batch that mixes layouts declares no order and falls back to the full
sort.
Two file states only: `tsid-sorted-*` (ordered, not finalized) and
`tsid-major-v3-*` (finalized). Nothing changes for streams without the
flag, non-Parquet metrics, or a non-UInt64 `__hash__`.
## Known limitation
The SQL query path still derives `sorted_by_time` from the SQL shape
only, so with the flag on a `SELECT … FROM <metrics> ORDER BY _timestamp
DESC LIMIT n` may assume a timestamp order that hash-sorted files do not
have. PromQL is unaffected (no ordering declared). To be addressed with
the query-side work.
## Validation
- `cargo fmt --check`; `cargo check` / `cargo clippy` on config, search,
compaction, jobs, promql-service, search_service (`--tests`) clean
- `cargo test -p config --lib -- tsid_layout` 2 passed; `cargo test -p
search --lib -- merge:: mode:: tsid_major listing_adapter` 9 passed
(mode selection, per-layout input order, hash `ORDER BY`, size-split
rotation, SPM plan shape); `cargo test -p compaction --lib -- merge` 31
passed
- End-to-end on #13839 (debug build, local disk, OTLP gauge 300 series):
ingester uploads `tsid-sorted-*` verified `(__hash__, _timestamp)`
ordered; hour-end `merge [tsid_major] input sort order:
hash_timestamp_asc` → plan `SortPreservingMergeExec` ← `DataSourceExec{N
single-file groups}` (no `SortExec`) → `tsid-major-v3-*`; incremental
`merge [hash_sorted]` 2 → 1 `tsid-sorted-*`; JSON-ingested stream (Utf8
`__hash__`) keeps the classic layout with no compactor errors.
## TODO (follow-up, not in this PR)
**Sort TSID-major streams by `(__hash__, _timestamp)` already at WAL
dump time.** `Partition::persist` full-sorts every dumped chunk anyway
(today by `_timestamp DESC` in `merge_record_batches`); switching the
sort key for TSID-major streams is near-free at dump time and removes
the biggest ingester-side cost of this layout — the full `SortExec` over
the merge input:
1. **persist**: lexsort the chunk by `(__hash__ ASC, _timestamp ASC)`
(same gate as the writers, `metrics_tsid_major_stream`) and stamp a
`sort_order: hash_timestamp_asc` key-value into the Parquet footer. A
file-name marker is not an option for WAL files —
`parse_time_range_from_filename` splits on `.` and a prefix would break
WAL time pruning.
2. **ingester mover**: it already reads every input's footer for the
schema; declare `HashTimestampAsc` on the table only when *all* inputs
carry the footer marker (else `FileSortOrder::None`, full sort — handles
mixed files during rolling upgrades / flag flips, same role
`MetricsFileLayout::of` plays for the compactor). The merge then becomes
a `SortPreservingMergeExec` over pre-sorted files instead of a full
sort.
3. **pack, single segment**: the as-is fast path (skipped for TSID-major
streams in this PR because segments were time-sorted) can come back —
the segment is already hash-sorted, upload it unchanged and mark it
`tsid-sorted-*`.
4. **pack, multiple segments**: one `MemTable` partition per segment
with a declared sort order, so the in-memory merge is also an SPM
instead of a full sort.
Query-side is already safe for this change: metrics SQL never declares
`sorted_by_time` and PromQL declares no order, so hash-sorted WAL files
cannot mislead a reader. The cost is slightly coarser time clustering
inside WAL files, negligible at WAL file sizes. Steps 1–2 carry most of
the win; 3–4 are incremental.
Behavior-neutral refactors split out of #13839 so that PR only carries
the TSID-major feature. Nothing here changes what is written, read or
deleted; every commit is independently reviewable.
## Commits
- `cba75d244b` **promql**: `apply_matchers` split into
`matcher_predicates(schema, matchers) -> Vec<Expr>` + `apply_matchers`
(unchanged behavior; lets other inputs evaluate the same matcher
semantics).
- `b5ff6fed21` **parquet**: `new_parquet_writer` / `append_metadata`
generic over `W: AsyncFileWriter` instead of `&mut Vec<u8>`; existing
callers unchanged.
- `b07d84216a` **merge**: `MergeParquetResult::{Single, Multiple}` →
`MergeParquetResult { files: Vec<MergedFile { buf, meta }>, file_format
}` + `into_single()`. The compactor's two copies of the upload / index /
push sequence become one loop (`compressed_size` is set from the buffer
for every file, as the `Multiple` arm already did); downsampling maps
its `(bufs, metas)` into it; ingester movers use `into_single()` instead
of panicking/erroring on `Multiple`.
- `014b6c5721` **compactor**: `deleted.rs` `derived_files()` /
`delete_from_storage()` replace three copies of collect / `storage::del`
/ ignore-not-found (data, `.ttv`, flattened).
- `7163786362` **search**: `generate_parquet_access_plan_from_ranges`
extracted from the bitmap path (`set_slices` → ranges); an out-of-range,
empty or unsorted range is rejected with `assert!` (`56de386522`; a
producer that violates the sorted/non-overlapping contract is a bug, not
something to degrade around) instead of the previous `unreachable!()` on
the bitmap path. Adds `FileSelection::RowRanges(Arc<Vec<Range<usize>>>)`
as the matching selection kind (no producer yet; Parquet only, Vortex
full scan like `RowGroups`).
- `754014ccf3` **test**: `ListingTableAdapter` plan-shape test for
`FileSortOrder::HashTimestampAsc` (from #13853): three overlapping
hash-sorted files → `SortPreservingMergeExec` over three single-file
groups, no `SortExec`, globally ordered rows. `tempfile` as `search`
dev-dependency.
## Validation
- `cargo fmt --check`
- `cargo clippy -p config -p promql -p search -p compaction -p
openobserve-jobs -p search_service -p promql-service --tests` clean
- `cargo test -p promql --lib -- utils` 10, `-p search --lib -- merge::
helpers listing_adapter` 14, `-p compaction --lib -- merge deleted` 31,
`-p config --lib -- test_file_key` 4 passed
- `merge/downsampling.rs` is enterprise-only and was updated by hand;
needs the enterprise CI compile.
Ashish KolheandSubhradeep Chakraborty
HuaijinandHengfei Yang
Prabhat Sharmaandgithub-actions[bot]
IluckySi
