## Summary
- Fixes the `Invalid PromQL syntax error handling` test failing in
enterprise CI
- Enterprise CI renders the no-data/error state slower than the 3s fixed
timeout used previously
- Replaced `isVisible({ timeout: 3000})` with `expect.poll` (10s window,
500ms–2s intervals) so the assertion is real but tolerates CI rendering
lag
## Root cause
When `sum(rate(` is entered and applied, the UI renders a no-data
indicator asynchronously. Locally (enterprise) this appears in ~2s. In
enterprise CI (slower hardware, more features enabled) it takes longer
than 3s, causing all three indicator checks to return `false` before
they appear.
## Test plan
- [x] Passes locally: `hasNoData=true` confirmed
- [ ] Enterprise CI Metrics shard goes green
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Two independent improvements to the Playwright E2E test suite.
---
### Part 1 — Sentinel audit fixes (original PR scope)
Fixes from the Sentinel audit review of PR #12358 (selector drift
stabilization).
- **Dead code removal**: delete `clickJobID()` from `logsPageEP.js`
(unreachable code containing `console.log`); remove unused `chromium`
import
- **Critical fix**: replace `console.log`/`console.error` in
`decryptLogSQL()` with `testLogger` equivalents
- **POM compliance**: replace raw `page.locator()` call in
`alerts-ui-operations.spec.js` `beforeEach` with
`pm.alertsPage.isIncidentsFeatureEnabled()`; add that method to
`alertsPage.js`
- **POM compliance**: move per-test `const pm = new PageManager(page)`
into `beforeEach` in `service-graph.spec.js`; replace raw
`page.waitForLoadState()` with
`pm.serviceGraphPage.waitForGraphReload()`; add `waitForGraphReload()`
to `serviceGraphPage.js`
- **Inline selector cleanup**: move metrics tab inline string literals
in `serviceGraphPage.js` into constructor properties
- **Tagging**: add
`@cipherKeys/@enterprise/@P1/@P2/@negative/@validation` to all 6 tests
in `cipherKeys.spec.js` (were untagged); add `@P1/@P2` to all 5 tests in
`regexPatternManagement.spec.js`
- **ESM fix**: correct `export const path = require('path')` → `import
path from 'path'` in `logoManagementPage.js`
- **Deduplication**: consolidate
`createFirstFunction`/`createSecondFunction` into shared
`createFunction(name, vrlCode)` in `pipelinesEP.js`
---
### Part 2 — Parallelize Metrics E2E shard
**Problem**: The `e2e / Metrics` CI shard was consistently the slowest
job at ~22 minutes per PR run, while all other shards complete in ~5-7
minutes.
**Root cause**: 9 of 11 Metrics spec files used
`test.describe.configure({ mode: 'serial' })` because they shared a
describe-scope `let pm` variable reassigned in `beforeEach`. This forced
every test in those files onto a single Playwright worker, eliminating
all parallelism despite `fullyParallel: true` and 5 workers being
configured.
**Fix applied to all 9 affected files** (`metrics.spec.js`,
`metrics-config.spec.js`, `metrics-config-tabs.spec.js`,
`metrics-visualizations.spec.js`,
`metrics-promql-query-persistence.spec.js`,
`metrics-table-column-order.spec.js`, `metrics-queries.spec.js`,
`metrics-aggregations.spec.js`, `metrics-advanced.spec.js`):
- Remove `test.describe.configure({ mode: 'serial' })`
- Replace shared `let pm` + `beforeEach` pattern with a `setupTest(page,
testInfo)` helper that creates and returns a fresh `PageManager` per
test — the same pattern already used in `metrics-promql-builder.spec.js`
(the one file that was already parallel)
- `metrics-table-column-order.spec.js`: pass `pm` explicitly into
`setupTableChart(pm, page, mode)` since it can no longer close over a
describe-scope variable
- Split consolidated mega-tests (for-loops over multiple independent
scenarios) in `metrics-queries`, `metrics-aggregations`, and
`metrics-advanced` into individual `test()` blocks so Playwright can
distribute them across workers
**No test assertions, selectors, query logic, or validation changed.**
| File | Tests before | Tests after |
|---|---|---|
| `metrics.spec.js` | 10 (serial) | 10 (parallel) |
| `metrics-config.spec.js` | 2 (serial) | 2 (parallel) |
| `metrics-config-tabs.spec.js` | 3 (serial) | 3 (parallel) |
| `metrics-visualizations.spec.js` | 4 (serial) | 4 (parallel) |
| `metrics-promql-query-persistence.spec.js` | 6 (serial) | 6 (parallel)
|
| `metrics-table-column-order.spec.js` | 20 (serial) | 20 (parallel) |
| `metrics-queries.spec.js` | 4 mega-tests (serial) | 12 individual
(parallel) |
| `metrics-aggregations.spec.js` | 3 mega-tests (serial) | 10 individual
(parallel) |
| `metrics-advanced.spec.js` | 4 mega-tests (serial) | 12 individual
(parallel) |
**Expected impact**: ~1300s (~22 min) → ~300-400s (~5-7 min) wall-clock
time.
---
## Test plan
- [ ] All 40 tests in the 4 Sentinel-audit spec files enumerate cleanly
via `npx playwright test --list`
- [ ] Tag filters `@cipherKeys`, `@P2`, `@sdr` match expected tests
- [ ] All Metrics spec files enumerate cleanly with no serial mode
warnings
- [ ] Metrics shard CI time drops to ~5-7 min on next PR run
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace Quasar-based selectors with O2 component equivalents across
alerts, dashboard, logs, and streams page objects. Key changes:
- OSwitch: target -btn suffix on inner button
- OSelect: use -trigger/-popover/-option suffixes, fallback to
[role="option"][data-test-value] when parent has no data-test
- OCombobox: use placeholder-based locators where no data-test exists
- OInput: descendant input selectors, pagination text format handling
- OTable: custom bottom slot pagination format ("N Stream(s)")
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: ktx-akshay <akshay.patil@kiara.tech>
Bumps
[systeminformation](https://github.com/sebhildebrandt/systeminformation)
from 5.31.1 to 5.31.6.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/sebhildebrandt/systeminformation/releases">systeminformation's
releases</a>.</em></p>
<blockquote>
<h2>v5.31.6</h2>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/sebhildebrandt/systeminformation/compare/v5.31.5...v5.31.6">https://github.com/sebhildebrandt/systeminformation/compare/v5.31.5...v5.31.6</a></p>
<h2>v5.31.5</h2>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/sebhildebrandt/systeminformation/compare/v5.31.4...v5.31.5">https://github.com/sebhildebrandt/systeminformation/compare/v5.31.4...v5.31.5</a></p>
<h2>v5.31.4</h2>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/sebhildebrandt/systeminformation/compare/v5.31.3...v5.31.4">https://github.com/sebhildebrandt/systeminformation/compare/v5.31.3...v5.31.4</a></p>
<h2>v5.31.3</h2>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/sebhildebrandt/systeminformation/compare/v5.31.2...v5.31.3">https://github.com/sebhildebrandt/systeminformation/compare/v5.31.2...v5.31.3</a></p>
<h2>v5.31.2</h2>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/sebhildebrandt/systeminformation/compare/v5.31.1...v5.31.2">https://github.com/sebhildebrandt/systeminformation/compare/v5.31.1...v5.31.2</a></p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/sebhildebrandt/systeminformation/blob/master/CHANGELOG.md">systeminformation's
changelog</a>.</em></p>
<blockquote>
<h1>Changelog</h1>
<h2>Major Changes - Version 5</h2>
<h4>New Functions</h4>
<ul>
<li><code>audio()</code> detailed audio information</li>
<li><code>bluetoothDevices()</code> detailed information detected
bluetooth devices</li>
<li><code>dockerImages()</code> detailed information docker images</li>
<li><code>dockerVolumes()</code> detailed information docker
volumes</li>
<li><code>printers()</code> detailed printer information</li>
<li><code>usb()</code> detailed USB information</li>
<li><code>wifiInterfaces()</code> detected Wi-Fi interfaces</li>
<li><code>wifiConnections()</code> active Wi-Fi connections</li>
</ul>
<h4>Breaking Changes</h4>
<p><strong>Be aware</strong>, that the new version 5.x <strong>is NOT
fully backward compatible</strong> to
version 4.x ...</p>
<p>We had to make <strong>several interface changes</strong> to keep
systeminformation as
consistent as possible. We highly
<a href="https://systeminformation.io/changes.html">recommend to go
through the complete list</a>
and adapt your own code to be again compatible to the new version 5.</p>
<table>
<thead>
<tr>
<th>Function</th>
<th>Old</th>
<th>New (V5)</th>
<th>Comments</th>
</tr>
</thead>
<tbody>
<tr>
<td>unsupported values</td>
<td>-1</td>
<td>null</td>
<td>values which are unknown or<!-- raw HTML omitted -->unsupported on
platform</td>
</tr>
<tr>
<td><code>battery()</code></td>
<td>hasbattery<!-- raw HTML omitted -->cyclecount<!-- raw HTML omitted
-->ischarging<!-- raw HTML omitted -->designedcapacity<!-- raw HTML
omitted -->maxcapacity<!-- raw HTML omitted -->acconnected<!-- raw HTML
omitted -->timeremaining</td>
<td>hasBattery<!-- raw HTML omitted -->cycleCount<!-- raw HTML omitted
-->isCharging<!-- raw HTML omitted -->designedCapacity<!-- raw HTML
omitted -->maxCapacity<!-- raw HTML omitted -->acConnected<!-- raw HTML
omitted -->timeRemaining</td>
<td>pascalCase conformity</td>
</tr>
<tr>
<td><code>blockDevices()</code></td>
<td>fstype</td>
<td>fsType</td>
<td>pascalCase conformity</td>
</tr>
<tr>
<td><code>cpu()</code></td>
<td>speedmin<!-- raw HTML omitted -->speedmax</td>
<td>speedMin<!-- raw HTML omitted -->speedMax</td>
<td>pascalCase conformity</td>
</tr>
<tr>
<td><code>cpu().speed</code><!-- raw HTML omitted
--><code>cpu().speedMin</code><!-- raw HTML omitted
--><code>cpu().speedMax</code></td>
<td>string values</td>
<td>now returning<!-- raw HTML omitted -->numerical values</td>
<td>better value handling</td>
</tr>
<tr>
<td><code>cpuCurrentspeed()</code></td>
<td></td>
<td>cpuCurrentSpeed()</td>
<td>function name changed<!-- raw HTML omitted -->pascalCase
conformity</td>
</tr>
<tr>
<td><code>currentLoad()</code></td>
<td>avgload<!-- raw HTML omitted -->currentload<!-- raw HTML omitted
-->currentload_user<!-- raw HTML omitted -->currentload_system<!-- raw
HTML omitted -->currentload_nice<!-- raw HTML omitted
-->currentload_idle<!-- raw HTML omitted -->currentload_irq<!-- raw HTML
omitted -->raw_currentload</td>
<td>avgLoad<!-- raw HTML omitted -->currentLoad<!-- raw HTML omitted
-->currentLoadUser<!-- raw HTML omitted -->currentLoadSystem<!-- raw
HTML omitted -->currentLoadNice<!-- raw HTML omitted
-->currentLoadIdle<!-- raw HTML omitted -->currentLoadIrq<!-- raw HTML
omitted -->rawCurrentLoad</td>
<td>pascalCase conformity</td>
</tr>
<tr>
<td><code>dockerContainerStats()</code></td>
<td>mem_usage<!-- raw HTML omitted -->mem_limit<!-- raw HTML omitted
-->mem_percent<!-- raw HTML omitted -->cpu_percent<!-- raw HTML omitted
-->cpu_stats<!-- raw HTML omitted -->precpu_stats<!-- raw HTML omitted
-->memory_stats</td>
<td>memUsage<!-- raw HTML omitted -->memLimit<!-- raw HTML omitted
-->memPercent<!-- raw HTML omitted -->cpuPercent<!-- raw HTML omitted
-->cpuStats<!-- raw HTML omitted -->precpuStats<!-- raw HTML omitted
-->memoryStats</td>
<td>pascalCase conformity</td>
</tr>
<tr>
<td><code>dockerContainerProcesses()</code></td>
<td>pid_host</td>
<td>pidHost</td>
<td>pascalCase conformity</td>
</tr>
<tr>
<td><code>graphics().display</code></td>
<td>pixeldepth<!-- raw HTML omitted -->resolutionx<!-- raw HTML omitted
-->resolutiony<!-- raw HTML omitted -->sizex<!-- raw HTML omitted
-->sizey</td>
<td>pixelDepth<!-- raw HTML omitted -->resolutionX<!-- raw HTML omitted
-->resolutionY<!-- raw HTML omitted -->sizeX<!-- raw HTML omitted
-->sizeY</td>
<td>pascalCase conformity</td>
</tr>
<tr>
<td><code>networkConnections()</code></td>
<td>localaddress<!-- raw HTML omitted -->localport<!-- raw HTML omitted
-->peeraddress<!-- raw HTML omitted -->peerport</td>
<td>localAddress<!-- raw HTML omitted -->localPort<!-- raw HTML omitted
-->peerAddress<!-- raw HTML omitted -->peerPort</td>
<td>pascalCase conformity</td>
</tr>
<tr>
<td><code>networkInterfaces()</code></td>
<td>carrier_changes</td>
<td>carrierChanges</td>
<td>pascalCase conformity</td>
</tr>
<tr>
<td><code>processes()</code></td>
<td>mem_vsz<!-- raw HTML omitted -->mem_rss<!-- raw HTML omitted
-->pcpu<!-- raw HTML omitted -->pcpuu<!-- raw HTML omitted -->pcpus<!--
raw HTML omitted -->pmem</td>
<td>memVsz<!-- raw HTML omitted -->memRss<!-- raw HTML omitted
-->cpu<!-- raw HTML omitted -->cpuu<!-- raw HTML omitted -->cpus<!-- raw
HTML omitted -->mem</td>
<td>pascalCase conformity<!-- raw HTML omitted -->renamed
attributes</td>
</tr>
<tr>
<td><code>processLoad()</code></td>
<td>result as object</td>
<td>result as array of objects</td>
<td>function now allows to provide more than<!-- raw HTML omitted -->one
process (as a comma separated list)</td>
</tr>
<tr>
<td><code>services()</code></td>
<td>pcpu<!-- raw HTML omitted -->pmem</td>
<td>cpu<!-- raw HTML omitted -->mem</td>
<td>renamed attributes</td>
</tr>
<tr>
<td><code>vbox()</code></td>
<td>HPET<!-- raw HTML omitted -->PAE<!-- raw HTML omitted -->APIC<!--
raw HTML omitted -->X2APIC<!-- raw HTML omitted -->ACPI<!-- raw HTML
omitted -->IOAPIC<!-- raw HTML omitted -->biosAPICmode<!-- raw HTML
omitted -->TRC</td>
<td>hpet<!-- raw HTML omitted -->pae<!-- raw HTML omitted -->apic<!--
raw HTML omitted -->x2Apic<!-- raw HTML omitted -->acpi<!-- raw HTML
omitted -->ioApic<!-- raw HTML omitted -->biosApicMode<!-- raw HTML
omitted -->rtc</td>
<td>pascalCase conformity</td>
</tr>
</tbody>
</table>
<h4>Other Improvements and Changes</h4>
<ul>
<li><code>baseboard()</code>: added memMax, memSlots</li>
<li><code>bios()</code>: added language and features (linux)</li>
<li><code>blockDevices()</code> added raid group member (linux)</li>
<li><code>cpu()</code>: extended AMD processor list</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/sebhildebrandt/systeminformation/commit/da1cbf5eb09907cf5c3431f4c9ea441b7a227617"><code>da1cbf5</code></a>
5.31.6</li>
<li><a
href="https://github.com/sebhildebrandt/systeminformation/commit/87dbb60f98dd39452bda211da0b6c63e4a7db898"><code>87dbb60</code></a>
fix: smaller issues</li>
<li><a
href="https://github.com/sebhildebrandt/systeminformation/commit/1d1bddf81267a3726b1200fa05ebe04b749f8d64"><code>1d1bddf</code></a>
fix: smaller issues</li>
<li><a
href="https://github.com/sebhildebrandt/systeminformation/commit/f69576f8da71571a988e7f430b068952c141bea9"><code>f69576f</code></a>
fix: networkInterfaces() ci wip</li>
<li><a
href="https://github.com/sebhildebrandt/systeminformation/commit/78624d1567d0410b3b6781caaf2441c4c32a810c"><code>78624d1</code></a>
fix: networkInterfaces() ci</li>
<li><a
href="https://github.com/sebhildebrandt/systeminformation/commit/ed1cac537c59763301d802ad1b55b4b8581e7553"><code>ed1cac5</code></a>
5.31.5</li>
<li><a
href="https://github.com/sebhildebrandt/systeminformation/commit/4dd5aa91492ee4953109135dabab9adbedb9df8e"><code>4dd5aa9</code></a>
netStats() netstat command adaption (mac OS)</li>
<li><a
href="https://github.com/sebhildebrandt/systeminformation/commit/59360e6e2a4ffd6b02d868716b3091e469be9264"><code>59360e6</code></a>
5.31.4</li>
<li><a
href="https://github.com/sebhildebrandt/systeminformation/commit/3ef3f3befc0218c0ac4cdc975ed328f7e5318867"><code>3ef3f3b</code></a>
diskLayout() USB tahoe compatibility (mac OS)</li>
<li><a
href="https://github.com/sebhildebrandt/systeminformation/commit/8edd13034fe729481cefc264ea20bb033a1a31be"><code>8edd130</code></a>
5.31.3</li>
<li>Additional commits viewable in <a
href="https://github.com/sebhildebrandt/systeminformation/compare/v5.31.1...v5.31.6">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/openobserve/openobserve/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Summary
Design at:
https://github.com/openobserve/designs/blob/main/infrastructure/bloomfilter/DESIGN.md
A bloom-filter pruning layer that sits **between `file_list` and
tantivy** in the search path. It drops candidate files before any `.ttv`
is opened, for **positive equality / IN** lookups on high-cardinality
fields (`trace_id`, `request_id`, …).
It is built **on top of** tantivy: the `.bf` is constructed by iterating
the tantivy term dictionary, so a field must be a tantivy secondary
index (`index_fields`) **and** opted into `bloom_filter_fields`. Only
the intersection is built — a bloom-only field produces nothing. To make
that intersection non-empty by construction, enabling a field for bloom
now folds it into `index_fields` automatically (see Config). Values are
not inspected; the operator chooses which fields are worth it (see
guidance below).
Design notes live in the module-level docs under `src/infra/src/bloom/`
(`mod.rs`, `writer.rs`, `reader.rs`, `sbbf.rs`).
## Design
- **Transposed (block-major) `.bf` layout.** All files in a `.bf` share
`num_blocks = B`, so `block_index = fastmap(hash(value), B)` is
identical across files and the blocks a query needs form **one
contiguous row**. The search side reads **one `M × 32`-byte row per
`.bf`** to answer membership for all `M` files — prune cost is
`O(groups)`, not `O(files)`. (`infra::bloom`, own ~120-LOC SBBF over
gxhash64.)
- **Sub-grouping.** A hour's new files are sorted by record count and
split into chunks of ≤ `ZO_BLOOM_FILTER_MAX_FILES_PER_BF` (default 256),
one `.bf` each. Bounds compactor build memory (`M × B × 32`) and `.bf`
object size; query reads become `ceil(files / 256)` rows.
- **Footer-only reader + batched fetch.** Footer (≈ `M × 12` B) comes
from `BLOOM_FOOTER_CACHE` or one 16 KB suffix GET (with a precise
re-read when a large footer overflows the probe); rows are pulled with
one `cache_storage::get_ranges` call. `cache_storage` was reworked to do
real batched cache reads (memory: 1 lookup + N slices; disk: 1
`File::open` + N `pread`).
- **B sizing.** `num_blocks_for(max_records_in_chunk,
ZO_BLOOM_FILTER_FPP)` (default fpp `0.01`) — record count is a safe NDV
upper bound, so the SBBF never saturates for high-cardinality fields.
- **Versioned, with orphan cleanup.** A file's `bloom_ver` is write-once
— a row is never re-stamped, so dumped rows keep pointing at the `.bf`
they were built into. But `bloom_ver` is **not** append-only at the
bucket level: a merge produces new output files with a fresh `bloom_ver`
and deletes the small inputs, so the old inputs' `.bf` can become
orphaned. After each merge round the compactor retires any `.bf` whose
`bloom_ver` is no longer referenced by a live `file_list` row in the
bucket (enqueued through the existing `file_list_deleted` path).
`bloom_ver` is preserved through every file_list INSERT/SELECT path
(incl. the SQLite read-cache) and the dump parquet.
- **Observability.** Prometheus metrics for prune
(`bloom_prune_keep_ratio`, `bloom_prune_duration_seconds`);
`bloom-inspect` CLI dumps a `.bf` footer (fields, B, row bytes, file
names); footer-cache hit/miss via the shared `BytesCache` tagged
metrics; `Local::get_ranges` now reports `STORAGE_READ_*` (previously
bypassed).
## When it helps (measured, S3, no disk cache)
Same stream `benchtest`, 170 files / 14.64 GB index, single matching
row. The only variable is whether the field's values are
time-correlated:
| Field | Type | tantivy only | bloom + tantivy | Outcome |
|---|---|---|---|---|
| `trace_id` (16-byte random) | random | **2584 ms** (opens all 170
`.ttv`) | 24 ms prune + 65 ms = **89 ms** | **~29× faster** |
| `request_id` (`…-7e74-…`, UUIDv7) | time-ordered | **154 ms** (sparse
index range-prunes) | 42 ms + 154 ms = **196 ms** | ~1.3× **slower** |
The whole 170-file group was answered by **one row read of 5440 bytes**
(`170 × 32`) + a cached footer.
**Guidance — enable only when all three hold (advisory; not enforced):**
1. **High cardinality** field.
2. **Random values** (not time-ordered) — so tantivy can't range-prune
and must scan every file. Time-ordered ids (UUIDv7, snowflake) are
already pruned by tantivy's sparse index in memory → bloom is pure
overhead there.
3. **Many files per hour** — hourly volume ≳ 10 ×
`ZO_COMPACT_MAX_FILE_SIZE` (≳10 `.ttv` files), so there's something to
prune. Bloom turns N file opens into one row read; at 1–2 files/hour
it's net overhead.
The 29× result above had 170 files/hour all answered by one `.bf`. ❌
Low-cardinality / non-indexed / range / `LIKE` / regex / `!=` / OR: no
effect.
## Config
- `ZO_BLOOM_FILTER_ENABLED` (default true) — global gate.
- Per stream: a field must be in **both** `index_fields` and
`bloom_filter_fields`. Adding a field to `bloom_filter_fields` auto-adds
it to `index_fields` (bloom is built on the secondary index), and the
default index fields include the default bloom fields.
- `ZO_BLOOM_FILTER_FPP` (default 0.01) — target false-positive
probability; sizes the SBBF block count (smaller fpp = fewer false
survivors, larger `.bf`). Out-of-range falls back to 0.01.
- `ZO_BLOOM_FILTER_MAX_FILES_PER_BF` (default 256) — files per `.bf`;
trades query reads vs build memory.
- `ZO_BLOOM_FILTER_DEFAULT_FIELDS` (default empty) — fields bloomed for
every stream.
- `ZO_BLOOM_FOOTER_CACHE_MAX_SIZE` (default 1% mem, clamped 32–256 MB).
Removed in this PR: `ZO_BLOOM_FILTER_DISABLED_ON_SEARCH`,
`ZO_BLOOM_FILTER_NDV_RATIO` (B is now sized from record count + fpp, not
an ndv ratio).
## Test plan
- [x] `cargo build`, `cargo clippy -p infra -p openobserve --lib -- -D
warnings`, `cargo fmt --check` clean
- [x] Unit tests: `cargo test -p infra --lib bloom`, `cargo test -p
openobserve --lib bloom`
- [x] E2E on S3 (results above): random field prunes 170→1 and is ~29×
faster; transposed read = 1 request / 5440 B per group
- [x] Reviewer: confirm gating expectations (operator opts fields in; no
auto value detection)
- [x] Test: orphan bloom: 2 * 256MB -> merge to 512MB -> merge to
1024MB, the first .bf should be deleted
- [x] Test: orphan bloom: 6 * 256MB -> merge to 1GB + 512MB -> 512 MB
merge to 1024MB, the first .bf include 2 files(1GB + 512MB), after merge
again, the first still have 2 files but one of them already deleted. we
should keep the first .bf, because still have files in use.
## Notes
- `.bf` format is `VERSION 0x01` (transposed). Not compatible with any
pre-merge experimental `.bf`; rebuild by wiping the `bloom/` dir and
resetting `bloom_ver=0`.
- Hash is gxhash64 (project-wide `config::utils::hash`), not the Parquet
spec's xxhash64 — `.bf` is internal, never read by external tools.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
- SDR Save Settings trigger point discrepency
- Edit the existing alert - Ai icon is overlapping
- On stream field Scroller is not working
- With other languages, the Side Navigation UI is broken
- Search box dont having any border (List search view )
- text area color not looks proper
- Pipeline history - pagination is not working properly
- iam page default pagination value change
- Org display on top right should have a bigger width - more real estate
can be used there
- Date - picker does not show selection when selection 27th may to 20th
may
- Date - picker also does not handle selection if use selects 27th may
first and 20th may later, it should select the correct date no matter
what user selects the date in.
- when searching across folders in dashboard, it shows no data instead
of skeleton loading
- https://github.com/openobserve/o2-enterprise/issues/1824
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
## Summary
- **`--no-sandbox` / `--disable-setuid-sandbox`** added to
`chromium.launch()` in `globalSetup.js` and `launchOptions` in
`playwright.config.js` (CI-only) — prevents silent Chromium hangs on
restricted runner environments
- **`globalTimeout: 110 min`** added to `playwright.config.js` — caps
the entire suite at 110 min (10 min under the job timeout) so hung runs
fail with a clear Playwright error instead of a silent CI kill
- **`restore-keys` fallback** added to the browser cache step — a
`package-lock.json` change no longer forces a full 163 MB re-download on
every shard simultaneously
- **Runner right-sizing**: `ui_integration_tests` →
`repo-openobserve-standard-16`; `merge_and_upload_reports` +
`playwright_summary` → `repo-openobserve-standard-4`
## Test plan
- [ ] Confirm `ui_integration_tests` shards start without Chromium
launch hangs
- [ ] Confirm cache hit on second run (restore-keys fallback working)
- [ ] Confirm `merge_and_upload_reports` and `playwright_summary`
succeed on 4 GB runners
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
## Summary
When generating the tantivy index after a parquet merge, we previously
read **all** columns from the merged parquet file even though only the
indexed fields are needed. This PR restricts the parquet read to just
the index fields, significantly reducing CPU work and peak memory during
compaction.
## Benchmark
Same workload on both runs: compactor merges **173 files** (original
9.49 GB / compressed 1.60 GB) into one parquet file, then builds the
tantivy index (7 row_groups, 800,000 rows, 2 indexing threads).
| Metric | Before | After | Delta |
| --- | ---: | ---: | ---: |
| `parallel::build_index` elapsed | 5945 ms | 2868 ms | **−51.8%**
(2.07×) |
| Tantivy index generation (total) | 7798 ms | 4814 ms | **−38.3%**
(1.62×) |
| Merge row-group indices | 1748 ms | 1850 ms | ~same |
About **2× faster** parallel index build, the total improve about 30%
with identical output.
> **Note:** The size of the win depends on the ratio of index fields to
total fields in the stream. This test used a stream with **~600 fields
and only 7 index fields**, so we skip reading the vast majority of
columns. For streams with few non-index fields the improvement will be
much smaller.
## Test plan
- [x] Verified compactor still emits a tantivy index of the same size
for the same input
- [x] CI green
## Summary
In the O2 tokenizer, camelCase identifiers (e.g. `HTTPSConnection`,
`getUserName`, `CamelCase`) are split into sub-tokens. Previously:
- **Ingest** indexed both the root token (`HTTPSConnection`) and its
splits (`HTTPS`, `Connection`).
- **Search** dropped the root and only emitted the splits.
This PR makes **search emit only the root token** and drop the splits.
### Why
When a user searches for `HTTPSConnection`, the query tokenizer used to
break it into `HTTPS` AND `Connection`. That matches any document that
happens to contain both words anywhere — including documents that never
contained the original camelCase identifier. Concretely:
- A log line like `"HTTPS handshake failed for Connection #42"` would
match a search for `HTTPSConnection`, which is wrong.
- Conversely, the ingest side already indexes the full `HTTPSConnection`
root token, so the search side can rely on it directly for an exact,
intent-preserving match.
Emitting only the root token on the search path keeps the query aligned
with the user's literal input and eliminates these false positives,
while ingest continues to index both root + splits so partial matches
(`Connection`, `HTTPS`) still work when the user actually types a
sub-word.
### Notes
- Non-ASCII delimiters that follow a camelCase token (e.g.
`CamelCase中文`) are still flushed correctly in search mode —
`buffer_offset` is set to the root token's `offset_to` before the
delimiter is pushed.
- Added regression tests covering camelCase followed by non-ASCII,
acronym-style prefixes (`HTTPSConnection中Foo`), back-to-back camelCase
tokens across a non-ASCII delimiter, and plain IP addresses (no
camelCase) under search mode.
## Test plan
- [x] `cargo test -p config tokenizer::o2_tokenizer` passes
- [x] Re-index a sample stream and confirm exact-match queries on
camelCase identifiers return only the expected documents
- [x] Confirm sub-word queries (e.g. just `Connection`) still match
documents containing `HTTPSConnection`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Problem
Histogram panels showed a false drop in the **leftmost bucket**.
`histogram()` rewrites to `date_bin(interval, _timestamp, epoch)`, which
assigns each record to the **left boundary** of its bucket. So a record
at `08:10:17` gets key `08:10:00`.
But the scan filter was `_timestamp >= start_time` where `start_time`
was the raw query time (e.g. `08:10:17`). Records in `[08:10:00,
08:10:17)` passed the bucket label check but were excluded from the scan
— making the first bucket appear partially filled.
Fixes#1753 (o2-enterprise)
## Fix
Snap `start_time` **down to the nearest bucket boundary** at the
**leader node**, before it is broadcast to all follower nodes via
`FlightSearchRequest`.
Location: `RemoteScanNodes::get_remote_node()` in
`distributed_plan/node.rs`
```rust
start_time: {
let t = self.req.time_range.as_ref().map(|x| x.0).unwrap_or(0);
if self.req.histogram_interval > 0 {
histogram_bucket_start(t, self.req.histogram_interval * 1_000_000)
} else {
t
}
},
```
The arithmetic is extracted into `histogram_bucket_start(timestamp_us,
interval_us)` in `sql/histogram.rs` so the formula lives in one place.
Doing the snap at the leader means:
- All followers receive an already-aligned `start_time`
- `SimpleHistogramVisitor` / `SimpleMultiHistogramVisitor` on followers
need no changes (their `start_time % rounding_by` is already 0)
- Covers both cluster leader and super cluster leader (both go through
`generate_remote_scan_rules → get_remote_node()`)
## Test plan
- [ ] Deploy and run a histogram query spanning a non-aligned
`start_time`
- [ ] Verify first bucket count matches adjacent full buckets
- [ ] Verify queries without `histogram_interval` are unaffected
## Summary
- Switch `ui_integration_tests`, `merge_and_upload_reports`, and
`playwright_summary` jobs from `ubicloud-standard-8` to
`repo-openobserve-standard-8`
- `build_binary` remains on `ubicloud-standard-16` (unchanged)
## Test plan
- [ ] Verify CI triggers and jobs pick up the correct `repo-openobserve`
runners
- [ ] Confirm `build_binary` still runs on ubicloud
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
- close https://github.com/openobserve/o2-enterprise/issues/1658
## Summary
Speed up tantivy index generation during compaction by indexing parquet
row_groups in parallel.
- New dispatcher in `src/service/tantivy/mod.rs` picks one of two
builders:
- **sequential** (`sequential.rs`) — legacy producer/consumer path, used
for non-parquet inputs or when `thread_num <= 1`.
- **parallel** (`parallel.rs`) — one `SingleSegmentIndexWriter` per
parquet row_group built under `spawn_blocking` +
`Semaphore(thread_num)`, then stitched via
`tantivy::indexer::merge_indices`.
- `create_tantivy_index` now takes raw parquet `Bytes` so each
per-row_group worker gets a cheap `Bytes::clone()` instead of re-reading
the file. Call sites in `src/job/files/parquet.rs` and
`src/service/compact/merge.rs` are updated.
## Config
- `ZO_COMPACT_TANTIVY_BUILDER_THREAD_NUM` (default `2`) — per-file
concurrent row_group workers. `<= 1` keeps the legacy single-threaded
path.
## Benchmark
Input: 21 row_groups, 2,672,970 rows, ~1.38 GB .ttv output.
| variant | cpu | total tantivy | peak mem | vs main (time) | vs main
(mem) |
|---|---:|---:|---:|---:|---:|
| **main (baseline)** | 2 | 108,673 ms | **10.62 GB** | — | — |
| PR, workers=2 (default) | 2 | 109,060 ms | 2.92 GB | ≈ same | **−72%**
|
| PR, workers=4 | 4 | 92,232 ms | 3.99 GB | **−14%** | **−62%** |
| PR, workers=10 | 10 | 76,167 ms | 7.75 GB | **−27%** | **−27%** |
- **Memory is the big win.** Even sequential mode (workers=1) cuts peak
from 10.6 GB → 3.0 GB (~3.6×).
- **At the default (workers=2), wall time matches main exactly while
peak memory drops 72%.** Safe upgrade.
- **Doubling workers buys time at the cost of memory** (per-row_group
segments buffer before merge). 4 workers is the best speed/memory
trade-off; ~10 is the cap (single-threaded `merge_indices` floor is ~60
s).
## Why it's correct
OpenObserve relies on `tantivy doc_id == global parquet row_index` to
map search hits back to row offsets. The parallel path preserves it
because:
1. Each per-row_group writer writes rows in stored order → local
`doc_id` == offset inside that row_group.
2. Worker results are sorted by `rg_idx` before `merge_indices`.
3. `merge_indices` stacks segments in input order (no `sort_by_field`
path in current tantivy).
4. Parquet stores row_groups in row order, so the stitched layout
matches the original global row order.
Two guards backstop this: the merged index must be a single segment, and
its `max_doc` must equal the sum of per-row_group row counts.
## Tests
- `test_row_group_parallel_preserves_row_order` — 4 row_groups, 3
workers (forces out-of-order completion), asserts every marker's `doc_id
== row_index`.
- `test_row_group_parallel_matches_sequential` — same data through both
paths yields identical `(marker → doc_id)` mappings.
- `test_row_group_parallel_single_row_group_fast_path` —
single-row_group input skips `merge_indices` and still preserves row
order.
## Summary
- Moves 3 enterprise-only tests out of OSS regression spec files into
dedicated `*-ent` spec files
- These spec files live in the OSS repo but are **not referenced in the
OSS CI workflow** — they are picked up only by the enterprise CI
- Fixes recurring failures in the OSS regression run caused by features
that don't exist in OSS builds
## Why the tests were failing in OSS
| Shard | Test | Root cause |
|---|---|---|
| Traces-Regression | Service graph side panel (Bug #11687) |
`[data-test="traces-service-graph-toggle"]` is wrapped in
`v-if="config.isEnterprise == 'true'"` in `SearchBar.vue` — element
never renders in OSS → 45s timeout |
| Logs-Regression | Scheduled Search navigation (Bug #7354) | `POST
/api/{org}/search_jobs` returns **404** in OSS — endpoint is
enterprise-only |
## Changes
**New files:**
-
`tests/ui-testing/playwright-tests/RegressionSet/logs-regression-ent.spec.js`
— Scheduled Search / `search_jobs` describe block (2 tests)
-
`tests/ui-testing/playwright-tests/RegressionSet/traces-regression-ent.spec.js`
— service graph side panel test (Bug #11687)
**Modified files:**
- `logs-regression-9044-7354.spec.js` — removed the `#7354` Scheduled
Search describe block; `#9044` scroll retention tests remain
- `traces-regression.spec.js` — removed Bug #11687 service graph test
block
The companion PR in `o2-enterprise` adds these two files to the
Logs-Regression and Traces-Regression shards of the enterprise workflow.
## Test plan
- [ ] OSS regression CI (`playwright_regression.yml`) should pass — no
more enterprise-only test failures
- [ ] Enterprise regression CI picks up `*-ent.spec.js` files via the
companion workflow PR
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Follow-up to #12249. The cross-type collision check in the file upload
handler passes the formatted `stream_name` to `get_url_jobs_for_table`,
but URL jobs are stored with the raw `table_name` from the HTTP path.
When names contain uppercase or special characters,
`format_stream_name("My_Table")` produces `"my_table"`, which won't
match the stored URL job name `"My_Table"` — causing the cross-type
protection to be silently bypassed.
## Fix
Changed `get_url_jobs_for_table(&org_id, &stream_name)` to
`get_url_jobs_for_table(&org_id, table_name.trim())` — matching how the
URL handler itself queries URL jobs at line 299.
## Before/After
```rust
// Before: formatted name passed to URL jobs lookup (WRONG)
get_url_jobs_for_table(&org_id, &stream_name)
// After: raw table_name passed (CORRECT, matches how URL handler queries)
get_url_jobs_for_table(&org_id, table_name.trim())
```
## Notes
- `ENRICHMENT_TABLES` cache key continues to use `stream_name`
(formatted) — that's correct
- `get_url_jobs_for_table` now uses `table_name.trim()` — matching the
URL handler at line 299
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Problem
Playwright CI jobs were hanging for 2–6 hours and then timing out
without running a single test. Root cause: `npx playwright install
--with-deps chromium` downloads the browser binary successfully (~3s),
but the `--with-deps` flag then runs a second `apt-get` call that waits
indefinitely for the dpkg lock held by Ubuntu's `unattended-upgrades`
background daemon on ubicloud runners.
Evidence: in a 2h8m run, Chromium downloaded at `08:51:49`, but the
`playwright install` process was still listed as a zombie when the job
was killed at `11:00:23`.
## Fix
Two-layer fix applied to all four workflow files (`playwright.yml` +
`playwright_regression.yml` in both OSS and enterprise):
**1. Kill background apt services at job start** (new first step)
```bash
sudo systemctl stop apt-daily.service apt-daily-upgrade.service unattended-upgrades.service
sudo systemctl stop apt-daily.timer apt-daily-upgrade.timer
sudo systemctl disable apt-daily.service apt-daily-upgrade.service unattended-upgrades.service
sudo flock --wait 30 /var/lib/dpkg/lock-frontend true
```
This eliminates the root cause — the dpkg lock contender is gone before
`playwright install` runs.
**2. Cache the browser binary** (`actions/cache@v4` keyed on
`package-lock.json` hash)
On cache hit, only system dependencies are installed (`playwright
install-deps chromium`), avoiding apt entirely. On cache miss, the full
`playwright install --with-deps chromium` runs with apt services already
stopped.
**3. Filter TestDino uploads to failures only**
Before uploading to TestDino, the JSON report is filtered to only
include tests with `status === 'unexpected'`. This prevents passing
tests from polluting the TestDino failure history.
## Files changed
- `.github/workflows/playwright.yml` — kill apt, cache browser,
cache-aware install, TestDino filter
- `.github/workflows/playwright_regression.yml` — kill apt, cache
browser, split into separate npm/playwright/test steps
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Bug fix for - https://github.com/openobserve/openobserve/issues/12247
- **Bug:** Creating a URL enrichment table with name `X`, then creating
a file upload enrichment also named `X` (or vice versa), silently
overwrites the first one at every storage layer.
- **Fix:** Added cross-type name collision checks at the API boundary.
The file upload handler now rejects if a URL enrichment already exists
with the same name, and the URL handler rejects if a file upload
enrichment already exists. Same-type updates (append/retry/replace) are
unaffected.
## Root Cause
Enrichment tables are uniquely identified by `(org_id, table_name)`
across all storage layers — database, caches, object store, and
coordinator keys. Both URL-based and file-upload enrichments share the
same `StreamType::EnrichmentTables` and the same key space, so creating
one with the same name as another silently clobbers the previous one.
## Changes
**1 file changed** —
[src/handler/http/request/enrichment_table/mod.rs](src/handler/http/request/enrichment_table/mod.rs)
### `save_enrichment_table` (file upload handler)
Added check after input validation: if the table name already exists in
`ENRICHMENT_TABLES` and has URL jobs, return `400` — "already exists as
a URL enrichment."
### `save_enrichment_table_from_url` (URL handler)
Added check after fetching existing jobs: if the table name exists in
`ENRICHMENT_TABLES` but has no URL jobs, return `400` — "already exists
as a file upload enrichment."
### Behavior matrix
| | Existing: URL | Existing: File Upload | Existing: None |
|---|---|---|---|
| **Create File Upload** | ❌ Reject (400) | ✅ Allow (update) | ✅ Allow |
| **Create URL** | ✅ Allow (update) | ❌ Reject (400) | ✅ Allow |
## Test Plan
- [ ] Manually verify: create a URL enrichment named `test_cross_type` →
attempt file upload with same name → should get 400 error
- [ ] Manually verify: create a file upload enrichment named
`test_cross_type2` → attempt URL creation with same name → should get
400 error
- [ ] Manually verify: same-type updates still work (file→file append,
URL→URL retry/replace)
- [ ] Playwright automation will be added in a follow-up PR in the
`RegressionSet` folder
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
- The previous PR (#12229) fixed most instances of `200GB` in `en.json`
but missed the `open_source.badge_text` key which powers the \"Get
Enterprise Edition\" dialog shown to OSS users
- Fixed: `about.enterprise_offer.open_source.badge_text` → `Free up to
50GB / day`
## Test plan
- [ ] Open the app as an OSS user (no enterprise license)
- [ ] Trigger the Enterprise upgrade dialog
- [ ] Confirm badge reads "Free up to 50GB / day"
🤖 Generated with [Claude Code](https://claude.ai/claude-code)
Fixes https://github.com/openobserve/o2-enterprise/issues/1796 (item 1:
"Reload Existing URLs" returns error).
## Summary
Commit
[`85d261b5d0`](https://github.com/openobserve/openobserve/commit/85d261b5d0e4b68dbc79ca18b844fc53e2368ee9)
(vulnerability scan fixes, Apr 14 2026) removed the `!retry` guard from
enrichment table URL validation. This broke the **reload** update mode
because the frontend sends an empty URL when retrying/reprocessing
existing URLs — the backend already has them stored and just needs to
re-trigger processing.
## Root Cause
Before the breaking commit, URL validation was guarded by `!retry`:
```rust
if !retry && let Err(err_msg) = validate_enrichment_url(&request_body.url) {
return MetaHttpResponse::bad_request(err_msg);
}
```
The security fix removed the guard under the assumption that skipping
validation on retry was an SSRF bypass vector. However, the retry flow
sends `url=""` intentionally — the existing URLs are in the database.
## Fix
Re-adds the `!retry` guard around URL validation. In retry mode, URL
validation is skipped entirely (the URLs come from DB). All other modes
(create, append, replace, replace_failed) still require a non-empty,
valid URL with full SSRF protection.
## Test Plan
- [ ] Create enrichment table from URL
- [ ] Edit table → Reload existing URLs → Should succeed (no "URL cannot
be empty")
- [ ] Edit table → Add new URL → Should still validate URL format
- [ ] Edit table → Replace all URLs → Should still validate URL format
- [ ] Edit table → Replace failed URL → Should still validate URL format
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
## Problem
`sort_by=duration` (and any non-timestamp sort) in the
`/traces/latest_stream` endpoint had three bugs:
1. All partitions collapsed into one full-range scan
2. Q1 results re-sorted by `trace_start_time`, discarding SQL engine
order
3. `fetch_size` capped at `query_default_limit` (1000), so top-k ranked
on a truncated sample
## Fix
Mirrors the approach from #11654 (_search_stream):
- Detect non-ts ORDER BY generically via \`sql_order_expr\` prefix — no
hardcoded field names
- Run Q1 across **all** partitions; feed deduped hits into a
\`TopKHeap\` bounded to \`k = from + size\`
- Q2 (span details) runs **once** after all partitions complete, over
the narrowed time window of the deliverable traces
- Final hits emitted in heap order — correct global rank by ORDER BY
column
## Refactor
Extracted `run_q2_for_traces` async helper to eliminate ~300 lines of
duplicated code between the ts-path and non-ts-path. Both paths now call
the same helper for the `TraceResponseItem` build + Q2a (per-trace
aggregates) + Q2b (per-service breakdown). Also removed the dead
`"duration"` sort arm from the ts-path and the now-unused `sort_by`
parameter from `process_latest_traces_stream`.
## Known Limitation
`zo_sql_duration` is computed per-partition, so traces whose spans
straddle a partition boundary get an underestimated duration. Rare in
practice; a 2-pass global aggregation would be needed to fix.
## Related
Companion to #11654 — correct pagination for ORDER BY non-timestamp
column in `_search_stream`
## What
Design at: #12199
Relate to:
https://github.com/openobserve/openobserve/issues/12105https://github.com/openobserve/openobserve/issues/12202
Adds a new CLI command `gc-file-list` that deletes orphaned stream files
left on remote object storage (s3/gcs/azure).
Sometimes the compactor retention job marks files as deleted in the
`file_list` table but fails to remove the physical objects from storage,
leaving garbage behind. This command cleans up that leftover data.
## How it works
For every stream (or a single one via `-s org/stream_type/stream_name`):
- Resolves the effective data retention:
`stream_settings.data_retention` if set, otherwise
`ZO_COMPACT_DATA_RETENTION_DAYS` (mirrors the compactor `retention`
logic).
- Deletes date directories whose data is **entirely** past the retention
cutoff (`day_end <= now - retention`), so the boundary day is always
preserved.
- Preserves any day overlapping an `extended_retention_days` ("red day")
range.
- Cleans both the data dir `files/{org}/{type}/{stream}/` and the
derived index dir `files/{org}/index/{stream}_{type}/`.
## Performance
To avoid loading every object of a TB-scale stream into memory, it walks
the `year/month/day` directory tree using delimiter listings (which only
return directory names) and only does a full object listing for the day
directories it actually deletes.
Supporting changes in `infra::storage`:
- New `storage::list_dirs(account, prefix)` returning immediate child
directories (common prefixes).
- Implemented `list_with_delimiter` for the remote store (previously
`NotImplemented`).
## Usage
```
openobserve gc-file-list # all streams
openobserve gc-file-list -s default/logs/mystream # one stream
openobserve gc-file-list -s default/logs/mystream -d # dry-run preview
openobserve gc-file-list -a account2 # override storage account (file_hash setups)
```
## Safety notes
- Only works against remote object storage (errors out on local disk).
- Streams with effective retention `<= 1` day are skipped as a safety
guard.
- Recommend running with `-d` (dry-run) first to review what would be
deleted.
## Tests
Added unit tests for the date/segment parsing and retention decision
helpers (`last_segment`, `day_bounds`, `month_start`, and the
deletes-old-day / keeps-boundary-day / keeps-extended-retention-day
cases).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Harsh Mahajan
Don Kendall
yujinqiu
Shrinath RaoandClaude Sonnet 4.6
dependabot[bot]andHengfei Yang
ktx-riya
ktx-mihir
ktx-vaidehiandktx-riya
Ashish Kolhe
Huaijin
Peter Dave Hello
Loakesh Indiran
