## Problem
Histogram panels showed a false drop in the **leftmost bucket**.
`histogram()` rewrites to `date_bin(interval, _timestamp, epoch)`, which
assigns each record to the **left boundary** of its bucket. So a record
at `08:10:17` gets key `08:10:00`.
But the scan filter was `_timestamp >= start_time` where `start_time`
was the raw query time (e.g. `08:10:17`). Records in `[08:10:00,
08:10:17)` passed the bucket label check but were excluded from the scan
— making the first bucket appear partially filled.
Fixes#1753 (o2-enterprise)
## Fix
Snap `start_time` **down to the nearest bucket boundary** at the
**leader node**, before it is broadcast to all follower nodes via
`FlightSearchRequest`.
Location: `RemoteScanNodes::get_remote_node()` in
`distributed_plan/node.rs`
```rust
start_time: {
let t = self.req.time_range.as_ref().map(|x| x.0).unwrap_or(0);
if self.req.histogram_interval > 0 {
histogram_bucket_start(t, self.req.histogram_interval * 1_000_000)
} else {
t
}
},
```
The arithmetic is extracted into `histogram_bucket_start(timestamp_us,
interval_us)` in `sql/histogram.rs` so the formula lives in one place.
Doing the snap at the leader means:
- All followers receive an already-aligned `start_time`
- `SimpleHistogramVisitor` / `SimpleMultiHistogramVisitor` on followers
need no changes (their `start_time % rounding_by` is already 0)
- Covers both cluster leader and super cluster leader (both go through
`generate_remote_scan_rules → get_remote_node()`)
## Test plan
- [ ] Deploy and run a histogram query spanning a non-aligned
`start_time`
- [ ] Verify first bucket count matches adjacent full buckets
- [ ] Verify queries without `histogram_interval` are unaffected
## Summary
- Switch `ui_integration_tests`, `merge_and_upload_reports`, and
`playwright_summary` jobs from `ubicloud-standard-8` to
`repo-openobserve-standard-8`
- `build_binary` remains on `ubicloud-standard-16` (unchanged)
## Test plan
- [ ] Verify CI triggers and jobs pick up the correct `repo-openobserve`
runners
- [ ] Confirm `build_binary` still runs on ubicloud
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
- close https://github.com/openobserve/o2-enterprise/issues/1658
## Summary
Speed up tantivy index generation during compaction by indexing parquet
row_groups in parallel.
- New dispatcher in `src/service/tantivy/mod.rs` picks one of two
builders:
- **sequential** (`sequential.rs`) — legacy producer/consumer path, used
for non-parquet inputs or when `thread_num <= 1`.
- **parallel** (`parallel.rs`) — one `SingleSegmentIndexWriter` per
parquet row_group built under `spawn_blocking` +
`Semaphore(thread_num)`, then stitched via
`tantivy::indexer::merge_indices`.
- `create_tantivy_index` now takes raw parquet `Bytes` so each
per-row_group worker gets a cheap `Bytes::clone()` instead of re-reading
the file. Call sites in `src/job/files/parquet.rs` and
`src/service/compact/merge.rs` are updated.
## Config
- `ZO_COMPACT_TANTIVY_BUILDER_THREAD_NUM` (default `2`) — per-file
concurrent row_group workers. `<= 1` keeps the legacy single-threaded
path.
## Benchmark
Input: 21 row_groups, 2,672,970 rows, ~1.38 GB .ttv output.
| variant | cpu | total tantivy | peak mem | vs main (time) | vs main
(mem) |
|---|---:|---:|---:|---:|---:|
| **main (baseline)** | 2 | 108,673 ms | **10.62 GB** | — | — |
| PR, workers=2 (default) | 2 | 109,060 ms | 2.92 GB | ≈ same | **−72%**
|
| PR, workers=4 | 4 | 92,232 ms | 3.99 GB | **−14%** | **−62%** |
| PR, workers=10 | 10 | 76,167 ms | 7.75 GB | **−27%** | **−27%** |
- **Memory is the big win.** Even sequential mode (workers=1) cuts peak
from 10.6 GB → 3.0 GB (~3.6×).
- **At the default (workers=2), wall time matches main exactly while
peak memory drops 72%.** Safe upgrade.
- **Doubling workers buys time at the cost of memory** (per-row_group
segments buffer before merge). 4 workers is the best speed/memory
trade-off; ~10 is the cap (single-threaded `merge_indices` floor is ~60
s).
## Why it's correct
OpenObserve relies on `tantivy doc_id == global parquet row_index` to
map search hits back to row offsets. The parallel path preserves it
because:
1. Each per-row_group writer writes rows in stored order → local
`doc_id` == offset inside that row_group.
2. Worker results are sorted by `rg_idx` before `merge_indices`.
3. `merge_indices` stacks segments in input order (no `sort_by_field`
path in current tantivy).
4. Parquet stores row_groups in row order, so the stitched layout
matches the original global row order.
Two guards backstop this: the merged index must be a single segment, and
its `max_doc` must equal the sum of per-row_group row counts.
## Tests
- `test_row_group_parallel_preserves_row_order` — 4 row_groups, 3
workers (forces out-of-order completion), asserts every marker's `doc_id
== row_index`.
- `test_row_group_parallel_matches_sequential` — same data through both
paths yields identical `(marker → doc_id)` mappings.
- `test_row_group_parallel_single_row_group_fast_path` —
single-row_group input skips `merge_indices` and still preserves row
order.
## Summary
- Moves 3 enterprise-only tests out of OSS regression spec files into
dedicated `*-ent` spec files
- These spec files live in the OSS repo but are **not referenced in the
OSS CI workflow** — they are picked up only by the enterprise CI
- Fixes recurring failures in the OSS regression run caused by features
that don't exist in OSS builds
## Why the tests were failing in OSS
| Shard | Test | Root cause |
|---|---|---|
| Traces-Regression | Service graph side panel (Bug #11687) |
`[data-test="traces-service-graph-toggle"]` is wrapped in
`v-if="config.isEnterprise == 'true'"` in `SearchBar.vue` — element
never renders in OSS → 45s timeout |
| Logs-Regression | Scheduled Search navigation (Bug #7354) | `POST
/api/{org}/search_jobs` returns **404** in OSS — endpoint is
enterprise-only |
## Changes
**New files:**
-
`tests/ui-testing/playwright-tests/RegressionSet/logs-regression-ent.spec.js`
— Scheduled Search / `search_jobs` describe block (2 tests)
-
`tests/ui-testing/playwright-tests/RegressionSet/traces-regression-ent.spec.js`
— service graph side panel test (Bug #11687)
**Modified files:**
- `logs-regression-9044-7354.spec.js` — removed the `#7354` Scheduled
Search describe block; `#9044` scroll retention tests remain
- `traces-regression.spec.js` — removed Bug #11687 service graph test
block
The companion PR in `o2-enterprise` adds these two files to the
Logs-Regression and Traces-Regression shards of the enterprise workflow.
## Test plan
- [ ] OSS regression CI (`playwright_regression.yml`) should pass — no
more enterprise-only test failures
- [ ] Enterprise regression CI picks up `*-ent.spec.js` files via the
companion workflow PR
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Follow-up to #12249. The cross-type collision check in the file upload
handler passes the formatted `stream_name` to `get_url_jobs_for_table`,
but URL jobs are stored with the raw `table_name` from the HTTP path.
When names contain uppercase or special characters,
`format_stream_name("My_Table")` produces `"my_table"`, which won't
match the stored URL job name `"My_Table"` — causing the cross-type
protection to be silently bypassed.
## Fix
Changed `get_url_jobs_for_table(&org_id, &stream_name)` to
`get_url_jobs_for_table(&org_id, table_name.trim())` — matching how the
URL handler itself queries URL jobs at line 299.
## Before/After
```rust
// Before: formatted name passed to URL jobs lookup (WRONG)
get_url_jobs_for_table(&org_id, &stream_name)
// After: raw table_name passed (CORRECT, matches how URL handler queries)
get_url_jobs_for_table(&org_id, table_name.trim())
```
## Notes
- `ENRICHMENT_TABLES` cache key continues to use `stream_name`
(formatted) — that's correct
- `get_url_jobs_for_table` now uses `table_name.trim()` — matching the
URL handler at line 299
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Problem
Playwright CI jobs were hanging for 2–6 hours and then timing out
without running a single test. Root cause: `npx playwright install
--with-deps chromium` downloads the browser binary successfully (~3s),
but the `--with-deps` flag then runs a second `apt-get` call that waits
indefinitely for the dpkg lock held by Ubuntu's `unattended-upgrades`
background daemon on ubicloud runners.
Evidence: in a 2h8m run, Chromium downloaded at `08:51:49`, but the
`playwright install` process was still listed as a zombie when the job
was killed at `11:00:23`.
## Fix
Two-layer fix applied to all four workflow files (`playwright.yml` +
`playwright_regression.yml` in both OSS and enterprise):
**1. Kill background apt services at job start** (new first step)
```bash
sudo systemctl stop apt-daily.service apt-daily-upgrade.service unattended-upgrades.service
sudo systemctl stop apt-daily.timer apt-daily-upgrade.timer
sudo systemctl disable apt-daily.service apt-daily-upgrade.service unattended-upgrades.service
sudo flock --wait 30 /var/lib/dpkg/lock-frontend true
```
This eliminates the root cause — the dpkg lock contender is gone before
`playwright install` runs.
**2. Cache the browser binary** (`actions/cache@v4` keyed on
`package-lock.json` hash)
On cache hit, only system dependencies are installed (`playwright
install-deps chromium`), avoiding apt entirely. On cache miss, the full
`playwright install --with-deps chromium` runs with apt services already
stopped.
**3. Filter TestDino uploads to failures only**
Before uploading to TestDino, the JSON report is filtered to only
include tests with `status === 'unexpected'`. This prevents passing
tests from polluting the TestDino failure history.
## Files changed
- `.github/workflows/playwright.yml` — kill apt, cache browser,
cache-aware install, TestDino filter
- `.github/workflows/playwright_regression.yml` — kill apt, cache
browser, split into separate npm/playwright/test steps
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Bug fix for - https://github.com/openobserve/openobserve/issues/12247
- **Bug:** Creating a URL enrichment table with name `X`, then creating
a file upload enrichment also named `X` (or vice versa), silently
overwrites the first one at every storage layer.
- **Fix:** Added cross-type name collision checks at the API boundary.
The file upload handler now rejects if a URL enrichment already exists
with the same name, and the URL handler rejects if a file upload
enrichment already exists. Same-type updates (append/retry/replace) are
unaffected.
## Root Cause
Enrichment tables are uniquely identified by `(org_id, table_name)`
across all storage layers — database, caches, object store, and
coordinator keys. Both URL-based and file-upload enrichments share the
same `StreamType::EnrichmentTables` and the same key space, so creating
one with the same name as another silently clobbers the previous one.
## Changes
**1 file changed** —
[src/handler/http/request/enrichment_table/mod.rs](src/handler/http/request/enrichment_table/mod.rs)
### `save_enrichment_table` (file upload handler)
Added check after input validation: if the table name already exists in
`ENRICHMENT_TABLES` and has URL jobs, return `400` — "already exists as
a URL enrichment."
### `save_enrichment_table_from_url` (URL handler)
Added check after fetching existing jobs: if the table name exists in
`ENRICHMENT_TABLES` but has no URL jobs, return `400` — "already exists
as a file upload enrichment."
### Behavior matrix
| | Existing: URL | Existing: File Upload | Existing: None |
|---|---|---|---|
| **Create File Upload** | ❌ Reject (400) | ✅ Allow (update) | ✅ Allow |
| **Create URL** | ✅ Allow (update) | ❌ Reject (400) | ✅ Allow |
## Test Plan
- [ ] Manually verify: create a URL enrichment named `test_cross_type` →
attempt file upload with same name → should get 400 error
- [ ] Manually verify: create a file upload enrichment named
`test_cross_type2` → attempt URL creation with same name → should get
400 error
- [ ] Manually verify: same-type updates still work (file→file append,
URL→URL retry/replace)
- [ ] Playwright automation will be added in a follow-up PR in the
`RegressionSet` folder
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
- The previous PR (#12229) fixed most instances of `200GB` in `en.json`
but missed the `open_source.badge_text` key which powers the \"Get
Enterprise Edition\" dialog shown to OSS users
- Fixed: `about.enterprise_offer.open_source.badge_text` → `Free up to
50GB / day`
## Test plan
- [ ] Open the app as an OSS user (no enterprise license)
- [ ] Trigger the Enterprise upgrade dialog
- [ ] Confirm badge reads "Free up to 50GB / day"
🤖 Generated with [Claude Code](https://claude.ai/claude-code)
Fixes https://github.com/openobserve/o2-enterprise/issues/1796 (item 1:
"Reload Existing URLs" returns error).
## Summary
Commit
[`85d261b5d0`](https://github.com/openobserve/openobserve/commit/85d261b5d0e4b68dbc79ca18b844fc53e2368ee9)
(vulnerability scan fixes, Apr 14 2026) removed the `!retry` guard from
enrichment table URL validation. This broke the **reload** update mode
because the frontend sends an empty URL when retrying/reprocessing
existing URLs — the backend already has them stored and just needs to
re-trigger processing.
## Root Cause
Before the breaking commit, URL validation was guarded by `!retry`:
```rust
if !retry && let Err(err_msg) = validate_enrichment_url(&request_body.url) {
return MetaHttpResponse::bad_request(err_msg);
}
```
The security fix removed the guard under the assumption that skipping
validation on retry was an SSRF bypass vector. However, the retry flow
sends `url=""` intentionally — the existing URLs are in the database.
## Fix
Re-adds the `!retry` guard around URL validation. In retry mode, URL
validation is skipped entirely (the URLs come from DB). All other modes
(create, append, replace, replace_failed) still require a non-empty,
valid URL with full SSRF protection.
## Test Plan
- [ ] Create enrichment table from URL
- [ ] Edit table → Reload existing URLs → Should succeed (no "URL cannot
be empty")
- [ ] Edit table → Add new URL → Should still validate URL format
- [ ] Edit table → Replace all URLs → Should still validate URL format
- [ ] Edit table → Replace failed URL → Should still validate URL format
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
## Problem
`sort_by=duration` (and any non-timestamp sort) in the
`/traces/latest_stream` endpoint had three bugs:
1. All partitions collapsed into one full-range scan
2. Q1 results re-sorted by `trace_start_time`, discarding SQL engine
order
3. `fetch_size` capped at `query_default_limit` (1000), so top-k ranked
on a truncated sample
## Fix
Mirrors the approach from #11654 (_search_stream):
- Detect non-ts ORDER BY generically via \`sql_order_expr\` prefix — no
hardcoded field names
- Run Q1 across **all** partitions; feed deduped hits into a
\`TopKHeap\` bounded to \`k = from + size\`
- Q2 (span details) runs **once** after all partitions complete, over
the narrowed time window of the deliverable traces
- Final hits emitted in heap order — correct global rank by ORDER BY
column
## Refactor
Extracted `run_q2_for_traces` async helper to eliminate ~300 lines of
duplicated code between the ts-path and non-ts-path. Both paths now call
the same helper for the `TraceResponseItem` build + Q2a (per-trace
aggregates) + Q2b (per-service breakdown). Also removed the dead
`"duration"` sort arm from the ts-path and the now-unused `sort_by`
parameter from `process_latest_traces_stream`.
## Known Limitation
`zo_sql_duration` is computed per-partition, so traces whose spans
straddle a partition boundary get an underestimated duration. Rare in
practice; a 2-pass global aggregation would be needed to fix.
## Related
Companion to #11654 — correct pagination for ORDER BY non-timestamp
column in `_search_stream`
## What
Design at: #12199
Relate to:
https://github.com/openobserve/openobserve/issues/12105https://github.com/openobserve/openobserve/issues/12202
Adds a new CLI command `gc-file-list` that deletes orphaned stream files
left on remote object storage (s3/gcs/azure).
Sometimes the compactor retention job marks files as deleted in the
`file_list` table but fails to remove the physical objects from storage,
leaving garbage behind. This command cleans up that leftover data.
## How it works
For every stream (or a single one via `-s org/stream_type/stream_name`):
- Resolves the effective data retention:
`stream_settings.data_retention` if set, otherwise
`ZO_COMPACT_DATA_RETENTION_DAYS` (mirrors the compactor `retention`
logic).
- Deletes date directories whose data is **entirely** past the retention
cutoff (`day_end <= now - retention`), so the boundary day is always
preserved.
- Preserves any day overlapping an `extended_retention_days` ("red day")
range.
- Cleans both the data dir `files/{org}/{type}/{stream}/` and the
derived index dir `files/{org}/index/{stream}_{type}/`.
## Performance
To avoid loading every object of a TB-scale stream into memory, it walks
the `year/month/day` directory tree using delimiter listings (which only
return directory names) and only does a full object listing for the day
directories it actually deletes.
Supporting changes in `infra::storage`:
- New `storage::list_dirs(account, prefix)` returning immediate child
directories (common prefixes).
- Implemented `list_with_delimiter` for the remote store (previously
`NotImplemented`).
## Usage
```
openobserve gc-file-list # all streams
openobserve gc-file-list -s default/logs/mystream # one stream
openobserve gc-file-list -s default/logs/mystream -d # dry-run preview
openobserve gc-file-list -a account2 # override storage account (file_hash setups)
```
## Safety notes
- Only works against remote object storage (errors out on local disk).
- Streams with effective retention `<= 1` day are skipped as a safety
guard.
- Recommend running with `-d` (dry-run) first to review what would be
deleted.
## Tests
Added unit tests for the date/segment parsing and retention decision
helpers (`last_segment`, `day_bounds`, `month_start`, and the
deletes-old-day / keeps-boundary-day / keeps-extended-retention-day
cases).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
related to
- https://github.com/openobserve/openobserve/pull/8003
- https://github.com/openobserve/openobserve/issues/7986
# Why We Can Delete `enable_align_histogram`
## What It Was
`enable_align_histogram` was a query parameter introduced in [PR
#8003](https://github.com/openobserve/openobserve/pull/8003) to fix
[Issue #7986](https://github.com/openobserve/openobserve/issues/7986).
The bug was that in the old two-step search flow (`_search_partition` →
`_search`), the two APIs independently generated histogram intervals,
leading to mismatched time buckets and inaccurate chart rendering in
dashboards.
The flag told `_search_partition` to:
1. Align partition boundaries to histogram interval boundaries (e.g.,
round to the start of a minute/hour/day).
2. Return the computed `histogram_interval` in the response so the
frontend could pass it to `_search`.
## Why It Is Now Dead Code
### 1. The frontend no longer calls `_search_partition`
All frontend search now uses `_search_stream` (HTTP/2 streaming), which
internally calls `search_partition()` with `enable_align_histogram =
false`. The old `_search_partition` endpoint is no longer consumed by
the UI. The only frontend caller (`getValuesPartition` in
`IndexList.vue`) is defined but never invoked.
### 2. `search_stream` handles alignment inherently
In the streaming search flow, partitions themselves define the search
time ranges. Each partition search runs on the exact aligned time
boundaries returned by the partition step. There is no separate
histogram query that needs an externally provided interval — alignment
is "baked into" the partition boundaries.
### 3. Histogram queries are already covered by `is_complex_query`
For histogram queries (which have `GROUP BY`), `is_complex_query`
returns `true`. Combined with `has_ts_column = true`, this triggers
`calculate_partition_settings()` to align `min_step` to the histogram
interval — the same alignment behavior that `enable_align_histogram`
previously provided, but without the extra flag.
### 4. All backend callers pass `false`
Of the 5 callers of `search_partition()`:
| Caller | Value |
|--------|-------|
| gRPC handler | hardcoded `false` |
| streaming execution | hardcoded `false` |
| traces handler | hardcoded `false` |
| HTTP `_search_partition` | from query param (frontend always passed
`true`, but frontend no longer calls this) |
| HTTP `_search_partition_multi` | from query param (frontend always
passed `true`, but frontend no longer calls this) |
No active code path ever sets `enable_align_histogram = true`.
## What Was Removed
- `get_enable_align_histogram_from_request()` — the HTTP query param
parser
- The `enable_align_histogram` parameter from:
- `search_partition()` function signature
- `search_partition_multi()` function signature
- `calculate_partition_settings()` function signature
- The alignment logic in `search_partition()`:
- `add_mini_partition` (always `false` without the flag)
- `PartitionGenerator` align parameter (now driven by `is_histogram`
alone)
- `resp.histogram_interval` override block
- The `|| enable_align_histogram` branch in
`calculate_partition_settings`
- The query parameter from both HTTP handler `utoipa` specs
- The `false` argument from all 5 call sites
## Summary
Adds Playwright E2E regression tests for 15 verified bug fixes. Tests
are added to the existing CI matrix files:
- 6 logs tests in `logs-regression-bugs-2.spec.js`
- 9 traces tests in `traces-regression-bugs.spec.js`
Also fixes framework compliance: all raw `page.locator()` calls moved to
page object getter methods, missing assertions added, and flaky test
logic fixed in #10595.
## Logs bugs covered
| Bug | Description |
|-----|-------------|
| #11606 | include/exclude search term gets added at incorrect position
|
| #9339 | collapse or expand the indexlist wont redraw the histogram
chart |
| #7310 | search term appended instead of replaced in streams dropdown |
| #5277 | column positions change back after re-running query |
| #4426 | text wrap toggle causes column misalignment on saved view
switch |
| #4091 | cancel query button on visualize page |
## Traces bugs covered
| Bug | Description |
|-----|-------------|
| #11816 | trace API called multiple times with auto search enabled |
| #11815 | traces not sorted correctly when sorted by spans column |
| #11531 | spans tab method column references non-existent field |
| #11446 | incorrect duration value in logs detail traces tab |
| #11384 | switching organization does not clear trace explorer results
|
| #11244 | timestamp values inconsistent between traces and logs |
| #8978 | no cancel option while running a query in traces |
| #4151 | trace values not visible on hover in dark mode |
| #2887 | traces should be orderable by duration column |
## Files changed
| File | Change |
|------|--------|
| `tests/ui-testing/pages/logsPages/logsPage.js` | 9 new getter methods
for selectors |
| `tests/ui-testing/pages/tracesPages/tracesPage.js` | 14 new getter
methods for selectors |
|
`tests/ui-testing/playwright-tests/RegressionSet/logs-regression-bugs-2.spec.js`
| 6 new tests, 0 raw locators, fixed#10595 |
|
`tests/ui-testing/playwright-tests/RegressionSet/traces-regression-bugs.spec.js`
| 9 new tests, 0 raw locators |
## Test plan
All 21 tests (15 new + 6 existing) pass against the dev environment. No
new CI config needed — both spec files are already listed in
`playwright_regression.yml`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
- **Dedup**: adds `queued_files` module so the same file can only sit in
the queue once — prevents repeated alert runs re-enqueuing the same
uncached files under high blob latency
- **Priority queue OOM fix**: removes the unbounded `VecDeque`
drain-task architecture; `queue_download` now pushes directly to a
capped `VecDeque` (`FILE_DOWNLOAD_QUEUE_SIZE`), eliminating the mpsc
indirection that bypassed all back-pressure
- **LIFO fix**: corrects `pop_front` → `pop_back` so workers process the
most recently enqueued (freshest) files first, as originally intended in
#6786
- **Notify**: replaces 5s polling sleep with `tokio::sync::Notify`;
fixes missed-wakeup race and burst-drain chaining
- **Tests**: queue cap honored (normal + priority), dedup correctness
Fixes https://github.com/openobserve/openobserve/issues/11371
## Summary
- Adds 2 new E2E regression test specs to `RegressionSet/` for recently
fixed bugs
- Adds 13 new page object methods to `logsPage.js` for scroll,
pagination, scheduled search, and sidebar navigation
https://github.com/openobserve/openobserve/issues/7354https://github.com/openobserve/openobserve/issues/9044
## Tests Added
### scroll-retention-9044.spec.js (3 tests)
- Scroll resets to top when navigating page 1 → page 2 via page number
button
- Scroll resets to top when navigating page 2 → page 1
- Scroll resets to top via Next Page (>) button
### undefined-length-7354.spec.js (2 tests)
- No "Undefined Length" error after navigating Logs → Scheduled Search →
Streams → Logs
- No console errors containing "undefined" or "length" after the
reproduction path
## CI
- Updated `playwright_regression.yml` to include both new spec files in
the `Logs-Regression` matrix
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
- close https://github.com/openobserve/o2-enterprise/issues/1656
# Performance
## Datasets
3TB original data, 12 cpus
## Querys
### query 1
```sql
SELECT
histogram(_timestamp) AS zo_sql_key,
kubernetes_namespace_name AS zo_sql_breakdown,
count(*) AS zo_sql_num
FROM
default
GROUP BY
zo_sql_key, zo_sql_breakdown ORDER BY zo_sql_key DESC
```
### query 2
```sql
SELECT
histogram(_timestamp) AS zo_sql_key,
kubernetes_namespace_name AS zo_sql_breakdown,
count(*) AS zo_sql_num
FROM
default
WHERE
kubernetes_pod_name <> 'ebs-csi-controller-76499b5578-trbwq'
GROUP BY
zo_sql_key, zo_sql_breakdown ORDER BY zo_sql_key DESC
```
after filter it will keep 99.79%'s data
### query 3
```sql
SELECT
histogram(_timestamp) AS zo_sql_key,
kubernetes_namespace_name AS zo_sql_breakdown,
count(*) AS zo_sql_num
FROM
default
WHERE
match_all('zinc')
GROUP BY
zo_sql_key, zo_sql_breakdown ORDER BY zo_sql_key DESC
```
after filter it will keep 41%'s data
### query 4
```sql
SELECT
histogram(_timestamp) AS zo_sql_key,
kubernetes_namespace_name AS zo_sql_breakdown,
count(*) AS zo_sql_num
FROM
default
WHERE
match_all('datafusion')
GROUP BY
zo_sql_key, zo_sql_breakdown ORDER BY zo_sql_key DESC
```
after filter it will keep 15%'s data
### query 5
```sql
SELECT
histogram(_timestamp) AS zo_sql_key,
kubernetes_namespace_name AS zo_sql_breakdown,
count(*) AS zo_sql_num
FROM
default
WHERE
match_all('error')
GROUP BY
zo_sql_key, zo_sql_breakdown ORDER BY zo_sql_key DESC
```
after filter it will keep 0.44%'s data
## Result
### main branch
query 1: 6.3s
query 2: 9.1s
query 3: 21s
query 4: 25s
query 5: 21s
### this pr
query 1: 4.2s
query 2: 5.1s
query 3: 1.8s
query 4: 1s
query 5: 300ms
# Summary
This PR adds a **multi-histogram** tantivy index optimization that
accelerates queries of the form:
```sql
SELECT histogram(_timestamp), <secondary_field>, count(*)
FROM table WHERE <filters>
GROUP BY histogram(_timestamp), <secondary_field>
```
When a secondary index exists on the breakdown field, the aggregation
(histogram + terms) is pushed down to the tantivy index level, avoiding
a full scan of the parquet data.
## Motivation
The existing `SimpleHistogram` optimizer only handles `GROUP BY
histogram(_timestamp)` with no secondary breakdown. Queries that
additionally group by an indexed field (e.g., `level`, `status_code`)
fall back to the full scan path and miss the tantivy optimization.
## Changes
- **New `SimpleMultiHistogram` optimization mode** — extended the
`IndexOptimizeMode` enum with a `SimpleMultiHistogram(min_value,
bucket_width, num_buckets, field)` variant that carries the breakdown
field name alongside histogram parameters.
- **Plan optimizer rule** (`is_simple_multi_histogram`) — a new tree
visitor in the physical optimizer that detects `AggregateExec` with
exactly 2 group-by expressions (one `date_bin`/histogram on
`_timestamp`, one column on an indexed field) and a single `count(*)`
aggregate.
- **Tantivy-level aggregation** (`handle_simple_multi_histogram`) —
executes a nested tantivy aggregation with an outer
`HistogramAggregation` on `_timestamp` and a sub-`TermsAggregation` on
the breakdown field, returning `Vec<(timestamp, breakdown_value,
count)>`.
- **Plumbing through the stack** — wired the new mode through the
follower index optimizer, tantivy search dispatch, multi-file result
merging, result cache (`CacheEntry::MultiHistogram`), and Arrow
`RecordBatch` conversion.
- **`index_fields` propagation** — the `FollowerIndexOptimizerRule` now
receives the set of indexed fields so the multi-histogram rule can
validate that the breakdown column actually has a secondary index.
## Summary
- **Add `reportFoldersPage.js`** page object with 30+ methods for folder
CRUD, report move, search, and cross-folder toggle
- **Add `reportFolders.spec.js`** with 12 tests covering:
- **P0 Smoke**: Sidebar renders with default folder, create new folder
- **P1 Functional**: Create report via API (creates temp dashboard from
scratch), move report between folders, cross-folder search, folder
sidebar search filter
- **P2 Edge Cases**: Move button disabled when source=destination, save
disabled for empty name, cancel closes dialog, default folder has no
edit/delete options, special characters allowed in folder names
- **Cleanup**: Deletes all test reports, folders, and leaves no
resources behind
- **Add `createDashboardViaApi`** to apiCleanup.js — creates a minimal
v8 dashboard from scratch, removing dependency on pre-existing
dashboards
- **Add `createReportViaApi`**, `fetchReportFolders`,
`deleteReportFolder`, `cleanupReportFolders` methods to apiCleanup.js
- Register ReportFoldersPage in PageManager
## Approach
- Everything created from scratch — unique report via API, unique
folders via UI, all cleaned up afterward
- No dependency on pre-existing dashboards or reports
- Zero `waitForTimeout` calls — all waits use `waitForLoadState`,
`waitFor({ state: 'visible/hidden' })`, or role-based waits
- Serial execution (tests build on each other: create folder → create
report → move report → cleanup)
## Test Results
```
12 passed (2.1m)
```
## CI Integration
- Added to `.github/workflows/playwright.yml` under `Reports` testfolder
alongside existing report tests
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
In alert manager pipeline handler function, we should not clear the
trigger data (containing the period_end_time representing the timestamp
used for the last scheduled pipeline query), if the pipeline is
disabled. Because, when enabling, this should have the data to know
where to start the pipeline from.
## ENVs
Replace `ZO_DATAFUSION_FILE_STAT_CACHE_MAX_ENTRIES` to
`ZO_DATAFUSION_FILE_STAT_CACHE_MAX_SIZE`
fixed https://github.com/openobserve/openobserve/issues/11835
- Changed `datafusion_file_stat_cache_max_entries` to
`datafusion_file_stat_cache_max_size` in configuration, allowing for
memory size limits instead of entry counts.
- Implemented logic to set default cache size to 5% of total memory if
not specified.
- Added new metrics for cache hits, misses, eviction counts, and
eviction time to monitor cache performance.
- Updated cache eviction logic to track current memory usage and log
warnings when limits are exceeded.
## Summary
- Drop `PARQUET_MAX_ROW_GROUP_SIZE` from 1M to 128k. The constant was
previously load-bearing for row_id → row_group mapping, which is why the
old comment said it couldn't change.
- Write the row-group-size value in effect at index-build time into each
`.ttv` puffin file's footer (`row_group_size` property) so readers can
recover it later.
- Generate parquet access plans from tantivy hits using the **per-file**
value (carried through `TantivyResult::RowIdsBitVec` → `CacheEntry` →
`FileKey.row_group_size` → `SegmentInfo` in the SEGMENTS cache).
- Falls back to `1024 * 1024` for legacy `.ttv` files written before the
property existed, so old data keeps working unchanged.
## Why this matters
With the constant in the path, the assumed row-group size at read time
had to match the one at write time. Now each tantivy index is
self-describing, so the writer's row-group-size limit can be tuned
freely going forward without breaking row_id lookups against historical
files.
## Notes for reviewers
- No DB schema change. No parquet footer reads at query time. No
tantivy/puffin binary format break — the property is additive via
`PuffinMeta.properties` (already `#[serde(default)]`).
- `FileKey.row_group_size` is set locally on the querier after tantivy
search; it's not serialized over gRPC (parallels `segment_ids`).
- `helpers.rs::generate_parquet_access_plan` now uses the per-file size;
legacy `.ttv` falls back to `LEGACY_ROW_GROUP_SIZE = 1024 * 1024`.
## Test plan
- [x] `cargo check --all-targets --workspace` clean
- [x] `cargo test --package tantivy_utils` — all 153 tests pass
- [x] `cargo test --package config --lib test_file_key` — pass
- [ ] Ingest with the new 128k size, then search — confirm row_ids
resolve correctly
- [ ] Search against a pre-existing `.ttv` (no `row_group_size`
property) — confirm fallback path still finds rows
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fixes:
- [x] In the IAM users view, please show the roles assigned to each user
alongside the user (currently not visible). Currently we only show
system roles.
- [x] If a system role has been assigned to an existing user, there's no
way to modify it. we should allow edits.
- [x] Customer expected to be able to set up users/roles ahead of
ingestion, when we click in IAM, it goes there but when we try to
perform operations then it redirects back to data sources section.
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
## Bug
Queries with `ORDER BY` on a non-timestamp column and `size: -1`
(unlimited) returned empty results when the TopKHeap path was active.
## Root Cause
In `execution.rs`, when `size = -1`, `original_size` was computed as
`0`:
```rust
let original_size = if req.query.size > 0 { req.query.size as usize } else { 0 };
// Per-partition: size = (from + 0) = 0 → partition fetches 0 rows
```
Every partition received `size=0`, so `TopKHeap` got no rows and
returned `[]`.
## Fixes
**`streaming/execution.rs`**
- When `size=-1`: use `ZO_QUERY_DEFAULT_LIMIT` as `heap_k` instead of 0
- Keep `size=-1` for per-partition requests so each partition returns
all its data
**`streaming/sorting.rs` — `TopKHeap::new`**
- Guard: `k=0` falls back to `ZO_QUERY_DEFAULT_LIMIT` (safety net;
`execution.rs` fix prevents this in practice)
## Fixed Behavior
Queries with non-ts `ORDER BY` and `size=-1` now correctly:
- Use `ZO_QUERY_DEFAULT_LIMIT` rows as the heap bound
- Send `size=-1` to each partition (not `size=0`)
- Return results instead of `[]`
Note: CTE and subquery queries that have no `_timestamp` in the outer
result take a **single partition** path (via `ts_column=None`) and never
reach `TopKHeap` — fixed in the enterprise repo (see companion PR).
## Tests
7 new `TopKHeap` unit tests in `sorting.rs`:
- Bounded `k` evicts correctly
- `k=1000` (unlimited) keeps all rows
- `from` offset applied after drain
- ASC order
- Multi-partition push
- `size=-1` non-ts ORDER BY (all rows survive)
- `k=0` fallback guard
## Test Queries (against a stream with `level`, `_timestamp` fields)
```sql
-- Should return data, correct DESC order
SELECT level, COUNT(*) as cnt FROM "stream" GROUP BY level ORDER BY cnt DESC
-- size: -1
-- Bounded
SELECT level, COUNT(*) as cnt FROM "stream" GROUP BY level ORDER BY cnt DESC
-- size: 10, from: 0
-- Paginated
SELECT level, COUNT(*) as cnt FROM "stream" GROUP BY level ORDER BY cnt DESC
-- size: 5, from: 5
```
## Related
Enterprise companion PR: openobserve/o2-enterprise#1727 (fix: suppress
_timestamp wildcard default for CTE and inline subquery sources)
#11936
## What changed
Fixed `TraceDetails.vue` so that `traceServiceMap` captures services
from **all** root span nodes in a trace, not just the first one.
Previously, traces with two or more root spans (e.g., from misconfigured
instrumentation) would show service data only for the first root.
## Why
When a trace has multiple root spans, the service-map waterfall widget
and trace-graph display were incomplete — only the first root's services
appeared. Users investigating instrumentation issues or multi-root
traces (which the existing UI already supports rendering) were missing
service data for the second and subsequent roots.
Loakesh Indiran
Shrinath RaoandClaude Sonnet 4.6
Huaijin
Ashish Kolhe
Neha PandClaude Opus 4.7
Hengfei Yang
Harsh Mahajan
Sai Nikhil
Bhargav
Omkar Kesarkhane
udev
