related to
- https://github.com/openobserve/openobserve/pull/8003
- https://github.com/openobserve/openobserve/issues/7986
# Why We Can Delete `enable_align_histogram`
## What It Was
`enable_align_histogram` was a query parameter introduced in [PR
#8003](https://github.com/openobserve/openobserve/pull/8003) to fix
[Issue #7986](https://github.com/openobserve/openobserve/issues/7986).
The bug was that in the old two-step search flow (`_search_partition` →
`_search`), the two APIs independently generated histogram intervals,
leading to mismatched time buckets and inaccurate chart rendering in
dashboards.
The flag told `_search_partition` to:
1. Align partition boundaries to histogram interval boundaries (e.g.,
round to the start of a minute/hour/day).
2. Return the computed `histogram_interval` in the response so the
frontend could pass it to `_search`.
## Why It Is Now Dead Code
### 1. The frontend no longer calls `_search_partition`
All frontend search now uses `_search_stream` (HTTP/2 streaming), which
internally calls `search_partition()` with `enable_align_histogram =
false`. The old `_search_partition` endpoint is no longer consumed by
the UI. The only frontend caller (`getValuesPartition` in
`IndexList.vue`) is defined but never invoked.
### 2. `search_stream` handles alignment inherently
In the streaming search flow, partitions themselves define the search
time ranges. Each partition search runs on the exact aligned time
boundaries returned by the partition step. There is no separate
histogram query that needs an externally provided interval — alignment
is "baked into" the partition boundaries.
### 3. Histogram queries are already covered by `is_complex_query`
For histogram queries (which have `GROUP BY`), `is_complex_query`
returns `true`. Combined with `has_ts_column = true`, this triggers
`calculate_partition_settings()` to align `min_step` to the histogram
interval — the same alignment behavior that `enable_align_histogram`
previously provided, but without the extra flag.
### 4. All backend callers pass `false`
Of the 5 callers of `search_partition()`:
| Caller | Value |
|--------|-------|
| gRPC handler | hardcoded `false` |
| streaming execution | hardcoded `false` |
| traces handler | hardcoded `false` |
| HTTP `_search_partition` | from query param (frontend always passed
`true`, but frontend no longer calls this) |
| HTTP `_search_partition_multi` | from query param (frontend always
passed `true`, but frontend no longer calls this) |
No active code path ever sets `enable_align_histogram = true`.
## What Was Removed
- `get_enable_align_histogram_from_request()` — the HTTP query param
parser
- The `enable_align_histogram` parameter from:
- `search_partition()` function signature
- `search_partition_multi()` function signature
- `calculate_partition_settings()` function signature
- The alignment logic in `search_partition()`:
- `add_mini_partition` (always `false` without the flag)
- `PartitionGenerator` align parameter (now driven by `is_histogram`
alone)
- `resp.histogram_interval` override block
- The `|| enable_align_histogram` branch in
`calculate_partition_settings`
- The query parameter from both HTTP handler `utoipa` specs
- The `false` argument from all 5 call sites
## Summary
Adds Playwright E2E regression tests for 15 verified bug fixes. Tests
are added to the existing CI matrix files:
- 6 logs tests in `logs-regression-bugs-2.spec.js`
- 9 traces tests in `traces-regression-bugs.spec.js`
Also fixes framework compliance: all raw `page.locator()` calls moved to
page object getter methods, missing assertions added, and flaky test
logic fixed in #10595.
## Logs bugs covered
| Bug | Description |
|-----|-------------|
| #11606 | include/exclude search term gets added at incorrect position
|
| #9339 | collapse or expand the indexlist wont redraw the histogram
chart |
| #7310 | search term appended instead of replaced in streams dropdown |
| #5277 | column positions change back after re-running query |
| #4426 | text wrap toggle causes column misalignment on saved view
switch |
| #4091 | cancel query button on visualize page |
## Traces bugs covered
| Bug | Description |
|-----|-------------|
| #11816 | trace API called multiple times with auto search enabled |
| #11815 | traces not sorted correctly when sorted by spans column |
| #11531 | spans tab method column references non-existent field |
| #11446 | incorrect duration value in logs detail traces tab |
| #11384 | switching organization does not clear trace explorer results
|
| #11244 | timestamp values inconsistent between traces and logs |
| #8978 | no cancel option while running a query in traces |
| #4151 | trace values not visible on hover in dark mode |
| #2887 | traces should be orderable by duration column |
## Files changed
| File | Change |
|------|--------|
| `tests/ui-testing/pages/logsPages/logsPage.js` | 9 new getter methods
for selectors |
| `tests/ui-testing/pages/tracesPages/tracesPage.js` | 14 new getter
methods for selectors |
|
`tests/ui-testing/playwright-tests/RegressionSet/logs-regression-bugs-2.spec.js`
| 6 new tests, 0 raw locators, fixed#10595 |
|
`tests/ui-testing/playwright-tests/RegressionSet/traces-regression-bugs.spec.js`
| 9 new tests, 0 raw locators |
## Test plan
All 21 tests (15 new + 6 existing) pass against the dev environment. No
new CI config needed — both spec files are already listed in
`playwright_regression.yml`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
- **Dedup**: adds `queued_files` module so the same file can only sit in
the queue once — prevents repeated alert runs re-enqueuing the same
uncached files under high blob latency
- **Priority queue OOM fix**: removes the unbounded `VecDeque`
drain-task architecture; `queue_download` now pushes directly to a
capped `VecDeque` (`FILE_DOWNLOAD_QUEUE_SIZE`), eliminating the mpsc
indirection that bypassed all back-pressure
- **LIFO fix**: corrects `pop_front` → `pop_back` so workers process the
most recently enqueued (freshest) files first, as originally intended in
#6786
- **Notify**: replaces 5s polling sleep with `tokio::sync::Notify`;
fixes missed-wakeup race and burst-drain chaining
- **Tests**: queue cap honored (normal + priority), dedup correctness
Fixes https://github.com/openobserve/openobserve/issues/11371
## Summary
- Adds 2 new E2E regression test specs to `RegressionSet/` for recently
fixed bugs
- Adds 13 new page object methods to `logsPage.js` for scroll,
pagination, scheduled search, and sidebar navigation
https://github.com/openobserve/openobserve/issues/7354https://github.com/openobserve/openobserve/issues/9044
## Tests Added
### scroll-retention-9044.spec.js (3 tests)
- Scroll resets to top when navigating page 1 → page 2 via page number
button
- Scroll resets to top when navigating page 2 → page 1
- Scroll resets to top via Next Page (>) button
### undefined-length-7354.spec.js (2 tests)
- No "Undefined Length" error after navigating Logs → Scheduled Search →
Streams → Logs
- No console errors containing "undefined" or "length" after the
reproduction path
## CI
- Updated `playwright_regression.yml` to include both new spec files in
the `Logs-Regression` matrix
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
- close https://github.com/openobserve/o2-enterprise/issues/1656
# Performance
## Datasets
3TB original data, 12 cpus
## Querys
### query 1
```sql
SELECT
histogram(_timestamp) AS zo_sql_key,
kubernetes_namespace_name AS zo_sql_breakdown,
count(*) AS zo_sql_num
FROM
default
GROUP BY
zo_sql_key, zo_sql_breakdown ORDER BY zo_sql_key DESC
```
### query 2
```sql
SELECT
histogram(_timestamp) AS zo_sql_key,
kubernetes_namespace_name AS zo_sql_breakdown,
count(*) AS zo_sql_num
FROM
default
WHERE
kubernetes_pod_name <> 'ebs-csi-controller-76499b5578-trbwq'
GROUP BY
zo_sql_key, zo_sql_breakdown ORDER BY zo_sql_key DESC
```
after filter it will keep 99.79%'s data
### query 3
```sql
SELECT
histogram(_timestamp) AS zo_sql_key,
kubernetes_namespace_name AS zo_sql_breakdown,
count(*) AS zo_sql_num
FROM
default
WHERE
match_all('zinc')
GROUP BY
zo_sql_key, zo_sql_breakdown ORDER BY zo_sql_key DESC
```
after filter it will keep 41%'s data
### query 4
```sql
SELECT
histogram(_timestamp) AS zo_sql_key,
kubernetes_namespace_name AS zo_sql_breakdown,
count(*) AS zo_sql_num
FROM
default
WHERE
match_all('datafusion')
GROUP BY
zo_sql_key, zo_sql_breakdown ORDER BY zo_sql_key DESC
```
after filter it will keep 15%'s data
### query 5
```sql
SELECT
histogram(_timestamp) AS zo_sql_key,
kubernetes_namespace_name AS zo_sql_breakdown,
count(*) AS zo_sql_num
FROM
default
WHERE
match_all('error')
GROUP BY
zo_sql_key, zo_sql_breakdown ORDER BY zo_sql_key DESC
```
after filter it will keep 0.44%'s data
## Result
### main branch
query 1: 6.3s
query 2: 9.1s
query 3: 21s
query 4: 25s
query 5: 21s
### this pr
query 1: 4.2s
query 2: 5.1s
query 3: 1.8s
query 4: 1s
query 5: 300ms
# Summary
This PR adds a **multi-histogram** tantivy index optimization that
accelerates queries of the form:
```sql
SELECT histogram(_timestamp), <secondary_field>, count(*)
FROM table WHERE <filters>
GROUP BY histogram(_timestamp), <secondary_field>
```
When a secondary index exists on the breakdown field, the aggregation
(histogram + terms) is pushed down to the tantivy index level, avoiding
a full scan of the parquet data.
## Motivation
The existing `SimpleHistogram` optimizer only handles `GROUP BY
histogram(_timestamp)` with no secondary breakdown. Queries that
additionally group by an indexed field (e.g., `level`, `status_code`)
fall back to the full scan path and miss the tantivy optimization.
## Changes
- **New `SimpleMultiHistogram` optimization mode** — extended the
`IndexOptimizeMode` enum with a `SimpleMultiHistogram(min_value,
bucket_width, num_buckets, field)` variant that carries the breakdown
field name alongside histogram parameters.
- **Plan optimizer rule** (`is_simple_multi_histogram`) — a new tree
visitor in the physical optimizer that detects `AggregateExec` with
exactly 2 group-by expressions (one `date_bin`/histogram on
`_timestamp`, one column on an indexed field) and a single `count(*)`
aggregate.
- **Tantivy-level aggregation** (`handle_simple_multi_histogram`) —
executes a nested tantivy aggregation with an outer
`HistogramAggregation` on `_timestamp` and a sub-`TermsAggregation` on
the breakdown field, returning `Vec<(timestamp, breakdown_value,
count)>`.
- **Plumbing through the stack** — wired the new mode through the
follower index optimizer, tantivy search dispatch, multi-file result
merging, result cache (`CacheEntry::MultiHistogram`), and Arrow
`RecordBatch` conversion.
- **`index_fields` propagation** — the `FollowerIndexOptimizerRule` now
receives the set of indexed fields so the multi-histogram rule can
validate that the breakdown column actually has a secondary index.
## Summary
- **Add `reportFoldersPage.js`** page object with 30+ methods for folder
CRUD, report move, search, and cross-folder toggle
- **Add `reportFolders.spec.js`** with 12 tests covering:
- **P0 Smoke**: Sidebar renders with default folder, create new folder
- **P1 Functional**: Create report via API (creates temp dashboard from
scratch), move report between folders, cross-folder search, folder
sidebar search filter
- **P2 Edge Cases**: Move button disabled when source=destination, save
disabled for empty name, cancel closes dialog, default folder has no
edit/delete options, special characters allowed in folder names
- **Cleanup**: Deletes all test reports, folders, and leaves no
resources behind
- **Add `createDashboardViaApi`** to apiCleanup.js — creates a minimal
v8 dashboard from scratch, removing dependency on pre-existing
dashboards
- **Add `createReportViaApi`**, `fetchReportFolders`,
`deleteReportFolder`, `cleanupReportFolders` methods to apiCleanup.js
- Register ReportFoldersPage in PageManager
## Approach
- Everything created from scratch — unique report via API, unique
folders via UI, all cleaned up afterward
- No dependency on pre-existing dashboards or reports
- Zero `waitForTimeout` calls — all waits use `waitForLoadState`,
`waitFor({ state: 'visible/hidden' })`, or role-based waits
- Serial execution (tests build on each other: create folder → create
report → move report → cleanup)
## Test Results
```
12 passed (2.1m)
```
## CI Integration
- Added to `.github/workflows/playwright.yml` under `Reports` testfolder
alongside existing report tests
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
In alert manager pipeline handler function, we should not clear the
trigger data (containing the period_end_time representing the timestamp
used for the last scheduled pipeline query), if the pipeline is
disabled. Because, when enabling, this should have the data to know
where to start the pipeline from.
## ENVs
Replace `ZO_DATAFUSION_FILE_STAT_CACHE_MAX_ENTRIES` to
`ZO_DATAFUSION_FILE_STAT_CACHE_MAX_SIZE`
fixed https://github.com/openobserve/openobserve/issues/11835
- Changed `datafusion_file_stat_cache_max_entries` to
`datafusion_file_stat_cache_max_size` in configuration, allowing for
memory size limits instead of entry counts.
- Implemented logic to set default cache size to 5% of total memory if
not specified.
- Added new metrics for cache hits, misses, eviction counts, and
eviction time to monitor cache performance.
- Updated cache eviction logic to track current memory usage and log
warnings when limits are exceeded.
## Summary
- Drop `PARQUET_MAX_ROW_GROUP_SIZE` from 1M to 128k. The constant was
previously load-bearing for row_id → row_group mapping, which is why the
old comment said it couldn't change.
- Write the row-group-size value in effect at index-build time into each
`.ttv` puffin file's footer (`row_group_size` property) so readers can
recover it later.
- Generate parquet access plans from tantivy hits using the **per-file**
value (carried through `TantivyResult::RowIdsBitVec` → `CacheEntry` →
`FileKey.row_group_size` → `SegmentInfo` in the SEGMENTS cache).
- Falls back to `1024 * 1024` for legacy `.ttv` files written before the
property existed, so old data keeps working unchanged.
## Why this matters
With the constant in the path, the assumed row-group size at read time
had to match the one at write time. Now each tantivy index is
self-describing, so the writer's row-group-size limit can be tuned
freely going forward without breaking row_id lookups against historical
files.
## Notes for reviewers
- No DB schema change. No parquet footer reads at query time. No
tantivy/puffin binary format break — the property is additive via
`PuffinMeta.properties` (already `#[serde(default)]`).
- `FileKey.row_group_size` is set locally on the querier after tantivy
search; it's not serialized over gRPC (parallels `segment_ids`).
- `helpers.rs::generate_parquet_access_plan` now uses the per-file size;
legacy `.ttv` falls back to `LEGACY_ROW_GROUP_SIZE = 1024 * 1024`.
## Test plan
- [x] `cargo check --all-targets --workspace` clean
- [x] `cargo test --package tantivy_utils` — all 153 tests pass
- [x] `cargo test --package config --lib test_file_key` — pass
- [ ] Ingest with the new 128k size, then search — confirm row_ids
resolve correctly
- [ ] Search against a pre-existing `.ttv` (no `row_group_size`
property) — confirm fallback path still finds rows
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
fixes:
- [x] In the IAM users view, please show the roles assigned to each user
alongside the user (currently not visible). Currently we only show
system roles.
- [x] If a system role has been assigned to an existing user, there's no
way to modify it. we should allow edits.
- [x] Customer expected to be able to set up users/roles ahead of
ingestion, when we click in IAM, it goes there but when we try to
perform operations then it redirects back to data sources section.
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
## Bug
Queries with `ORDER BY` on a non-timestamp column and `size: -1`
(unlimited) returned empty results when the TopKHeap path was active.
## Root Cause
In `execution.rs`, when `size = -1`, `original_size` was computed as
`0`:
```rust
let original_size = if req.query.size > 0 { req.query.size as usize } else { 0 };
// Per-partition: size = (from + 0) = 0 → partition fetches 0 rows
```
Every partition received `size=0`, so `TopKHeap` got no rows and
returned `[]`.
## Fixes
**`streaming/execution.rs`**
- When `size=-1`: use `ZO_QUERY_DEFAULT_LIMIT` as `heap_k` instead of 0
- Keep `size=-1` for per-partition requests so each partition returns
all its data
**`streaming/sorting.rs` — `TopKHeap::new`**
- Guard: `k=0` falls back to `ZO_QUERY_DEFAULT_LIMIT` (safety net;
`execution.rs` fix prevents this in practice)
## Fixed Behavior
Queries with non-ts `ORDER BY` and `size=-1` now correctly:
- Use `ZO_QUERY_DEFAULT_LIMIT` rows as the heap bound
- Send `size=-1` to each partition (not `size=0`)
- Return results instead of `[]`
Note: CTE and subquery queries that have no `_timestamp` in the outer
result take a **single partition** path (via `ts_column=None`) and never
reach `TopKHeap` — fixed in the enterprise repo (see companion PR).
## Tests
7 new `TopKHeap` unit tests in `sorting.rs`:
- Bounded `k` evicts correctly
- `k=1000` (unlimited) keeps all rows
- `from` offset applied after drain
- ASC order
- Multi-partition push
- `size=-1` non-ts ORDER BY (all rows survive)
- `k=0` fallback guard
## Test Queries (against a stream with `level`, `_timestamp` fields)
```sql
-- Should return data, correct DESC order
SELECT level, COUNT(*) as cnt FROM "stream" GROUP BY level ORDER BY cnt DESC
-- size: -1
-- Bounded
SELECT level, COUNT(*) as cnt FROM "stream" GROUP BY level ORDER BY cnt DESC
-- size: 10, from: 0
-- Paginated
SELECT level, COUNT(*) as cnt FROM "stream" GROUP BY level ORDER BY cnt DESC
-- size: 5, from: 5
```
## Related
Enterprise companion PR: openobserve/o2-enterprise#1727 (fix: suppress
_timestamp wildcard default for CTE and inline subquery sources)
#11936
## What changed
Fixed `TraceDetails.vue` so that `traceServiceMap` captures services
from **all** root span nodes in a trace, not just the first one.
Previously, traces with two or more root spans (e.g., from misconfigured
instrumentation) would show service data only for the first root.
## Why
When a trace has multiple root spans, the service-map waterfall widget
and trace-graph display were incomplete — only the first root's services
appeared. Users investigating instrumentation issues or multi-root
traces (which the existing UI already supports rendering) were missing
service data for the second and subsequent roots.
Design at: feat: impl tantivy footer cache
- close https://github.com/openobserve/o2-enterprise/issues/1684
### 1. Generic Bytes Cache (`infra::cache::bytes_cache::BytesCache`)
A new reusable cache for byte buffers, tagged so multiple caches can
coexist with independent metrics.
```
┌──────────────────────────────────────────┐
│ BytesCache │
│ │
│ DashMap<String, Bytes> (readers) │
│ Mutex<VecDeque<String>> (cacher/FIFO) │
│ AtomicI64 (current_memory)│
│ usize (max_bytes) │
│ String (tag) │
│ │
│ get(key) → Option<Bytes> │
│ put(key, value) → evict if over limit │
│ evict() → batch-evict 5% oldest entries │
└──────────────────────────────────────────┘
```
**Key characteristics:**
- **Concurrency:** `DashMap` for reads, `parking_lot::Mutex` around the
FIFO queue for writes.
- **Eviction:** FIFO (not LRU), evicting 5% of entries in a batch when
memory exceeds `max_bytes`. Batch eviction avoids locking on every
insertion while staying bounded.
- **Metrics:** Per-tag gauges for memory size, entry count; counters for
hits, misses, GC runs; histogram for GC time.
- **Use case flexibility:** The `tag` parameter lets different
subsystems reuse the same cache code (e.g., `"tantivy_footer_cache"`,
`"puffin_footer_cache"`).
### 2. FooterCache L2 (`FOOTER_DATA_CACHE`)
A global `BytesCache` instance wraps the `FooterCache` as an L2 cache:
```
Query
│
▼
FooterCache::from_directory(dir, cache_key)
│
├─ FOOTER_DATA_CACHE.get(cache_key) ──► hit → deserialize & return
│
└─ miss
│
├─ read .footer_cache file from directory
├─ FOOTER_DATA_CACHE.put(cache_key, bytes) ← store for next time
└─ deserialize & return
```
**Serialization format** (binary, no external dependencies):
```
┌──────────────────────┐
│ raw footer data │ ← concatenated OwnedBytes for each (file, range)
├──────────────────────┤
│ JSON metadata │ ← FooterCacheMeta { files: Map<path, Vec<Item>> }
├──────────────────────┤
│ u64 footer_offset │ ← LE byte offset to metadata
├──────────────────────┤
│ u32 version (1) │ ← LE version for forward compatibility
└──────────────────────┘
```
**Configuration:**
- `ZO_INVERTED_INDEX_FOOTER_CACHE_MAX_SIZE` (env): max MB for the L2
`BytesCache`, default 5% total memory.
### 3. Puffin Footer Caching
The Puffin footer reader (`PuffinFooterBytesReader`) now also uses
`FOOTER_DATA_CACHE`. On first read, the footer tail + payload are
combined and cached under the key `"puffin_footer_{location}"`.
Subsequent reads for the same location skip the storage round-trip
entirely.
### 4. Parallel Warm-Up
`warm_up_terms` now dispatches all four warm-up categories concurrently
via `tokio::try_join!` instead of awaiting them sequentially:
```
Before (sequential):
warm_up_fields → warm_up_fields_term → warm_up_terms → warm_up_fast_fields
After (parallel):
┌─ warm_up_terms ──────────┐
├─ warm_up_fields ─────────┤
├─ warm_up_fields_term ────┤
└─ warm_up_fast_fields ────┘
all joined
```
## Configuration Reference
| Env Variable | Default | Description |
|---|---|---|
| `ZO_INVERTED_INDEX_FOOTER_CACHE_MAX_SIZE` | 5% total memory | Max
memory for the footer L2 bytes cache |
## Metrics Reference
| Metric | Labels | Type | Description |
|---|---|---|---|
| `zo_bytes_cache_memory_size` | `tag` | Gauge | Current memory used by
a bytes cache |
| `zo_bytes_cache_entry_count` | `tag` | Gauge | Number of entries in a
bytes cache |
| `zo_bytes_cache_gc_time` | `tag` | Histogram | Time spent per GC
eviction run |
| `zo_bytes_cache_gc_count` | `tag` | Counter | Total number of GC
eviction runs |
| `zo_bytes_cache_hits_total` | `tag` | Counter | Total cache hits |
| `zo_bytes_cache_miss_total` | `tag` | Counter | Total cache misses |
## Summary
- Restores `applyQueryButton` call in `initQueryBuilderTestLite` — the
initial match_all search serves as a UI priming step that ensures the
auto-search from stream selection has completed before the test runs its
own query
- Adds `runQueryAndWaitForResults` before `clickBuildToggle` in SQL OFF
mode tests — matching the flow in `setupQueryAndSwitchToBuild` so the
builder has query results to parse into filters
## Root Cause
PR #11797 moved ingestion from `beforeEach` to `beforeAll` and
introduced `initQueryBuilderTestLite` which skipped the initial
match_all search. Two issues emerged:
1. **No UI priming**: After `selectStream`, an auto-search fires
automatically. Without waiting for it to complete, the test's own query
could run while the UI was in an intermediate state, causing the builder
to parse zero filter conditions.
2. **SQL OFF tests skipped query execution**: These tests set search bar
content and clicked build toggle without running the query first. The
builder had no executed query results to parse into filters.
## Changes
| File | Change |
|------|--------|
| `queryBuilder-helpers.js` | Add `applyQueryButton(pm)` to
`initQueryBuilderTestLite` (same as full init) |
| `logsQueryBuilder-filters-advanced.spec.js` | Add
`runQueryAndWaitForResults()` before `clickBuildToggle()` in 2 SQL OFF
tests |
## Test Results
- Local (5 workers, 77 tests): **74 passed**, 3 pre-existing
intermittent failures (CTE chart type, auto-mode chart type, NOT LIKE
filter parsing)
- The 8 CI flaky tests from #11797 are all in the passing set
## Test plan
- [ ] CI: Logs-Builder-Basic and Logs-Builder-Advanced shards pass with
0 flaky tests
- [ ] All 77 QB tests pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
On my local machine, merging a 256GB file into 10GB files with 6 merge
threads requires too much memory. In the current main branch, each merge
thread uses about 8GB of memory, so 6 threads require 48GB in total.
Since my machine has only 32GB of memory, it starts using swap, which
makes the merge process slow.
this pr reduce the memory usage for merge files, that speed up the merge
speed.
Note: if the merge do not trigger the swap the merge speed almost same,
because most of time spend for merge parquet is write parquet file,
collect all data by datafusion is only 10%-20%.
## main
2026-05-15T10:02:45.357306+00:00 INFO
openobserve::service::search::datafusion::peak_memory_pool: [trace_id ]
DataFusion peak memory usage: 9093509452 bytes (8672.25 MB)
2026-05-15T10:02:45.369673+00:00 INFO
openobserve::service::compact::merge: [COMPACTOR:WORKER:0] merged 20
files into a new file:
files/default/logs/default/2026/04/29/08/74609930655246581765c7b.parquet,
original_size: 10250729590, compressed_size: 26601268, took: 29443 ms
## this pr
2026-05-15T10:05:27.015179+00:00 INFO
openobserve::service::search::datafusion::peak_memory_pool: [trace_id ]
DataFusion peak memory usage: 65057480 bytes (62.04 MB)
2026-05-15T10:05:27.015429+00:00 INFO
openobserve::service::compact::merge: [COMPACTOR:WORKER:3] merged 20
files into a new file:
files/default/logs/default/2026/04/29/08/746099374351712256066ac.parquet,
original_size: 10453255400, compressed_size: 25995715, took: 4811 ms
Summary
Migrated the "Add Dashboard from Gallery" feature from GitHub API to S3
(direct bucket access), removing the GitHub rate-limit dependency, and
updated all corresponding test mocks to match the new S3 XML response
format.
What changed
1. Data source: GitHub API → S3 List Objects V2
The gallery was previously fetched from
api.github.com/repos/openobserve/dashboards/contents, which is subject
to GitHub's rate limiting (60 req/hr for unauthenticated requests). It
now reads directly from the S3 bucket openobserve-datasources-bucket via
the List Objects V2 XML API.
loadDashboards() — folder listing now calls:
GET
https://...s3.amazonaws.com/?list-type=2&prefix=dashboards/&delimiter=/
And parses the XML response via parseS3Folders(), which reads
<CommonPrefixes><Prefix> elements.
handleNext() — file listing inside a folder now calls the same S3
endpoint with a deeper prefix, parsed via parseS3Files() which reads
<Contents><Key> elements.
confirmAdd() — dashboard JSON download changed from
raw.githubusercontent.com/... to the direct S3 object URL.
2. Test mock format: JSON → S3 XML
All tests that mocked fetch were still returning the old GitHub JSON
format ([{name, type}]). Since the component now calls
DOMParser.parseFromString(xml, "application/xml"), the JSON strings were
unparseable as XML, resulting in empty dashboards arrays and causing the
"sort dashboards alphabetically" test to fail with undefined.
Added two helpers (s3FolderListXml, s3FileListXml) and updated every
affected fetch mock to return valid S3 List Objects V2 XML. Also
strengthened a few assertions that were previously passing vacuously (an
empty array also "doesn't contain .github").
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
#11841
## What changed
Added `queryParams.stream = effectiveStreamName.value` to the share URL
generation in `TraceDetails.vue` so the stream name is included when
generating a share/short URL.
## Why
When sharing a URL from the Trace Details view, the stream name
parameter was missing from the generated URL. This caused the shared
link to open without the correct stream context, making it unable to
display the trace properly on navigation.
## Approach
The stream name is already resolved by the `effectiveStreamName`
computed property — which handles both embedded mode (via
`props.streamNameProp`) and standalone mode (via
`router.currentRoute.value.query.stream`). The fix simply passes this
value into the `queryParams` object before the URL is serialized.
## Problem
`ORDER BY <non-timestamp col>` (e.g. `ORDER BY duration DESC`) gave
wrong results and risked OOM because per-partition \`hits_to_skip\`
assumed rows are time-ordered — but the globally k-th row can be in any
partition.
## Solution
Each partition fetches its local top-`k = from+size` rows. A
**TopKHeap** on the leader merges them globally, skips `from`, returns
`size`. Heap is bounded to `k` elements at all times — O(k) memory
regardless of partition count.
**Multi-column ORDER BY** (`ORDER BY duration DESC, _timestamp DESC`) is
fully honored: heap comparison and final sort respect all columns in
order, using secondary columns as tiebreakers.
## Scenario behavior
| Query | Behavior |
|---|---|
| `ORDER BY _timestamp DESC` | Unchanged — per-partition path |
| `ORDER BY duration DESC` | Heap merge path — correct pagination |
| `ORDER BY duration DESC, _timestamp DESC` | Heap merge; ties in
`duration` broken by `_timestamp` |
| `ORDER BY duration DESC` page 2 | Heap collects global top-k, skips
`from`, returns `size` |
| JOIN / subquery / GROUP BY | `is_aggregate=true` → single partition
(unchanged) |
| `ZO_DISABLE_PARTITIONS_FOR_NON_TS_ORDER_BY=true` | Forces 1 partition
(circuit breaker preserved) |
| Partial cache hit + non-ts ORDER BY | TODO — pre-existing bug, no
regression |
## What is NOT changed
- `ORDER BY _timestamp` execution path
- `_search` (non-streaming) — DataFusion handles it correctly
- Aggregate / histogram queries
#11816
## What changed
Added an optional `skipSearch` parameter to `SearchBar.vue`'s
`applyFilters` method. When `true`, the method applies filters to state
without emitting a `searchdata` event. In `Index.vue`,
`onMetricsFiltersUpdated` now passes `skipSearch=true` when live mode is
ON, letting the datetime trigger (`triggerAbsoluteQueryDebounced`)
handle the search call. When live mode is OFF, `skipSearch=false` so the
search still fires (since the datetime trigger won't run).
## Why
When auto-search is enabled and a user drags on RED charts to select a
time range, both `applyFilters` and `triggerAbsoluteQueryDebounced` were
called simultaneously — each triggering a separate `searchdata` API
call. Per the user, "searchdata should only be called from
triggerAbsoluteQueryDebounced." The fix ensures only one search call
happens by having `applyFilters` skip its own search when live mode is
on.
#11815
## What changed
Fixed a bug in `TenstackTable.vue` where the virtual scroll row click
handler used `tableRows[virtualRow.index]` instead of
`formattedRows[virtualRow.index]?.original` to emit row data. When the
table is sorted server-side, `tableRows` retains the original unsorted
order while `formattedRows` reflects the sorted order, so clicking the
first visible row emitted data from the wrong source row. Also added a
regression test for the sorted virtual-scroll case.
## Why
When a user sorted the Traces search results by the spans column and
clicked the first visible row, the `click:dataRow` event emitted the
original unsorted row (e.g., "alpha" at index 0) instead of the sorted
row (e.g., "gamma" at index 0 after sorting desc). This caused
downstream consumers to receive incorrect row data.
Huaijin
Bhargav
Hengfei Yang
Neha PandClaude Opus 4.7
Loakesh Indiran
Sai Nikhil
Subhra264
Omkar Kesarkhane
udev
Harsh Mahajan
Ashish Kolheandgithub-actions[bot]
Shrinath RaoandClaude Opus 4.7
Abhay Padamani
