- [x] ~add sort when to many groups, we can do this, but for now do not
do this~
- [x] remove the unsafe for memory pool
- [x] add approx_distinct improve, already fork, will add in next pr
- [x] fork vortex to add cast fix, and export
`.exclude_schemes([IntDictScheme.id()])`
- [x] ~enable_physical_uncorrelated_scalar_subquery test~
## Problem
When **"Correlate without service attribute"** (`service_optional =
true`) is enabled, the same physical service can appear as two (or more)
separate entries in **Discovered Services**.
**Root cause:** `ServiceKey` is always `(service_name, set_id,
disambiguation)`. When a log has no `service.name` attribute, the
processor falls back to the stream name (e.g. `repro_logs`) as
`service_name`. A trace for the same workload carries the real
`service.name` (e.g. `ecs-api-service`). Both entries share identical
`(set_id, disambiguation)` but are stored as separate rows — they never
merge.
**What the DB looks like (bug):**
| service_name | set_id | disambiguation | logs_streams | traces_streams
|
|---|---|---|---|---|
| `ecs-api-service` | `kubernetes` | `{"k8s-namespace":"ecs-prod"}` |
`[]` | `["repro_traces"]` |
| `repro_logs` | `kubernetes` | `{"k8s-namespace":"ecs-prod"}` |
`["repro_logs"]` | `[]` |
Two rows. `disambiguation` is byte-for-byte identical. Only
`service_name` differs.
In production this gets worse — ECS metric streams (which also have no
`service.name`) each become their own phantom row under the same
disambiguation. We observed 85 rows for a single ECS service, with only
2 rows carrying actual streams.
## Fix
`GET /api/:org_id/service_streams` now performs a read-time merge when
`service_optional = true`:
1. Fetches raw rows from the DB (unchanged)
2. Loads the org's service identity config (already cached)
3. If `service_optional = true` — groups rows by `(set_id,
disambiguation)`, unions their `logs_streams` / `metrics_streams` /
`traces_streams` arrays, and picks the real service name (the one that
does NOT appear in its own stream arrays) as the display name
4. Returns the merged list
**After fix — what the API returns:**
```json
[{
"service_name": "ecs-api-service",
"set_id": "kubernetes",
"disambiguation": {"k8s-namespace": "ecs-prod"},
"logs_streams": ["repro_logs"],
"traces_streams": ["repro_traces"]
}]
```
One entry. Real service name. All streams unified.
The DB rows are not changed — merge is purely at read time in the
handler. When `service_optional = false` the handler returns raw rows as
before (no change in behaviour).
Fixes#1848
This adds a UI and API based trigger for re triggering license check
limits, so we do not have to restart nodes
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Currently, on the roles edit page, the frontend makes role permission
api for each and every resources. So, for current 41 resources, there
are 41 concurrent api calls, each going through the openfga rbac check
middleware. That means, there are 41 concurrent redundant openfga
checks, and then 41 openfga query to get the role permissions.
This pr adds another api `org_id/roles/role_name/permissions` that
combines all the resource permissions of the given role name and return
the same eliminating the need for making api calls for each rbac
resource.
Adds support to show tentative cost in usage page in cloud
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
## Summary
- **Cloud visibility**: `list()` in `src/service/db/alerts/templates.rs`
was gating `DEFAULT_ORG` templates behind `!cfg!(feature = \"cloud\")`,
so any non-default org on cloud received an empty template list. Removed
the flag entirely — all `DEFAULT_ORG` templates (prebuilt and custom)
are now visible to every org, matching the existing ENT behaviour. This
is safe because cloud org identifiers are UUID-assigned; no regular user
can create an org named `default`.
- **Prebuilt templates fully read-only for non-root**: previously only
delete was blocked (403); updates were silently allowed. Now both
`save()` and `delete()` return `PrebuiltReadOnly` (403) for non-root
callers. Root users retain full access via `is_root_user()` bypass
(consistent with existing patterns in the codebase). Create with a
reserved name returns `ReservedName` (400) for everyone including root.
- **Frontend hardcoding removed**: `AddDestination.vue` was using a
`DEFAULT_TEMPLATE_SENTINEL` (`"__default__"`) and constructing prebuilt
template names client-side. Dropdown is now populated from the
`isPrebuilt: true` entries returned by `GET /alerts/templates`.
`TemplateList.vue` edit button now matches delete: shown-but-disabled
with `systemReadOnly` tooltip for prebuilt rows.
## Root cause
The `!cfg!(feature = "cloud")` flag in `list()` was added to prevent
`DEFAULT_ORG` data bleeding across cloud tenants. This correctly scopes
custom templates but also hides system prebuilt templates that every org
needs. Since prebuilt templates are now enforced as read-only
(non-root), exposing all DEFAULT_ORG templates to all orgs carries no
cross-tenant mutation risk.
## Test plan
- [ ] Non-default org on cloud: `GET /api/{org}/alerts/templates`
returns `prebuilt_slack`, `prebuilt_email`, etc. with `isPrebuilt: true`
- [ ] Non-root user: `PUT /api/{org}/alerts/templates/prebuilt_slack`
returns 403
- [ ] Non-root user: `DELETE /api/{org}/alerts/templates/prebuilt_slack`
returns 403
- [ ] Root user: `PUT /api/{org}/alerts/templates/prebuilt_slack`
succeeds
- [ ] Root user: `DELETE /api/{org}/alerts/templates/prebuilt_slack`
succeeds
- [ ] Any user: `POST /api/{org}/alerts/templates` with name
`prebuilt_slack` returns 400
- [ ] Destinations page: prebuilt destination template dropdown shows
`prebuilt_<type>` sourced from API
- [ ] `TemplateList`: edit button disabled (not hidden) for prebuilt
rows, with "System Read Only" tooltip
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
## Inferred services for traces
Design at: #12515
Uninstrumented dependencies (databases, redis, kafka, external APIs) are
invisible in trace views today because OpenObserve only surfaces the
instrumented `service_name`. Users ask "why can't you see we call the db
or redis?". This adds **inferred services**: we derive a minimal
identity for those dependencies from CLIENT/PRODUCER span peer
attributes and surface them in the traces list and the service map.
### What's included
- **Ingestion** — derive three fields on CLIENT(3)/PRODUCER(4) spans
following OTel semantic conventions (current + legacy names):
`infer_service_name`, `infer_service_type`
(`database`/`queue`/`rpc`/`external`), `infer_service_system` (e.g.
`postgresql`/`kafka`/`grpc`/`http`). `peer.service` wins naming; ports
are stripped and bare IPs redacted to avoid one node per pod IP. Wired
into both `handle_otlp_request` and `ingest_json`; the three field names
are added to `BLOCK_FIELDS` so a user attribute of the same name can't
shadow them.
- **Traces list** — the per-service time/cost breakdown now groups by
`COALESCE(infer_service_name, service_name)` (schema-gated) and returns
`service_type`, so dependencies appear in the distribution and the UI
can render them dotted.
- **Service graph** — a second, schema-gated SQL over CLIENT/PRODUCER
spans (anti-joined against instrumented services so a dep that *is*
instrumented elsewhere isn't double-counted) emits inferred edges with
`connection_type`; `ServiceNode.service_type` /
`ServiceEdge.connection_type` carry the inferred category for dotted
nodes/edges. The dependency has no server span, so its latency is the
client span's own duration.
### Backward compatibility
Every new SQL path is gated on the field's presence in the stream
schema, so streams that predate the feature (or never call an
uninstrumented dependency) generate byte-identical SQL — no "field not
found". Old parquet reads the field as NULL → `COALESCE` falls back to
`service_name` gracefully.
### Companion PR
Service-graph topology tagging (`build_topology`) lives in the
enterprise repo: **openobserve/o2-enterprise#1876**. Build/merge the two
together.
### Testing
- `cargo check` / clippy / fmt clean
- unit tests: inferred derivation (20), traces-list breakdown
serialization, config `service_graph` types; full traces suite (417)
passes
- verified end-to-end locally: an OTLP trace with a redis CLIENT span
shows up in `GET /traces/latest` as `service_type: database` and as a
dotted node on the service map
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes#12461
## Problem
PromQL queries in a super-cluster returned data **only from the hub
cluster** — remote (spoke) clusters were silently skipped. SQL queries
across the same super-cluster worked correctly.
**Root cause:** The `data` gRPC handler on the spoke never registered
the incoming `trace_id` in `SEARCH_SERVER`. When the PromQL engine
called `insert_sender(trace_id)` to attach its abort channel, it found
no entry and immediately returned `SearchCancelQuery` (error code
20009). The spoke returned empty data with no visible error — the hub
received nothing from the spoke and returned only its local results.
The `query` gRPC handler already had the correct registration. The
`data` handler was missing it.
## Fix
Before spawning the data fetch task on the spoke, the `data` gRPC
handler now registers the incoming `trace_id` in `SEARCH_SERVER` as a
follower task — exactly the same registration the `query` handler
already performed. This allows the PromQL engine to successfully attach
its abort sender and proceed with execution. After the task completes,
the trace_id is removed from `SEARCH_SERVER` to avoid memory leaks.
## Test Queries
Ingest a metric with a distinguishing label to both hub and spoke, wait
~30s for WAL flush, then run:
**PromQL — before fix:** only hub cluster returned, spoke missing
**PromQL — after fix:** both hub and spoke returned
```bash
curl -s "http://<hub-host>/api/<org>/prometheus/api/v1/query_range" \
-u '<user>:<pass>' \
--data-urlencode "query=sum by (cluster) (total_payment_req)" \
--data-urlencode "start=$(($(date +%s) - 3600))" \
--data-urlencode "end=$(date +%s)" \
--data-urlencode "step=60"
```
**SQL — works both before and after (baseline check):**
```bash
curl -s -X POST "http://<hub-host>/api/<org>/_search?type=metrics" \
-H "Authorization: Basic <token>" \
-H "Content-Type: application/json" \
-d '{
"query": {
"sql": "SELECT cluster, SUM(value) as total FROM \"total_payment_req\" GROUP BY cluster",
"start_time": '"$(($(date +%s) - 3600))"'000000,
"end_time": '"$(date +%s)"'000000,
"size": 10
}
}'
```
- close https://github.com/openobserve/o2-enterprise/issues/1865
## TODO
- [x] make split_batch_for_grpc_response based on the real memory sizew
- [x] coalesce and gc view before send to leader
```
SELECT trace_id, count(*) as cnt, max(_timestamp) as ts FROM "k8s_logs" group by trace_id order by cnt desc limit 10
utf8view enable:
main: 30s (15k batches send to leader)
this branch: 15s (1.5k batches send to leader)
utf8view disable:
main: 13s (42k batches send to leader)
this branch: 12s (1.5k batches send to leader)
```
one found is the AggregateExec's time_calculating_group_ids, utf8 is
faster than utf8view
## Summary
Fixes#11904 — streaming search kept running on the queriers even after
the user closed the browser tab.
The SSE search endpoints spawn the query in a detached task and stream
results back over `text/event-stream`. When the client disconnects, the
response stream is dropped and we stop *sending* results, but the
distributed query continued executing on the queriers, wasting resources
on results nobody will read.
This wires client disconnect into the existing query-cancellation
machinery.
## What changed
- **`SearchStreamGuard`** (`search/utils.rs`) — moved into the response
stream's closure so it shares the stream's lifetime. It marks itself
finished when a terminal event (`Done` / `Error` / `Cancelled`) is
produced; if the stream is dropped before that (client gone), its `Drop`
fires the cancel for the parent `trace_id`.
- **`cancel_query_internal`** (`search/query_manager.rs`) — best-effort
cancel for internal callers (no HTTP response), reusing the existing
super_cluster/regular split.
- Wired the guard into **`search_http2_stream`**,
**`values_http2_stream`**, and **`search_multi_stream`**.
Cancellation is keyed by the parent `trace_id`; the query manager
removes entries by prefix, so multi-stream sub-queries
(`{trace_id}-q{n}`) are covered by a single cancel.
## Notes
- Distributed cancellation is an **enterprise** feature (the whole
`cancel_query` / `QueryManager` path is enterprise-gated). On the
open-source build the guard only logs the disconnect; OSS already stops
sending once the channel closes.
- Normal completion never triggers a cancel (terminal-event →
`mark_finished`). Worst case on an edge race is a harmless no-op cancel
for an already-finished `trace_id`.
- Complements the existing manual "Cancel" button — same underlying
machinery, triggered automatically on disconnect.
This branch also includes an unrelated config tweak (`fix: enable
compact fast mode for local`) enabling compact `fast_mode` automatically
when running in local mode.
## Testing
- `cargo check` (OSS) and `cargo check --features enterprise` both clean
- Unit test for the guard state machine passes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
- [x] Bulk ingestion panic
- [x] Create function panic
- [x] Dashboards/Alerts/Incidents panic when list data
- [x] Match http response error code
- [x] Skip some endpoint that fuzz can't generate data
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes#12401 Design at: #12402
## What
Adds timezone support to the `histogram()` SQL function so buckets align
to a user's local time instead of always UTC.
## How to use
### 1. Per-query — 3rd argument to `histogram()`
```sql
SELECT histogram(_timestamp, '1 hour', '+08:00') AS key, count(*) FROM "logs" GROUP BY key;
SELECT histogram(_timestamp, '1 day', 'America/Los_Angeles') AS key, count(*) FROM "logs" GROUP BY key;
```
### 2. Per-request — `timezone` field in the search payload
Applies to every `histogram()` in the query that has no 3rd argument
(including the bare 1-arg `histogram(_timestamp)` form):
```jsonc
POST /api/{org}/_search
{
"query": {
"sql": "SELECT histogram(_timestamp) AS key, count(*) FROM \"logs\" GROUP BY key",
"start_time": 1704067200000000,
"end_time": 1704153600000000,
"timezone": "+08:00"
}
}
```
or
```jsonc
POST /api/{org}/_search
{
"query": {
"sql": "SELECT histogram(_timestamp) AS key, count(*) FROM \"logs\" GROUP BY key",
"start_time": 1704067200000000,
"end_time": 1704153600000000,
"timezone": "America/Los_Angeles"
}
}
```
### Precedence
```
histogram() 3rd arg > request "timezone" field > UTC (default, unchanged)
```
### Accepted formats
- **Fixed offset:** `+08:00`, `-05:30`, `UTC`, `""` (empty = UTC)
- **IANA name:** `America/Los_Angeles`, `Asia/Kolkata`, … (resolved via
`chrono-tz`)
### Result semantics
The returned bucket value is **local wall-clock time** (same convention
as `date_format(field, fmt, tz)`) — e.g. a daily `+08:00` bucket comes
back as local midnight. No reconversion is needed by API / dashboard /
alert consumers.
## API changes
- `config::meta::search::Query` (the `/api/{org}/_search` request body)
gains an optional field:
- `timezone: Option<String>` — `#[serde(default, skip_serializing_if =
"Option::is_none")]`. Backwards compatible: absent ⇒ UTC, identical to
today.
- proto `cluster.SearchQuery` gains `optional string timezone = 17;`
(regenerated `cluster.rs`).
## Function changes
- `histogram(field, interval)` → now also accepts `histogram(field,
interval, timezone)` (optional 3rd arg).
- Rewrite (`RewriteHistogram` optimizer rule): when a timezone resolves
to a non-zero offset, the **source** timestamp is shifted by that offset
before `date_bin`, so buckets and returned values are in local
wall-clock time. The `date_bin` origin stays at UTC.
- New helpers in `config::utils::time`:
- `parse_timezone_to_offset_opt(&str) -> Option<i64>` — non-panicking
fixed-offset parse.
- `parse_timezone_to_offset_at(&str, reference_micros) -> Option<i64>` —
fixed offset, else resolves an IANA name to its offset at the reference
instant.
- Invalid timezone ⇒ clear query error (no panic).
## Notes / caveats
- **DST:** named zones resolve to a *single* offset at the query's time
window — exact for normal windows, off by 1h only for the slice of a
range that straddles a DST transition.
- **Logs UI unchanged:** the logs chart already converts UTC→local for
display, so the timezone is intentionally **not** injected into the logs
histogram SQL (that would double-convert). This feature targets raw SQL
/ dashboards / alerts / API.
- **No-timezone path is byte-identical to before** (offset 0 ⇒ unchanged
plan).
## Testing
- `config::utils::time`: fixed offsets, IST `+05:30`, `Asia/Shanghai`,
`America/Los_Angeles` winter (PST) & summer (PDT), invalid input.
- `rewrite_histogram`: daily/hourly local bucketing, request-timezone
fallback, 3rd-arg precedence, named-zone winter + summer DST. 11 tests,
all green.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Follow-up to #12249. The cross-type collision check in the file upload
handler passes the formatted `stream_name` to `get_url_jobs_for_table`,
but URL jobs are stored with the raw `table_name` from the HTTP path.
When names contain uppercase or special characters,
`format_stream_name("My_Table")` produces `"my_table"`, which won't
match the stored URL job name `"My_Table"` — causing the cross-type
protection to be silently bypassed.
## Fix
Changed `get_url_jobs_for_table(&org_id, &stream_name)` to
`get_url_jobs_for_table(&org_id, table_name.trim())` — matching how the
URL handler itself queries URL jobs at line 299.
## Before/After
```rust
// Before: formatted name passed to URL jobs lookup (WRONG)
get_url_jobs_for_table(&org_id, &stream_name)
// After: raw table_name passed (CORRECT, matches how URL handler queries)
get_url_jobs_for_table(&org_id, table_name.trim())
```
## Notes
- `ENRICHMENT_TABLES` cache key continues to use `stream_name`
(formatted) — that's correct
- `get_url_jobs_for_table` now uses `table_name.trim()` — matching the
URL handler at line 299
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
Bug fix for - https://github.com/openobserve/openobserve/issues/12247
- **Bug:** Creating a URL enrichment table with name `X`, then creating
a file upload enrichment also named `X` (or vice versa), silently
overwrites the first one at every storage layer.
- **Fix:** Added cross-type name collision checks at the API boundary.
The file upload handler now rejects if a URL enrichment already exists
with the same name, and the URL handler rejects if a file upload
enrichment already exists. Same-type updates (append/retry/replace) are
unaffected.
## Root Cause
Enrichment tables are uniquely identified by `(org_id, table_name)`
across all storage layers — database, caches, object store, and
coordinator keys. Both URL-based and file-upload enrichments share the
same `StreamType::EnrichmentTables` and the same key space, so creating
one with the same name as another silently clobbers the previous one.
## Changes
**1 file changed** —
[src/handler/http/request/enrichment_table/mod.rs](src/handler/http/request/enrichment_table/mod.rs)
### `save_enrichment_table` (file upload handler)
Added check after input validation: if the table name already exists in
`ENRICHMENT_TABLES` and has URL jobs, return `400` — "already exists as
a URL enrichment."
### `save_enrichment_table_from_url` (URL handler)
Added check after fetching existing jobs: if the table name exists in
`ENRICHMENT_TABLES` but has no URL jobs, return `400` — "already exists
as a file upload enrichment."
### Behavior matrix
| | Existing: URL | Existing: File Upload | Existing: None |
|---|---|---|---|
| **Create File Upload** | ❌ Reject (400) | ✅ Allow (update) | ✅ Allow |
| **Create URL** | ✅ Allow (update) | ❌ Reject (400) | ✅ Allow |
## Test Plan
- [ ] Manually verify: create a URL enrichment named `test_cross_type` →
attempt file upload with same name → should get 400 error
- [ ] Manually verify: create a file upload enrichment named
`test_cross_type2` → attempt URL creation with same name → should get
400 error
- [ ] Manually verify: same-type updates still work (file→file append,
URL→URL retry/replace)
- [ ] Playwright automation will be added in a follow-up PR in the
`RegressionSet` folder
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Fixes https://github.com/openobserve/o2-enterprise/issues/1796 (item 1:
"Reload Existing URLs" returns error).
## Summary
Commit
[`85d261b5d0`](https://github.com/openobserve/openobserve/commit/85d261b5d0e4b68dbc79ca18b844fc53e2368ee9)
(vulnerability scan fixes, Apr 14 2026) removed the `!retry` guard from
enrichment table URL validation. This broke the **reload** update mode
because the frontend sends an empty URL when retrying/reprocessing
existing URLs — the backend already has them stored and just needs to
re-trigger processing.
## Root Cause
Before the breaking commit, URL validation was guarded by `!retry`:
```rust
if !retry && let Err(err_msg) = validate_enrichment_url(&request_body.url) {
return MetaHttpResponse::bad_request(err_msg);
}
```
The security fix removed the guard under the assumption that skipping
validation on retry was an SSRF bypass vector. However, the retry flow
sends `url=""` intentionally — the existing URLs are in the database.
## Fix
Re-adds the `!retry` guard around URL validation. In retry mode, URL
validation is skipped entirely (the URLs come from DB). All other modes
(create, append, replace, replace_failed) still require a non-empty,
valid URL with full SSRF protection.
## Test Plan
- [ ] Create enrichment table from URL
- [ ] Edit table → Reload existing URLs → Should succeed (no "URL cannot
be empty")
- [ ] Edit table → Add new URL → Should still validate URL format
- [ ] Edit table → Replace all URLs → Should still validate URL format
- [ ] Edit table → Replace failed URL → Should still validate URL format
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
## Problem
`sort_by=duration` (and any non-timestamp sort) in the
`/traces/latest_stream` endpoint had three bugs:
1. All partitions collapsed into one full-range scan
2. Q1 results re-sorted by `trace_start_time`, discarding SQL engine
order
3. `fetch_size` capped at `query_default_limit` (1000), so top-k ranked
on a truncated sample
## Fix
Mirrors the approach from #11654 (_search_stream):
- Detect non-ts ORDER BY generically via \`sql_order_expr\` prefix — no
hardcoded field names
- Run Q1 across **all** partitions; feed deduped hits into a
\`TopKHeap\` bounded to \`k = from + size\`
- Q2 (span details) runs **once** after all partitions complete, over
the narrowed time window of the deliverable traces
- Final hits emitted in heap order — correct global rank by ORDER BY
column
## Refactor
Extracted `run_q2_for_traces` async helper to eliminate ~300 lines of
duplicated code between the ts-path and non-ts-path. Both paths now call
the same helper for the `TraceResponseItem` build + Q2a (per-trace
aggregates) + Q2b (per-service breakdown). Also removed the dead
`"duration"` sort arm from the ts-path and the now-unused `sort_by`
parameter from `process_latest_traces_stream`.
## Known Limitation
`zo_sql_duration` is computed per-partition, so traces whose spans
straddle a partition boundary get an underestimated duration. Rare in
practice; a 2-pass global aggregation would be needed to fix.
## Related
Companion to #11654 — correct pagination for ORDER BY non-timestamp
column in `_search_stream`
related to
- https://github.com/openobserve/openobserve/pull/8003
- https://github.com/openobserve/openobserve/issues/7986
# Why We Can Delete `enable_align_histogram`
## What It Was
`enable_align_histogram` was a query parameter introduced in [PR
#8003](https://github.com/openobserve/openobserve/pull/8003) to fix
[Issue #7986](https://github.com/openobserve/openobserve/issues/7986).
The bug was that in the old two-step search flow (`_search_partition` →
`_search`), the two APIs independently generated histogram intervals,
leading to mismatched time buckets and inaccurate chart rendering in
dashboards.
The flag told `_search_partition` to:
1. Align partition boundaries to histogram interval boundaries (e.g.,
round to the start of a minute/hour/day).
2. Return the computed `histogram_interval` in the response so the
frontend could pass it to `_search`.
## Why It Is Now Dead Code
### 1. The frontend no longer calls `_search_partition`
All frontend search now uses `_search_stream` (HTTP/2 streaming), which
internally calls `search_partition()` with `enable_align_histogram =
false`. The old `_search_partition` endpoint is no longer consumed by
the UI. The only frontend caller (`getValuesPartition` in
`IndexList.vue`) is defined but never invoked.
### 2. `search_stream` handles alignment inherently
In the streaming search flow, partitions themselves define the search
time ranges. Each partition search runs on the exact aligned time
boundaries returned by the partition step. There is no separate
histogram query that needs an externally provided interval — alignment
is "baked into" the partition boundaries.
### 3. Histogram queries are already covered by `is_complex_query`
For histogram queries (which have `GROUP BY`), `is_complex_query`
returns `true`. Combined with `has_ts_column = true`, this triggers
`calculate_partition_settings()` to align `min_step` to the histogram
interval — the same alignment behavior that `enable_align_histogram`
previously provided, but without the extra flag.
### 4. All backend callers pass `false`
Of the 5 callers of `search_partition()`:
| Caller | Value |
|--------|-------|
| gRPC handler | hardcoded `false` |
| streaming execution | hardcoded `false` |
| traces handler | hardcoded `false` |
| HTTP `_search_partition` | from query param (frontend always passed
`true`, but frontend no longer calls this) |
| HTTP `_search_partition_multi` | from query param (frontend always
passed `true`, but frontend no longer calls this) |
No active code path ever sets `enable_align_histogram = true`.
## What Was Removed
- `get_enable_align_histogram_from_request()` — the HTTP query param
parser
- The `enable_align_histogram` parameter from:
- `search_partition()` function signature
- `search_partition_multi()` function signature
- `calculate_partition_settings()` function signature
- The alignment logic in `search_partition()`:
- `add_mini_partition` (always `false` without the flag)
- `PartitionGenerator` align parameter (now driven by `is_histogram`
alone)
- `resp.histogram_interval` override block
- The `|| enable_align_histogram` branch in
`calculate_partition_settings`
- The query parameter from both HTTP handler `utoipa` specs
- The `false` argument from all 5 call sites
## Problem
`ORDER BY <non-timestamp col>` (e.g. `ORDER BY duration DESC`) gave
wrong results and risked OOM because per-partition \`hits_to_skip\`
assumed rows are time-ordered — but the globally k-th row can be in any
partition.
## Solution
Each partition fetches its local top-`k = from+size` rows. A
**TopKHeap** on the leader merges them globally, skips `from`, returns
`size`. Heap is bounded to `k` elements at all times — O(k) memory
regardless of partition count.
**Multi-column ORDER BY** (`ORDER BY duration DESC, _timestamp DESC`) is
fully honored: heap comparison and final sort respect all columns in
order, using secondary columns as tiebreakers.
## Scenario behavior
| Query | Behavior |
|---|---|
| `ORDER BY _timestamp DESC` | Unchanged — per-partition path |
| `ORDER BY duration DESC` | Heap merge path — correct pagination |
| `ORDER BY duration DESC, _timestamp DESC` | Heap merge; ties in
`duration` broken by `_timestamp` |
| `ORDER BY duration DESC` page 2 | Heap collects global top-k, skips
`from`, returns `size` |
| JOIN / subquery / GROUP BY | `is_aggregate=true` → single partition
(unchanged) |
| `ZO_DISABLE_PARTITIONS_FOR_NON_TS_ORDER_BY=true` | Forces 1 partition
(circuit breaker preserved) |
| Partial cache hit + non-ts ORDER BY | TODO — pre-existing bug, no
regression |
## What is NOT changed
- `ORDER BY _timestamp` execution path
- `_search` (non-streaming) — DataFusion handles it correctly
- Aggregate / histogram queries
Harsh Mahajan
Huaijin
Subhra264
Loakesh Indiran
udev
Manan Shukla
Abhay PadamaniandHengfei Yang
yujinqiu
Neha PandClaude Opus 4.7