## Summary
Makes the compactor's bloom-filter **build side** enterprise-only by
moving it into the `o2-enterprise` crate (see
openobserve/o2-enterprise#1889). The search-side pruner and shared
`infra::bloom` SBBF support are unchanged.
- Delete `src/service/compact/bloom_build.rs` and
`src/service/tantivy/bloom_builder.rs` — they now live in
`o2_enterprise::enterprise::bloom`.
- Gate the `merge.rs` build hook behind `#[cfg(feature =
"enterprise")]`, calling
`o2_enterprise::enterprise::bloom::compact::build_for_stream`. In
non-enterprise builds the hook is a no-op.
- Drop the now-unused `get_tantivy_directory` re-export from
`grpc::storage`.
## Dependency
Requires openobserve/o2-enterprise#1889 (the enterprise build is broken
until that lands).
## Test plan
- [x] `cargo check --bin openobserve` (OSS, default features) —
warning-clean
- [x] `cargo check --bin openobserve --features enterprise` — gated call
site resolves
- [x] `cargo check -p o2_enterprise`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- close https://github.com/openobserve/openobserve/issues/12533
it will improve all case when we use tantivy to get row_id, then give
parquet to execute.
# Performance
400M records, original size 750GB
## Q1
```sql
SELECT
kubernetes_namespace_name,
avg(code) AS cnt
FROM
default
WHERE
match_all('zinc')
GROUP BY
kubernetes_namespace_name ORDER BY cnt DESC LIMIT 10
```
the `zinc` apprear in 140M records, so after match_all('zinc'), tantivy
will return 140M row_ids.
### main
tantivy time: 3s
totol time: 4.5s
### this pr
tantivy time: 95ms
total time: 1.5s
## Summary
- **Cloud visibility**: `list()` in `src/service/db/alerts/templates.rs`
was gating `DEFAULT_ORG` templates behind `!cfg!(feature = \"cloud\")`,
so any non-default org on cloud received an empty template list. Removed
the flag entirely — all `DEFAULT_ORG` templates (prebuilt and custom)
are now visible to every org, matching the existing ENT behaviour. This
is safe because cloud org identifiers are UUID-assigned; no regular user
can create an org named `default`.
- **Prebuilt templates fully read-only for non-root**: previously only
delete was blocked (403); updates were silently allowed. Now both
`save()` and `delete()` return `PrebuiltReadOnly` (403) for non-root
callers. Root users retain full access via `is_root_user()` bypass
(consistent with existing patterns in the codebase). Create with a
reserved name returns `ReservedName` (400) for everyone including root.
- **Frontend hardcoding removed**: `AddDestination.vue` was using a
`DEFAULT_TEMPLATE_SENTINEL` (`"__default__"`) and constructing prebuilt
template names client-side. Dropdown is now populated from the
`isPrebuilt: true` entries returned by `GET /alerts/templates`.
`TemplateList.vue` edit button now matches delete: shown-but-disabled
with `systemReadOnly` tooltip for prebuilt rows.
## Root cause
The `!cfg!(feature = "cloud")` flag in `list()` was added to prevent
`DEFAULT_ORG` data bleeding across cloud tenants. This correctly scopes
custom templates but also hides system prebuilt templates that every org
needs. Since prebuilt templates are now enforced as read-only
(non-root), exposing all DEFAULT_ORG templates to all orgs carries no
cross-tenant mutation risk.
## Test plan
- [ ] Non-default org on cloud: `GET /api/{org}/alerts/templates`
returns `prebuilt_slack`, `prebuilt_email`, etc. with `isPrebuilt: true`
- [ ] Non-root user: `PUT /api/{org}/alerts/templates/prebuilt_slack`
returns 403
- [ ] Non-root user: `DELETE /api/{org}/alerts/templates/prebuilt_slack`
returns 403
- [ ] Root user: `PUT /api/{org}/alerts/templates/prebuilt_slack`
succeeds
- [ ] Root user: `DELETE /api/{org}/alerts/templates/prebuilt_slack`
succeeds
- [ ] Any user: `POST /api/{org}/alerts/templates` with name
`prebuilt_slack` returns 400
- [ ] Destinations page: prebuilt destination template dropdown shows
`prebuilt_<type>` sourced from API
- [ ] `TemplateList`: edit button disabled (not hidden) for prebuilt
rows, with "System Read Only" tooltip
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
## Problem
Closes#12491 — a quick refresh (several queries in a row) occasionally
results in `wait in queue` for ~**10s**, then a retry works.
Intermittent, not frequent.
## Root cause
The search-queue dist-lock (`/search/cluster_queue/<work_group>`,
`work_group.rs`) is a NATS KV key whose value embeds an expiration of
`now + LOCKER_WATCHER_UPDATE_TTL` (**10s**). A detached **keep-alive
task** re-`put`s a fresh `now + 10s` value every 3s (`keep_alive_lock`).
`unlock()` did:
```
state = 2 -> tx.send(()) (signal keep-alive to stop) -> bucket.purge(key)
```
but the stop signal and the purge were **not synchronized** with the
keep-alive task. A keep-alive `put` that was already in flight (or fired
from the same `select!` tick as the stop signal) could land **after**
`purge`, recreating the key with a 10s-future expiration.
Because the locker bucket TTL is `lock_max_age` (2h), nothing deletes
that orphan. The next waiter spins in `wait_for_delete` — no delete
event ever comes, and the 1s fallback `check_exist_lock` keeps seeing
"not expired" — until the embedded **10s** expiration passes and
`check_exist_lock` finally purges it. Hence the exact 10s wait.
### Why it's intermittent and tied to "quick refresh"
The keep-alive only `put`s if the lock is held **≥3s**. That happens
under contention: in enterprise the dist-lock is held across the whole
`work_group_need_wait` polling loop, and in OSS for the query duration.
Quick refresh → many concurrent queries → work-group saturated → lock
held >3s → keep-alive fires → narrow race window (≈ put-latency / 3s) →
orphan that clears itself after 10s ("retry works").
## Fix
- `unlock()` now awaits the keep-alive task's `JoinHandle` **before**
`purge`, so any in-flight `put` completes first and the purge is
guaranteed to be the last write.
- The keep-alive `select!` is `biased` toward the stop signal and
re-checks the lock `state` before each `put`, so it never issues a `put`
once unlock has begun.
## Testing
- `cargo check` / `cargo clippy` / `cargo fmt` on `infra` pass.
- The race is timing-dependent and needs a live NATS server, so it isn't
covered by a deterministic unit test; the reasoning above and the
existing locker unit tests stand.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Inferred services for traces
Design at: #12515
Uninstrumented dependencies (databases, redis, kafka, external APIs) are
invisible in trace views today because OpenObserve only surfaces the
instrumented `service_name`. Users ask "why can't you see we call the db
or redis?". This adds **inferred services**: we derive a minimal
identity for those dependencies from CLIENT/PRODUCER span peer
attributes and surface them in the traces list and the service map.
### What's included
- **Ingestion** — derive three fields on CLIENT(3)/PRODUCER(4) spans
following OTel semantic conventions (current + legacy names):
`infer_service_name`, `infer_service_type`
(`database`/`queue`/`rpc`/`external`), `infer_service_system` (e.g.
`postgresql`/`kafka`/`grpc`/`http`). `peer.service` wins naming; ports
are stripped and bare IPs redacted to avoid one node per pod IP. Wired
into both `handle_otlp_request` and `ingest_json`; the three field names
are added to `BLOCK_FIELDS` so a user attribute of the same name can't
shadow them.
- **Traces list** — the per-service time/cost breakdown now groups by
`COALESCE(infer_service_name, service_name)` (schema-gated) and returns
`service_type`, so dependencies appear in the distribution and the UI
can render them dotted.
- **Service graph** — a second, schema-gated SQL over CLIENT/PRODUCER
spans (anti-joined against instrumented services so a dep that *is*
instrumented elsewhere isn't double-counted) emits inferred edges with
`connection_type`; `ServiceNode.service_type` /
`ServiceEdge.connection_type` carry the inferred category for dotted
nodes/edges. The dependency has no server span, so its latency is the
client span's own duration.
### Backward compatibility
Every new SQL path is gated on the field's presence in the stream
schema, so streams that predate the feature (or never call an
uninstrumented dependency) generate byte-identical SQL — no "field not
found". Old parquet reads the field as NULL → `COALESCE` falls back to
`service_name` gracefully.
### Companion PR
Service-graph topology tagging (`build_topology`) lives in the
enterprise repo: **openobserve/o2-enterprise#1876**. Build/merge the two
together.
### Testing
- `cargo check` / clippy / fmt clean
- unit tests: inferred derivation (20), traces-list breakdown
serialization, config `service_graph` types; full traces suite (417)
passes
- verified end-to-end locally: an OTLP trace with a redis CLIENT span
shows up in `GET /traces/latest` as `service_type: database` and as a
dotted node on the service map
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes#12461
## Problem
PromQL queries in a super-cluster returned data **only from the hub
cluster** — remote (spoke) clusters were silently skipped. SQL queries
across the same super-cluster worked correctly.
**Root cause:** The `data` gRPC handler on the spoke never registered
the incoming `trace_id` in `SEARCH_SERVER`. When the PromQL engine
called `insert_sender(trace_id)` to attach its abort channel, it found
no entry and immediately returned `SearchCancelQuery` (error code
20009). The spoke returned empty data with no visible error — the hub
received nothing from the spoke and returned only its local results.
The `query` gRPC handler already had the correct registration. The
`data` handler was missing it.
## Fix
Before spawning the data fetch task on the spoke, the `data` gRPC
handler now registers the incoming `trace_id` in `SEARCH_SERVER` as a
follower task — exactly the same registration the `query` handler
already performed. This allows the PromQL engine to successfully attach
its abort sender and proceed with execution. After the task completes,
the trace_id is removed from `SEARCH_SERVER` to avoid memory leaks.
## Test Queries
Ingest a metric with a distinguishing label to both hub and spoke, wait
~30s for WAL flush, then run:
**PromQL — before fix:** only hub cluster returned, spoke missing
**PromQL — after fix:** both hub and spoke returned
```bash
curl -s "http://<hub-host>/api/<org>/prometheus/api/v1/query_range" \
-u '<user>:<pass>' \
--data-urlencode "query=sum by (cluster) (total_payment_req)" \
--data-urlencode "start=$(($(date +%s) - 3600))" \
--data-urlencode "end=$(date +%s)" \
--data-urlencode "step=60"
```
**SQL — works both before and after (baseline check):**
```bash
curl -s -X POST "http://<hub-host>/api/<org>/_search?type=metrics" \
-H "Authorization: Basic <token>" \
-H "Content-Type: application/json" \
-d '{
"query": {
"sql": "SELECT cluster, SUM(value) as total FROM \"total_payment_req\" GROUP BY cluster",
"start_time": '"$(($(date +%s) - 3600))"'000000,
"end_time": '"$(date +%s)"'000000,
"size": 10
}
}'
```
- close https://github.com/openobserve/o2-enterprise/issues/1865
## TODO
- [x] make split_batch_for_grpc_response based on the real memory sizew
- [x] coalesce and gc view before send to leader
```
SELECT trace_id, count(*) as cnt, max(_timestamp) as ts FROM "k8s_logs" group by trace_id order by cnt desc limit 10
utf8view enable:
main: 30s (15k batches send to leader)
this branch: 15s (1.5k batches send to leader)
utf8view disable:
main: 13s (42k batches send to leader)
this branch: 12s (1.5k batches send to leader)
```
one found is the AggregateExec's time_calculating_group_ids, utf8 is
faster than utf8view
## Summary
Closes#12426.
In cloud-native deployments (ECS/Fargate, Kubernetes) secrets managers
inject individual values, not composed strings. That makes it impossible
to cleanly provide a full `ZO_META_POSTGRES_DSN` without custom tooling
(init containers, Lambda functions, etc.).
This PR adds five individual connection vars as an alternative:
| Variable | Default |
|---|---|
| `ZO_META_POSTGRES_HOST` | `""` |
| `ZO_META_POSTGRES_PORT` | `5432` |
| `ZO_META_POSTGRES_USER` | `""` |
| `ZO_META_POSTGRES_PASSWORD` | `""` |
| `ZO_META_POSTGRES_DBNAME` | `""` |
**`ZO_META_POSTGRES_DSN` takes full precedence when set, no breaking
change.**
When the DSN is empty, `check_common_config()` composes it from the
individual vars at startup. User, password and dbname are
percent-encoded so credentials with special characters (common with
secrets-manager generated passwords) survive sqlx's percent-decoding
when the DSN is parsed. The connection code itself is untouched, every
consumer of `meta_postgres_dsn` (including the migration adapter) just
sees a normal DSN.
## Changes
- `src/config/src/config.rs`: adds 5 new `env_config` fields, validates
that either the DSN or all individual vars are set, and composes the DSN
from the individual vars when it is empty
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Problem
`query_by_ids` runs `SELECT ... WHERE id IN (...)` with **no partition
key filter**. On the partitioned `file_list` table this means:
- **planning** must open and lock **every** partition (~101ms on 384
partitions)
- **execution** probes **every** partition's id index (~10ms)
Measured on production-scale data (2.7B rows / 1.2TB / 384 daily
partitions, PostgreSQL 17): **~110ms per call**, vs ~1.5ms on a
non-partitioned table. `query_by_ids` is on the search hot path (id →
FileKey reverse lookup), so every search pays this cost.
## Fix
The ids passed to `query_by_ids` are produced by `query_ids` **using a
known time range** — the search path already holds it
(`service::file_list::query_by_ids` already takes `time_range` but did
not pass it down). This PR passes the range down to the infra layer and
derives a `date` filter via the existing `derive_date_range`, enabling
plan-time partition pruning:
```sql
SELECT ... FROM file_list WHERE id IN (...)
AND date >= '2026/05/06/00' AND date < '2026/05/14/00' -- new
```
## Benchmark
From the file_list partitioning performance test (2.7B rows, 384 daily
partitions, c6i.8xlarge, PG 17.10,
jit=off/work_mem=128MB/rpc=1.1/ANALYZEd). Same in-window batch of 1000
ids, median of 3 runs, total = execution + planning:
| Query window | Before (total) | After (total) | Improvement |
Partitions scanned |
|---|---:|---:|---|---|
| 1h | 112.7ms (11.7 + 101.0) | **1.18ms** (0.82 + 0.36) | **95x** | 384
→ 1 |
| 1d | 110.3ms (9.4 + 100.9) | **1.60ms** (0.97 + 0.62) | **69x** | 384
→ 2 |
| 7d | 109.4ms (8.4 + 100.9) | **2.93ms** (0.79 + 2.14) | **37x** | 384
→ 8 |
With the filter, the partitioned table matches or **beats** the
non-partitioned baseline (execution 0.8~1.0ms vs 1.2~1.3ms — per-day
partition id indexes are shallower than one monolithic 2.7B-row index).
## Safety
- **Semantically safe**: the ids were selected by `query_ids` with
exactly the same derived date range, so the added filter cannot drop
rows.
- **No side effect on non-partitioned tables**: measured 1.5ms → 1.7ms
(within noise) on the single-table baseline.
- **Guarded**: the filter is only applied when `time_range` is
`Some((start, end))` with `start > 0 && end >= start`; all other callers
pass `None` and keep the previous behavior.
- SQLite backend ignores the parameter (not partitioned; already a
single index probe).
## Changes
- `infra::file_list::FileList::query_by_ids` trait: add `time_range:
Option<(i64, i64)>`
- postgres impl: derive and append the date filter when a valid range is
given
- sqlite impl: accept and ignore the parameter
- `service::file_list::query_by_ids`: pass its existing `time_range`
down (db + local cache)
- `cli::basic::bloom`: pass `None`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- close https://github.com/openobserve/openobserve/issues/11956
- [x] check how PartitialReduce work with AggregateTopk
- [x] check how PartitialReduce work with StreamingAgg
# New Env
```
ZO_FEATURE_PARTIAL_REDUCE_ENABLED=true (default is true)
```
# Performance
## First datasets
80M records
single node optimzier disable(to simulate the distribute query)
### Q1
```sql
SELECT count(distinct request_id) FROM "k8s_logs"
```
80M request_id unique value
with partial reduce: 61s (80M records send from follower to leader)
without partial reduce: 54s (80M records send from follower to leader)
### Q2
```sql
SELECT count(distinct client_ip) FROM "k8s_logs"
```
10M client_ip unique request_id
with partial reduce: 5s (10M records send from follower to leader)
without partial reduce: 54s (80M records send from follower to leader)
### Q3
```sql
SELECT
host,
COUNT(DISTINCT client_ip) AS cnt
FROM
k8s_logs
GROUP BY
host ORDER BY cnt DESC
```
2000 host unique value, 10M client_ip unique value
with partial reduce: 41s (75M records send from follower to leader)
without partial reduce: 26s (80M records send from follower to leader)
### Q4
```sql
SELECT
kubernetes_namespace_name,
COUNT(DISTINCT client_ip) AS cnt
FROM
k8s_logs
GROUP BY
kubernetes_namespace_name ORDER BY cnt DESC
```
10 kubernetes_namespace_name unique value, 10M client_ip unique value
with partial reduce: 14s (38M records send from follower to leader)
without partial reduce: 17s (80M records send from follower to leader)
### Q5
```sql
SELECT
histogram(_timestamp, '5 minute') AS ts,
host,
approx_distinct(client_ip) AS cnt,
ROW_NUMBER() OVER (
PARTITION BY histogram(_timestamp, '5 minute')
ORDER BY approx_distinct(client_ip) DESC
) AS rn
FROM
k8s_logs
GROUP BY
ts, host
QUALIFY rn <= 10
ORDER BY ts DESC, cnt DESC
```
2000 host unique value, 10M client_ip unique value
with partial reduce: 3.2s (15k records send from follower to leader)
without partial reduce: 4.2s (46k records send from follower to leader)
## Second datasets
100M records, 250GB original size, 37GB compression size
### Q6
```sql
SELECT
client_ip,
count(*) AS cnt,
max(_timestamp) AS ts
FROM
"k8s_logs"
GROUP BY
client_ip ORDER BY cnt DESC LIMIT 100
```
0.5M client ip unique value
with partial reduce: 5s (517k records send from follower to leader)
without partial reduce: 19s (100M records send from follower to leader)
### Q7
```sql
SELECT
trace_id,
count(*) AS cnt,
max(_timestamp) AS ts
FROM
"k8s_logs"
GROUP BY
trace_id ORDER BY cnt DESC LIMIT 100
```
12M client ip unique value
with partial reduce: 13s (12M records send from follower to leader)
without partial reduce: 23s (100M records send from follower to leader)
**for streaming aggreagte cache genearte**
with partitial reduce first run: 23s
without partial reduce first run: 37s
# Design
## Background
OpenObserve uses a distributed query architecture where a **leader
(querier)** node dispatches physical execution plans to **follower
(ingester/compactor)** nodes via Arrow Flight RPC. The `RemoteScanExec`
node represents work pushed to remote followers; results stream back to
the leader for final processing.
For aggregation queries, DataFusion produces a two-phase plan:
1. **`Partial`** — each follower computes partial aggregates (partial
SUM, COUNT, group keys, etc.)
2. **`Final`** — the leader merges all partial results into the final
answer
**Problem:** All partial aggregate rows from every follower are sent to
the leader unconditionally. When there are many unique group-by keys or
large datasets, this generates significant network traffic even when
many follower partitions produce overlapping keys that could be
pre-merged locally.
---
## Solution
DataFusion supports a third aggregate mode, `PartialReduce`, which sits
between `Partial` and `Final`. It takes the output of a `Partial`
aggregate and applies a further reduction pass — merging rows with the
same group keys — before they leave the node.
Insert a `PartialReduce` node on the **follower side** (between the
`Partial` aggregate and the `RemoteScanExec`). Followers send fewer rows
over the network; the leader's `Final` phase still produces a correct
result.
---
## Execution Plan Transformation
**Before:**
```
FinalAggregateExec (leader)
└─ RepartitionExec (leader)
└─ RemoteScanExec (network — all partial rows)
└─ PartialAggregateExec (follower, N partitions)
└─ ScanExec
```
**After:**
```
FinalAggregateExec (leader)
└─ RepartitionExec (leader)
└─ RemoteScanExec (network — reduced rows)
└─ PartialReduceAggregateExec (follower)
└─ CoalescePartitionsExec (follower — merges N partitions into 1)
└─ PartialAggregateExec (follower, N partitions)
└─ ScanExec
```
`CoalescePartitionsExec` merges all follower partitions before the
`PartialReduce` pass to maximize the reduction ratio.
---
## Trade-offs
**Pros:**
- Reduces network bytes transferred from followers to leader for
aggregation queries
- Zero correctness risk for non-aggregate queries (passthrough)
- Runtime feature flag, safe to disable without redeployment
**Cons:**
- Adds a CPU pass on followers; worthwhile only when network is the
bottleneck
- Queries with high-cardinality GROUP BY and no key overlap between
partitions get no reduction, only extra follower CPU
- `CoalescePartitionsExec` serializes follower partitions before the
reduce pass, reducing follower-side parallelism
## Issue
When `_search_multi_stream` processes a histogram query, streaming
execution splits the time range into partitions. Each partition calls
`do_search()` with its own (smaller) time range, causing
`Sql::new_with_options()` to recompute `histogram_interval` via
`generate_histogram_interval()` based on the **partition's** time range
duration instead of the **original query's** time range.
Since different partitions have different time range durations, each
gets a different auto-generated histogram interval (e.g. 10s for a small
partition, 30s for the full range). This produces inconsistent bucket
boundaries across partitions in the streaming response.
The non-streaming `_search_multi` endpoint does not have this issue
because it searches the entire time range as a single unit.
### Root cause
In `src/service/search/streaming/mod.rs`,
`process_search_stream_request()` enters either:
- the **cache path**: `handle_cache_responses_and_deltas()` →
`process_delta()` → per-partition `do_search()`
- the **no-cache path**: `do_partitioned_search()` → per-partition
`do_search()`
In both paths, each partition clones `req` with a narrowed
`start_time`/`end_time`, so when `do_search()` creates a fresh `Sql`
struct, `histogram_interval` is 0 and `Sql::new_with_options()`
recomputes it from the partition's (shorter) time range.
### Fix
Before either path is entered, parse the SQL once with
`HistogramIntervalVisitor` using the **full query time range** and set
the resulting interval on `req.query.histogram_interval`. All subsequent
partition clones inherit this value, and `Sql::new_with_options()` takes
the `query.histogram_interval > 0` branch — using the preset value
instead of recomputing from the partition range.
## Summary
Fixes#11904 — streaming search kept running on the queriers even after
the user closed the browser tab.
The SSE search endpoints spawn the query in a detached task and stream
results back over `text/event-stream`. When the client disconnects, the
response stream is dropped and we stop *sending* results, but the
distributed query continued executing on the queriers, wasting resources
on results nobody will read.
This wires client disconnect into the existing query-cancellation
machinery.
## What changed
- **`SearchStreamGuard`** (`search/utils.rs`) — moved into the response
stream's closure so it shares the stream's lifetime. It marks itself
finished when a terminal event (`Done` / `Error` / `Cancelled`) is
produced; if the stream is dropped before that (client gone), its `Drop`
fires the cancel for the parent `trace_id`.
- **`cancel_query_internal`** (`search/query_manager.rs`) — best-effort
cancel for internal callers (no HTTP response), reusing the existing
super_cluster/regular split.
- Wired the guard into **`search_http2_stream`**,
**`values_http2_stream`**, and **`search_multi_stream`**.
Cancellation is keyed by the parent `trace_id`; the query manager
removes entries by prefix, so multi-stream sub-queries
(`{trace_id}-q{n}`) are covered by a single cancel.
## Notes
- Distributed cancellation is an **enterprise** feature (the whole
`cancel_query` / `QueryManager` path is enterprise-gated). On the
open-source build the guard only logs the disconnect; OSS already stops
sending once the channel closes.
- Normal completion never triggers a cancel (terminal-event →
`mark_finished`). Worst case on an edge race is a harmless no-op cancel
for an already-finished `trace_id`.
- Complements the existing manual "Cancel" button — same underlying
machinery, triggered automatically on disconnect.
This branch also includes an unrelated config tweak (`fix: enable
compact fast mode for local`) enabling compact `fast_mode` automatically
when running in local mode.
## Testing
- `cargo check` (OSS) and `cargo check --features enterprise` both clean
- Unit test for the guard state machine passes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Fixes#11364.
Decide **client-side** gRPC TLS from the connection address scheme
(`https://` vs `http://`) instead of the global `ZO_GRPC_TLS_ENABLED`
toggle. This decouples client TLS from server TLS, which lets a super
cluster sit behind a TLS-terminating load balancer (AWS ALB/NLB) without
the deadlock described in the issue — **no new env vars and no separate
client/server toggle**.
All gRPC client connections funnel through
`infra::client::grpc::create_channel` (search/metrics/node/cluster-info
clients, the flight client, file downloader, broadcast), so this is a
single-spot change.
### Behaviour
| Config | Server (still on `ZO_GRPC_TLS_ENABLED`) | Advertised addr |
Client (now scheme-based) |
|---|---|---|---|
| `tls_enabled=true` (today) | TLS | `https://…` | TLS ✓ |
| `tls_enabled=false` (today) | h2c | `http://…` | h2c ✓ |
| **Super cluster behind LB:** `tls_enabled=false` +
`O2_SUPER_CLUSTER_PUBLIC_ADDR=https://lb:443` | h2c (ALB terminates TLS)
| intra: `http://`, cross-cluster: `https://lb` | intra h2c,
cross-cluster TLS ✓ |
The server keeps running h2c behind the ALB, intra-cluster pods stay
h2c, and only the cross-cluster connection (which sees `https://`)
negotiates TLS on the client side. Fully backward compatible.
### Notes
- The client SNI/domain is now only overridden when
`ZO_GRPC_TLS_CERT_DOMAIN` is set; otherwise tonic derives it from the
URI host, which matches the load balancer's certificate. (Previously
`.domain_name("")` was always called, which would send an empty SNI in
the new flow.)
- With `ZO_GRPC_TLS_ENABLED=false`, `check_grpc_config` does not require
cert/key files, and the default `webpki` roots cover a public ACM cert —
so **no cert files need to be mounted** for this setup, side-stepping
the Helm `extraVolumes` concern in the issue.
- Relies on the enterprise `O2_SUPER_CLUSTER_PUBLIC_ADDR` being settable
to a full `https://host:port` URL and passed through verbatim to
`create_channel`.
## Test plan
- [x] `cargo test -p infra --lib client::grpc::` — added unit tests
covering the scheme-based decision (`https://` → TLS,
`http://`/`grpc://`/bare/empty → h2c, case-sensitive scheme match).
- [x] `cargo check -p infra`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Problem
Fixes#12096.
During a rolling restart / redeploy, the router intermittently returns
`502 Proxy request failed: error sending request for url
(http://<node>:5080/...)` for querier/ingester routes (e.g.
`/api/{org}/functions`). Restarting nodes one-by-one does not help.
## Root cause
The error comes from the **router's proxy** (`src/router/http/mod.rs`),
not the querier. The router picks one node from its cached node list and
proxies to it. While a node is restarting there is an unavoidable window
where:
- the node has begun graceful shutdown — it stops accepting new
connections and its pooled keep-alive connections are being closed — but
- the router has not yet processed the node's offline/leave event.
Propagation is not instant (NATS event lag), and the health-check
fallback only removes a node after `ZO_HEALTH_CHECK_FAILED_TIMES`
(default 3) consecutive failures polled every `ZO_NODE_HEARTBEAT_TTL/2`
(≈15s) → up to ~45s.
The actual defect: the proxy had **no failover**. It returned `502` on
the *first* connection error even though other healthy nodes existed, so
every request that landed on the restarting node surfaced as a
user-facing error. That's why a one-by-one restart still produced errors
— each node's restart opened its own failure window.
## Fix
Add retry-with-failover in the router proxy:
- `resolve_target` → **`resolve_candidates`**: returns the ordered
candidate node list (preferred node first via the existing dispatch
strategy, the rest as fail-over targets) instead of a single target.
- `proxy_request` and `proxy_with_body_routing` now **loop over
candidates**, rebuilding the request per node, and on a connection-level
send error fail over to the next node — up to `1 + ZO_ROUTE_MAX_RETRIES`
attempts.
- `is_retryable_send_error`: retry on `is_connect()` for **any** method
(a connect failure means the request was never processed, so it is
always safe); for idempotent **GET** also retry on timeouts / dropped
connections.
- The non-streaming body is buffered once and re-sent on each attempt.
For body-routing, the consistent-hash node stays preferred (result-cache
affinity), with the other online queriers as fallback (failover only
costs a cache miss).
## New config
- **`ZO_ROUTE_MAX_RETRIES`** — max number of other nodes to fail over to
when the selected node is unreachable. Default `2`; `0` disables retry.
## Testing
- `cargo check` (config + main crate): clean
- `cargo clippy -p openobserve --lib`: clean
- `cargo test -p openobserve --lib router::http`: **13 passed** — added
`test_order_nodes_*`, `test_max_attempts_bounds`, `test_payload_method`;
updated `test_resolve_candidates_with_base_uri`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- [x] Bulk ingestion panic
- [x] Create function panic
- [x] Dashboards/Alerts/Incidents panic when list data
- [x] Match http response error code
- [x] Skip some endpoint that fuzz can't generate data
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Closes#12401 Design at: #12402
## What
Adds timezone support to the `histogram()` SQL function so buckets align
to a user's local time instead of always UTC.
## How to use
### 1. Per-query — 3rd argument to `histogram()`
```sql
SELECT histogram(_timestamp, '1 hour', '+08:00') AS key, count(*) FROM "logs" GROUP BY key;
SELECT histogram(_timestamp, '1 day', 'America/Los_Angeles') AS key, count(*) FROM "logs" GROUP BY key;
```
### 2. Per-request — `timezone` field in the search payload
Applies to every `histogram()` in the query that has no 3rd argument
(including the bare 1-arg `histogram(_timestamp)` form):
```jsonc
POST /api/{org}/_search
{
"query": {
"sql": "SELECT histogram(_timestamp) AS key, count(*) FROM \"logs\" GROUP BY key",
"start_time": 1704067200000000,
"end_time": 1704153600000000,
"timezone": "+08:00"
}
}
```
or
```jsonc
POST /api/{org}/_search
{
"query": {
"sql": "SELECT histogram(_timestamp) AS key, count(*) FROM \"logs\" GROUP BY key",
"start_time": 1704067200000000,
"end_time": 1704153600000000,
"timezone": "America/Los_Angeles"
}
}
```
### Precedence
```
histogram() 3rd arg > request "timezone" field > UTC (default, unchanged)
```
### Accepted formats
- **Fixed offset:** `+08:00`, `-05:30`, `UTC`, `""` (empty = UTC)
- **IANA name:** `America/Los_Angeles`, `Asia/Kolkata`, … (resolved via
`chrono-tz`)
### Result semantics
The returned bucket value is **local wall-clock time** (same convention
as `date_format(field, fmt, tz)`) — e.g. a daily `+08:00` bucket comes
back as local midnight. No reconversion is needed by API / dashboard /
alert consumers.
## API changes
- `config::meta::search::Query` (the `/api/{org}/_search` request body)
gains an optional field:
- `timezone: Option<String>` — `#[serde(default, skip_serializing_if =
"Option::is_none")]`. Backwards compatible: absent ⇒ UTC, identical to
today.
- proto `cluster.SearchQuery` gains `optional string timezone = 17;`
(regenerated `cluster.rs`).
## Function changes
- `histogram(field, interval)` → now also accepts `histogram(field,
interval, timezone)` (optional 3rd arg).
- Rewrite (`RewriteHistogram` optimizer rule): when a timezone resolves
to a non-zero offset, the **source** timestamp is shifted by that offset
before `date_bin`, so buckets and returned values are in local
wall-clock time. The `date_bin` origin stays at UTC.
- New helpers in `config::utils::time`:
- `parse_timezone_to_offset_opt(&str) -> Option<i64>` — non-panicking
fixed-offset parse.
- `parse_timezone_to_offset_at(&str, reference_micros) -> Option<i64>` —
fixed offset, else resolves an IANA name to its offset at the reference
instant.
- Invalid timezone ⇒ clear query error (no panic).
## Notes / caveats
- **DST:** named zones resolve to a *single* offset at the query's time
window — exact for normal windows, off by 1h only for the slice of a
range that straddles a DST transition.
- **Logs UI unchanged:** the logs chart already converts UTC→local for
display, so the timezone is intentionally **not** injected into the logs
histogram SQL (that would double-convert). This feature targets raw SQL
/ dashboards / alerts / API.
- **No-timezone path is byte-identical to before** (offset 0 ⇒ unchanged
plan).
## Testing
- `config::utils::time`: fixed offsets, IST `+05:30`, `Asia/Shanghai`,
`America/Los_Angeles` winter (PST) & summer (PDT), invalid input.
- `rewrite_histogram`: daily/hourly local bucketing, request-timezone
fallback, 3rd-arg precedence, named-zone winter + summer DST. 11 tests,
all green.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#12351
Fixed RUM middleware in `middleware_data.rs` to extract
`ootags`/`o2tags` from the request body when the field is not present in
URL query parameters. Previously, the middleware only inspected query
parameters for these fields, but the RUM SDK sends them in the request
body.
## Why
User logs confirmed the issue: `ootags raw=None, o2tags raw=None`
appeared consistently in the middleware logs even when the RUM SDK
included tags like `env`, `service`, `version`, and `sdk_version`. Geo
keys (which come from a different extraction path) were unaffected,
narrowing the root cause to query-param-only parsing of
`ootags`/`o2tags`. The RUM SDK sends these tags in the POST body, not as
query string parameters.
## Approach
Added body-parsing logic to read and decode the request body when
`ootags`/`o2tags` are absent from query parameters. The body is parsed
once using `url::form_urlencoded` and merged into the existing key-value
map before the `filter_tags` step runs, ensuring the same downstream
pipeline works unchanged.
---------
Co-authored-by: Neha P <neha@openobserve.ai>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Co-authored-by: Neha P <144308844+neha00290@users.noreply.github.com>
## Summary
Design at:
https://github.com/openobserve/designs/blob/main/infrastructure/bloomfilter/DESIGN.md
A bloom-filter pruning layer that sits **between `file_list` and
tantivy** in the search path. It drops candidate files before any `.ttv`
is opened, for **positive equality / IN** lookups on high-cardinality
fields (`trace_id`, `request_id`, …).
It is built **on top of** tantivy: the `.bf` is constructed by iterating
the tantivy term dictionary, so a field must be a tantivy secondary
index (`index_fields`) **and** opted into `bloom_filter_fields`. Only
the intersection is built — a bloom-only field produces nothing. To make
that intersection non-empty by construction, enabling a field for bloom
now folds it into `index_fields` automatically (see Config). Values are
not inspected; the operator chooses which fields are worth it (see
guidance below).
Design notes live in the module-level docs under `src/infra/src/bloom/`
(`mod.rs`, `writer.rs`, `reader.rs`, `sbbf.rs`).
## Design
- **Transposed (block-major) `.bf` layout.** All files in a `.bf` share
`num_blocks = B`, so `block_index = fastmap(hash(value), B)` is
identical across files and the blocks a query needs form **one
contiguous row**. The search side reads **one `M × 32`-byte row per
`.bf`** to answer membership for all `M` files — prune cost is
`O(groups)`, not `O(files)`. (`infra::bloom`, own ~120-LOC SBBF over
gxhash64.)
- **Sub-grouping.** A hour's new files are sorted by record count and
split into chunks of ≤ `ZO_BLOOM_FILTER_MAX_FILES_PER_BF` (default 256),
one `.bf` each. Bounds compactor build memory (`M × B × 32`) and `.bf`
object size; query reads become `ceil(files / 256)` rows.
- **Footer-only reader + batched fetch.** Footer (≈ `M × 12` B) comes
from `BLOOM_FOOTER_CACHE` or one 16 KB suffix GET (with a precise
re-read when a large footer overflows the probe); rows are pulled with
one `cache_storage::get_ranges` call. `cache_storage` was reworked to do
real batched cache reads (memory: 1 lookup + N slices; disk: 1
`File::open` + N `pread`).
- **B sizing.** `num_blocks_for(max_records_in_chunk,
ZO_BLOOM_FILTER_FPP)` (default fpp `0.01`) — record count is a safe NDV
upper bound, so the SBBF never saturates for high-cardinality fields.
- **Versioned, with orphan cleanup.** A file's `bloom_ver` is write-once
— a row is never re-stamped, so dumped rows keep pointing at the `.bf`
they were built into. But `bloom_ver` is **not** append-only at the
bucket level: a merge produces new output files with a fresh `bloom_ver`
and deletes the small inputs, so the old inputs' `.bf` can become
orphaned. After each merge round the compactor retires any `.bf` whose
`bloom_ver` is no longer referenced by a live `file_list` row in the
bucket (enqueued through the existing `file_list_deleted` path).
`bloom_ver` is preserved through every file_list INSERT/SELECT path
(incl. the SQLite read-cache) and the dump parquet.
- **Observability.** Prometheus metrics for prune
(`bloom_prune_keep_ratio`, `bloom_prune_duration_seconds`);
`bloom-inspect` CLI dumps a `.bf` footer (fields, B, row bytes, file
names); footer-cache hit/miss via the shared `BytesCache` tagged
metrics; `Local::get_ranges` now reports `STORAGE_READ_*` (previously
bypassed).
## When it helps (measured, S3, no disk cache)
Same stream `benchtest`, 170 files / 14.64 GB index, single matching
row. The only variable is whether the field's values are
time-correlated:
| Field | Type | tantivy only | bloom + tantivy | Outcome |
|---|---|---|---|---|
| `trace_id` (16-byte random) | random | **2584 ms** (opens all 170
`.ttv`) | 24 ms prune + 65 ms = **89 ms** | **~29× faster** |
| `request_id` (`…-7e74-…`, UUIDv7) | time-ordered | **154 ms** (sparse
index range-prunes) | 42 ms + 154 ms = **196 ms** | ~1.3× **slower** |
The whole 170-file group was answered by **one row read of 5440 bytes**
(`170 × 32`) + a cached footer.
**Guidance — enable only when all three hold (advisory; not enforced):**
1. **High cardinality** field.
2. **Random values** (not time-ordered) — so tantivy can't range-prune
and must scan every file. Time-ordered ids (UUIDv7, snowflake) are
already pruned by tantivy's sparse index in memory → bloom is pure
overhead there.
3. **Many files per hour** — hourly volume ≳ 10 ×
`ZO_COMPACT_MAX_FILE_SIZE` (≳10 `.ttv` files), so there's something to
prune. Bloom turns N file opens into one row read; at 1–2 files/hour
it's net overhead.
The 29× result above had 170 files/hour all answered by one `.bf`. ❌
Low-cardinality / non-indexed / range / `LIKE` / regex / `!=` / OR: no
effect.
## Config
- `ZO_BLOOM_FILTER_ENABLED` (default true) — global gate.
- Per stream: a field must be in **both** `index_fields` and
`bloom_filter_fields`. Adding a field to `bloom_filter_fields` auto-adds
it to `index_fields` (bloom is built on the secondary index), and the
default index fields include the default bloom fields.
- `ZO_BLOOM_FILTER_FPP` (default 0.01) — target false-positive
probability; sizes the SBBF block count (smaller fpp = fewer false
survivors, larger `.bf`). Out-of-range falls back to 0.01.
- `ZO_BLOOM_FILTER_MAX_FILES_PER_BF` (default 256) — files per `.bf`;
trades query reads vs build memory.
- `ZO_BLOOM_FILTER_DEFAULT_FIELDS` (default empty) — fields bloomed for
every stream.
- `ZO_BLOOM_FOOTER_CACHE_MAX_SIZE` (default 1% mem, clamped 32–256 MB).
Removed in this PR: `ZO_BLOOM_FILTER_DISABLED_ON_SEARCH`,
`ZO_BLOOM_FILTER_NDV_RATIO` (B is now sized from record count + fpp, not
an ndv ratio).
## Test plan
- [x] `cargo build`, `cargo clippy -p infra -p openobserve --lib -- -D
warnings`, `cargo fmt --check` clean
- [x] Unit tests: `cargo test -p infra --lib bloom`, `cargo test -p
openobserve --lib bloom`
- [x] E2E on S3 (results above): random field prunes 170→1 and is ~29×
faster; transposed read = 1 request / 5440 B per group
- [x] Reviewer: confirm gating expectations (operator opts fields in; no
auto value detection)
- [x] Test: orphan bloom: 2 * 256MB -> merge to 512MB -> merge to
1024MB, the first .bf should be deleted
- [x] Test: orphan bloom: 6 * 256MB -> merge to 1GB + 512MB -> 512 MB
merge to 1024MB, the first .bf include 2 files(1GB + 512MB), after merge
again, the first still have 2 files but one of them already deleted. we
should keep the first .bf, because still have files in use.
## Notes
- `.bf` format is `VERSION 0x01` (transposed). Not compatible with any
pre-merge experimental `.bf`; rebuild by wiping the `bloom/` dir and
resetting `bloom_ver=0`.
- Hash is gxhash64 (project-wide `config::utils::hash`), not the Parquet
spec's xxhash64 — `.bf` is internal, never read by external tools.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
## Summary
When generating the tantivy index after a parquet merge, we previously
read **all** columns from the merged parquet file even though only the
indexed fields are needed. This PR restricts the parquet read to just
the index fields, significantly reducing CPU work and peak memory during
compaction.
## Benchmark
Same workload on both runs: compactor merges **173 files** (original
9.49 GB / compressed 1.60 GB) into one parquet file, then builds the
tantivy index (7 row_groups, 800,000 rows, 2 indexing threads).
| Metric | Before | After | Delta |
| --- | ---: | ---: | ---: |
| `parallel::build_index` elapsed | 5945 ms | 2868 ms | **−51.8%**
(2.07×) |
| Tantivy index generation (total) | 7798 ms | 4814 ms | **−38.3%**
(1.62×) |
| Merge row-group indices | 1748 ms | 1850 ms | ~same |
About **2× faster** parallel index build, the total improve about 30%
with identical output.
> **Note:** The size of the win depends on the ratio of index fields to
total fields in the stream. This test used a stream with **~600 fields
and only 7 index fields**, so we skip reading the vast majority of
columns. For streams with few non-index fields the improvement will be
much smaller.
## Test plan
- [x] Verified compactor still emits a tantivy index of the same size
for the same input
- [x] CI green
## Summary
In the O2 tokenizer, camelCase identifiers (e.g. `HTTPSConnection`,
`getUserName`, `CamelCase`) are split into sub-tokens. Previously:
- **Ingest** indexed both the root token (`HTTPSConnection`) and its
splits (`HTTPS`, `Connection`).
- **Search** dropped the root and only emitted the splits.
This PR makes **search emit only the root token** and drop the splits.
### Why
When a user searches for `HTTPSConnection`, the query tokenizer used to
break it into `HTTPS` AND `Connection`. That matches any document that
happens to contain both words anywhere — including documents that never
contained the original camelCase identifier. Concretely:
- A log line like `"HTTPS handshake failed for Connection #42"` would
match a search for `HTTPSConnection`, which is wrong.
- Conversely, the ingest side already indexes the full `HTTPSConnection`
root token, so the search side can rely on it directly for an exact,
intent-preserving match.
Emitting only the root token on the search path keeps the query aligned
with the user's literal input and eliminates these false positives,
while ingest continues to index both root + splits so partial matches
(`Connection`, `HTTPS`) still work when the user actually types a
sub-word.
### Notes
- Non-ASCII delimiters that follow a camelCase token (e.g.
`CamelCase中文`) are still flushed correctly in search mode —
`buffer_offset` is set to the root token's `offset_to` before the
delimiter is pushed.
- Added regression tests covering camelCase followed by non-ASCII,
acronym-style prefixes (`HTTPSConnection中Foo`), back-to-back camelCase
tokens across a non-ASCII delimiter, and plain IP addresses (no
camelCase) under search mode.
## Test plan
- [x] `cargo test -p config tokenizer::o2_tokenizer` passes
- [x] Re-index a sample stream and confirm exact-match queries on
camelCase identifiers return only the expected documents
- [x] Confirm sub-word queries (e.g. just `Connection`) still match
documents containing `HTTPSConnection`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Problem
Histogram panels showed a false drop in the **leftmost bucket**.
`histogram()` rewrites to `date_bin(interval, _timestamp, epoch)`, which
assigns each record to the **left boundary** of its bucket. So a record
at `08:10:17` gets key `08:10:00`.
But the scan filter was `_timestamp >= start_time` where `start_time`
was the raw query time (e.g. `08:10:17`). Records in `[08:10:00,
08:10:17)` passed the bucket label check but were excluded from the scan
— making the first bucket appear partially filled.
Fixes#1753 (o2-enterprise)
## Fix
Snap `start_time` **down to the nearest bucket boundary** at the
**leader node**, before it is broadcast to all follower nodes via
`FlightSearchRequest`.
Location: `RemoteScanNodes::get_remote_node()` in
`distributed_plan/node.rs`
```rust
start_time: {
let t = self.req.time_range.as_ref().map(|x| x.0).unwrap_or(0);
if self.req.histogram_interval > 0 {
histogram_bucket_start(t, self.req.histogram_interval * 1_000_000)
} else {
t
}
},
```
The arithmetic is extracted into `histogram_bucket_start(timestamp_us,
interval_us)` in `sql/histogram.rs` so the formula lives in one place.
Doing the snap at the leader means:
- All followers receive an already-aligned `start_time`
- `SimpleHistogramVisitor` / `SimpleMultiHistogramVisitor` on followers
need no changes (their `start_time % rounding_by` is already 0)
- Covers both cluster leader and super cluster leader (both go through
`generate_remote_scan_rules → get_remote_node()`)
## Test plan
- [ ] Deploy and run a histogram query spanning a non-aligned
`start_time`
- [ ] Verify first bucket count matches adjacent full buckets
- [ ] Verify queries without `histogram_interval` are unaffected
- close https://github.com/openobserve/o2-enterprise/issues/1658
## Summary
Speed up tantivy index generation during compaction by indexing parquet
row_groups in parallel.
- New dispatcher in `src/service/tantivy/mod.rs` picks one of two
builders:
- **sequential** (`sequential.rs`) — legacy producer/consumer path, used
for non-parquet inputs or when `thread_num <= 1`.
- **parallel** (`parallel.rs`) — one `SingleSegmentIndexWriter` per
parquet row_group built under `spawn_blocking` +
`Semaphore(thread_num)`, then stitched via
`tantivy::indexer::merge_indices`.
- `create_tantivy_index` now takes raw parquet `Bytes` so each
per-row_group worker gets a cheap `Bytes::clone()` instead of re-reading
the file. Call sites in `src/job/files/parquet.rs` and
`src/service/compact/merge.rs` are updated.
## Config
- `ZO_COMPACT_TANTIVY_BUILDER_THREAD_NUM` (default `2`) — per-file
concurrent row_group workers. `<= 1` keeps the legacy single-threaded
path.
## Benchmark
Input: 21 row_groups, 2,672,970 rows, ~1.38 GB .ttv output.
| variant | cpu | total tantivy | peak mem | vs main (time) | vs main
(mem) |
|---|---:|---:|---:|---:|---:|
| **main (baseline)** | 2 | 108,673 ms | **10.62 GB** | — | — |
| PR, workers=2 (default) | 2 | 109,060 ms | 2.92 GB | ≈ same | **−72%**
|
| PR, workers=4 | 4 | 92,232 ms | 3.99 GB | **−14%** | **−62%** |
| PR, workers=10 | 10 | 76,167 ms | 7.75 GB | **−27%** | **−27%** |
- **Memory is the big win.** Even sequential mode (workers=1) cuts peak
from 10.6 GB → 3.0 GB (~3.6×).
- **At the default (workers=2), wall time matches main exactly while
peak memory drops 72%.** Safe upgrade.
- **Doubling workers buys time at the cost of memory** (per-row_group
segments buffer before merge). 4 workers is the best speed/memory
trade-off; ~10 is the cap (single-threaded `merge_indices` floor is ~60
s).
## Why it's correct
OpenObserve relies on `tantivy doc_id == global parquet row_index` to
map search hits back to row offsets. The parallel path preserves it
because:
1. Each per-row_group writer writes rows in stored order → local
`doc_id` == offset inside that row_group.
2. Worker results are sorted by `rg_idx` before `merge_indices`.
3. `merge_indices` stacks segments in input order (no `sort_by_field`
path in current tantivy).
4. Parquet stores row_groups in row order, so the stitched layout
matches the original global row order.
Two guards backstop this: the merged index must be a single segment, and
its `max_doc` must equal the sum of per-row_group row counts.
## Tests
- `test_row_group_parallel_preserves_row_order` — 4 row_groups, 3
workers (forces out-of-order completion), asserts every marker's `doc_id
== row_index`.
- `test_row_group_parallel_matches_sequential` — same data through both
paths yields identical `(marker → doc_id)` mappings.
- `test_row_group_parallel_single_row_group_fast_path` —
single-row_group input skips `merge_indices` and still preserves row
order.
## Summary
Follow-up to #12249. The cross-type collision check in the file upload
handler passes the formatted `stream_name` to `get_url_jobs_for_table`,
but URL jobs are stored with the raw `table_name` from the HTTP path.
When names contain uppercase or special characters,
`format_stream_name("My_Table")` produces `"my_table"`, which won't
match the stored URL job name `"My_Table"` — causing the cross-type
protection to be silently bypassed.
## Fix
Changed `get_url_jobs_for_table(&org_id, &stream_name)` to
`get_url_jobs_for_table(&org_id, table_name.trim())` — matching how the
URL handler itself queries URL jobs at line 299.
## Before/After
```rust
// Before: formatted name passed to URL jobs lookup (WRONG)
get_url_jobs_for_table(&org_id, &stream_name)
// After: raw table_name passed (CORRECT, matches how URL handler queries)
get_url_jobs_for_table(&org_id, table_name.trim())
```
## Notes
- `ENRICHMENT_TABLES` cache key continues to use `stream_name`
(formatted) — that's correct
- `get_url_jobs_for_table` now uses `table_name.trim()` — matching the
URL handler at line 299
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Hengfei YangandClaude Opus 4.8
Huaijin
mac119
Sai Nikhilandudev
Abhay Padamani
Loakesh Indiran
Subhra264
Derk SchooltinkandHengfei Yang
Harsh Mahajan
lphuc2250gma
Don Kendall
yujinqiu
Ashish Kolhe