Close#12345
## Problem
OTLP histogram metrics sent over HTTP/JSON (e.g. otel-collector with
`encoding: json`) created streams but stored 0 events, with no
`_count`/`_sum`/`_bucket` streams. Int-valued counters/gauges were
silently recorded as 0. The protobuf path was unaffected, which is why
this only surfaced for collectors explicitly configured with JSON
encoding.
## Root cause
`otlp_json` deserializes the body directly into opentelemetry-proto's
serde-derived types, which only partially implement the proto3 JSON
mapping (64-bit integers encoded as strings, enums by name, zero/empty
fields omitted). `Metric.data` is a flattened `Option` oneof, and serde
converts any inner deserialization error of a flattened `Option` into
`None` instead of failing — so the whole metric silently became `data:
None`, while the stream itself had already been created by the schema
setup. Same mechanism on `NumberDataPoint.value` turned string `asInt`
values into 0.
## Changes
- Upgrade opentelemetry stack 0.31 → 0.32, which fixes string-encoded
u64 handling for plain histograms and summaries upstream.
- Add `otlp_json_compat` normalizer for the gaps still present in 0.32
(each verified empirically, with unit tests):
- string `asInt` values on data points and exemplars
- `aggregationTemporality` as enum name
- `Exemplar`'s value oneof is not flattened in 0.32
- missing `serde(default)` on
`ExponentialHistogramDataPoint`/`Buckets`/`ValueAtQuantile`/`Exemplar`,
which rejected omitted zero/empty fields
- Count and log metrics whose data oneof failed to decode
(`partial_success.rejected_data_points`) instead of silently skipping
them, and stop overwriting specific partial-success error messages with
the generic retention message.
- Adapt to OTLP 0.32 API additions (`KeyValue.key_strindex`,
`AnyValue::StringValueStrindex`).
- Upgrade workspace reqwest 0.12 → 0.13, required because
opentelemetry-http 0.32 only implements `HttpClient` for reqwest 0.13
(`rustls` feature replaces `rustls-tls-native-roots` via the platform
verifier; `json`/`query`/`form` are now explicit features). The
`segment` crate is pinned to reqwest 0.12, so `TELEMETRY_CLIENT` moved
into `telemetry.rs` as a plain `reqwest::Client` that posts the Track
message itself, replicating `segment::HttpClient::send`.
- Remove the unused pyroscope profiling integration (optional feature,
dependencies, agent setup, `ZO_PROF_PYROSCOPE_*` config).
## Verification
- 10 unit tests in `otlp_json_compat` covering the full payload matrix
(collector-style string integers, enum names, omitted zero fields,
exemplars with and without trace context, already-normalized payloads,
malformed input).
- End-to-end: posted the issue's collector-style JSON payload to
`/api/default/v1/metrics` — `_bucket` (3 events, correct cumulative
counts with exemplars attached), `_count`, `_sum` populated for
histogram and exponential histogram, summary quantiles stored, `asInt`
counter stored as 42 instead of 0.
- `cargo clippy -- -D warnings` clean; lib test failures are
pre-existing local-environment issues that reproduce identically on
main.
#12351
Fixed RUM middleware in `middleware_data.rs` to extract
`ootags`/`o2tags` from the request body when the field is not present in
URL query parameters. Previously, the middleware only inspected query
parameters for these fields, but the RUM SDK sends them in the request
body.
## Why
User logs confirmed the issue: `ootags raw=None, o2tags raw=None`
appeared consistently in the middleware logs even when the RUM SDK
included tags like `env`, `service`, `version`, and `sdk_version`. Geo
keys (which come from a different extraction path) were unaffected,
narrowing the root cause to query-param-only parsing of
`ootags`/`o2tags`. The RUM SDK sends these tags in the POST body, not as
query string parameters.
## Approach
Added body-parsing logic to read and decode the request body when
`ootags`/`o2tags` are absent from query parameters. The body is parsed
once using `url::form_urlencoded` and merged into the existing key-value
map before the `filter_tags` step runs, ensuring the same downstream
pipeline works unchanged.
---------
Co-authored-by: Neha P <neha@openobserve.ai>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Co-authored-by: Neha P <144308844+neha00290@users.noreply.github.com>
## Summary
Design at:
https://github.com/openobserve/designs/blob/main/infrastructure/bloomfilter/DESIGN.md
A bloom-filter pruning layer that sits **between `file_list` and
tantivy** in the search path. It drops candidate files before any `.ttv`
is opened, for **positive equality / IN** lookups on high-cardinality
fields (`trace_id`, `request_id`, …).
It is built **on top of** tantivy: the `.bf` is constructed by iterating
the tantivy term dictionary, so a field must be a tantivy secondary
index (`index_fields`) **and** opted into `bloom_filter_fields`. Only
the intersection is built — a bloom-only field produces nothing. To make
that intersection non-empty by construction, enabling a field for bloom
now folds it into `index_fields` automatically (see Config). Values are
not inspected; the operator chooses which fields are worth it (see
guidance below).
Design notes live in the module-level docs under `src/infra/src/bloom/`
(`mod.rs`, `writer.rs`, `reader.rs`, `sbbf.rs`).
## Design
- **Transposed (block-major) `.bf` layout.** All files in a `.bf` share
`num_blocks = B`, so `block_index = fastmap(hash(value), B)` is
identical across files and the blocks a query needs form **one
contiguous row**. The search side reads **one `M × 32`-byte row per
`.bf`** to answer membership for all `M` files — prune cost is
`O(groups)`, not `O(files)`. (`infra::bloom`, own ~120-LOC SBBF over
gxhash64.)
- **Sub-grouping.** A hour's new files are sorted by record count and
split into chunks of ≤ `ZO_BLOOM_FILTER_MAX_FILES_PER_BF` (default 256),
one `.bf` each. Bounds compactor build memory (`M × B × 32`) and `.bf`
object size; query reads become `ceil(files / 256)` rows.
- **Footer-only reader + batched fetch.** Footer (≈ `M × 12` B) comes
from `BLOOM_FOOTER_CACHE` or one 16 KB suffix GET (with a precise
re-read when a large footer overflows the probe); rows are pulled with
one `cache_storage::get_ranges` call. `cache_storage` was reworked to do
real batched cache reads (memory: 1 lookup + N slices; disk: 1
`File::open` + N `pread`).
- **B sizing.** `num_blocks_for(max_records_in_chunk,
ZO_BLOOM_FILTER_FPP)` (default fpp `0.01`) — record count is a safe NDV
upper bound, so the SBBF never saturates for high-cardinality fields.
- **Versioned, with orphan cleanup.** A file's `bloom_ver` is write-once
— a row is never re-stamped, so dumped rows keep pointing at the `.bf`
they were built into. But `bloom_ver` is **not** append-only at the
bucket level: a merge produces new output files with a fresh `bloom_ver`
and deletes the small inputs, so the old inputs' `.bf` can become
orphaned. After each merge round the compactor retires any `.bf` whose
`bloom_ver` is no longer referenced by a live `file_list` row in the
bucket (enqueued through the existing `file_list_deleted` path).
`bloom_ver` is preserved through every file_list INSERT/SELECT path
(incl. the SQLite read-cache) and the dump parquet.
- **Observability.** Prometheus metrics for prune
(`bloom_prune_keep_ratio`, `bloom_prune_duration_seconds`);
`bloom-inspect` CLI dumps a `.bf` footer (fields, B, row bytes, file
names); footer-cache hit/miss via the shared `BytesCache` tagged
metrics; `Local::get_ranges` now reports `STORAGE_READ_*` (previously
bypassed).
## When it helps (measured, S3, no disk cache)
Same stream `benchtest`, 170 files / 14.64 GB index, single matching
row. The only variable is whether the field's values are
time-correlated:
| Field | Type | tantivy only | bloom + tantivy | Outcome |
|---|---|---|---|---|
| `trace_id` (16-byte random) | random | **2584 ms** (opens all 170
`.ttv`) | 24 ms prune + 65 ms = **89 ms** | **~29× faster** |
| `request_id` (`…-7e74-…`, UUIDv7) | time-ordered | **154 ms** (sparse
index range-prunes) | 42 ms + 154 ms = **196 ms** | ~1.3× **slower** |
The whole 170-file group was answered by **one row read of 5440 bytes**
(`170 × 32`) + a cached footer.
**Guidance — enable only when all three hold (advisory; not enforced):**
1. **High cardinality** field.
2. **Random values** (not time-ordered) — so tantivy can't range-prune
and must scan every file. Time-ordered ids (UUIDv7, snowflake) are
already pruned by tantivy's sparse index in memory → bloom is pure
overhead there.
3. **Many files per hour** — hourly volume ≳ 10 ×
`ZO_COMPACT_MAX_FILE_SIZE` (≳10 `.ttv` files), so there's something to
prune. Bloom turns N file opens into one row read; at 1–2 files/hour
it's net overhead.
The 29× result above had 170 files/hour all answered by one `.bf`. ❌
Low-cardinality / non-indexed / range / `LIKE` / regex / `!=` / OR: no
effect.
## Config
- `ZO_BLOOM_FILTER_ENABLED` (default true) — global gate.
- Per stream: a field must be in **both** `index_fields` and
`bloom_filter_fields`. Adding a field to `bloom_filter_fields` auto-adds
it to `index_fields` (bloom is built on the secondary index), and the
default index fields include the default bloom fields.
- `ZO_BLOOM_FILTER_FPP` (default 0.01) — target false-positive
probability; sizes the SBBF block count (smaller fpp = fewer false
survivors, larger `.bf`). Out-of-range falls back to 0.01.
- `ZO_BLOOM_FILTER_MAX_FILES_PER_BF` (default 256) — files per `.bf`;
trades query reads vs build memory.
- `ZO_BLOOM_FILTER_DEFAULT_FIELDS` (default empty) — fields bloomed for
every stream.
- `ZO_BLOOM_FOOTER_CACHE_MAX_SIZE` (default 1% mem, clamped 32–256 MB).
Removed in this PR: `ZO_BLOOM_FILTER_DISABLED_ON_SEARCH`,
`ZO_BLOOM_FILTER_NDV_RATIO` (B is now sized from record count + fpp, not
an ndv ratio).
## Test plan
- [x] `cargo build`, `cargo clippy -p infra -p openobserve --lib -- -D
warnings`, `cargo fmt --check` clean
- [x] Unit tests: `cargo test -p infra --lib bloom`, `cargo test -p
openobserve --lib bloom`
- [x] E2E on S3 (results above): random field prunes 170→1 and is ~29×
faster; transposed read = 1 request / 5440 B per group
- [x] Reviewer: confirm gating expectations (operator opts fields in; no
auto value detection)
- [x] Test: orphan bloom: 2 * 256MB -> merge to 512MB -> merge to
1024MB, the first .bf should be deleted
- [x] Test: orphan bloom: 6 * 256MB -> merge to 1GB + 512MB -> 512 MB
merge to 1024MB, the first .bf include 2 files(1GB + 512MB), after merge
again, the first still have 2 files but one of them already deleted. we
should keep the first .bf, because still have files in use.
## Notes
- `.bf` format is `VERSION 0x01` (transposed). Not compatible with any
pre-merge experimental `.bf`; rebuild by wiping the `bloom/` dir and
resetting `bloom_ver=0`.
- Hash is gxhash64 (project-wide `config::utils::hash`), not the Parquet
spec's xxhash64 — `.bf` is internal, never read by external tools.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
related to
- https://github.com/openobserve/openobserve/pull/8003
- https://github.com/openobserve/openobserve/issues/7986
# Why We Can Delete `enable_align_histogram`
## What It Was
`enable_align_histogram` was a query parameter introduced in [PR
#8003](https://github.com/openobserve/openobserve/pull/8003) to fix
[Issue #7986](https://github.com/openobserve/openobserve/issues/7986).
The bug was that in the old two-step search flow (`_search_partition` →
`_search`), the two APIs independently generated histogram intervals,
leading to mismatched time buckets and inaccurate chart rendering in
dashboards.
The flag told `_search_partition` to:
1. Align partition boundaries to histogram interval boundaries (e.g.,
round to the start of a minute/hour/day).
2. Return the computed `histogram_interval` in the response so the
frontend could pass it to `_search`.
## Why It Is Now Dead Code
### 1. The frontend no longer calls `_search_partition`
All frontend search now uses `_search_stream` (HTTP/2 streaming), which
internally calls `search_partition()` with `enable_align_histogram =
false`. The old `_search_partition` endpoint is no longer consumed by
the UI. The only frontend caller (`getValuesPartition` in
`IndexList.vue`) is defined but never invoked.
### 2. `search_stream` handles alignment inherently
In the streaming search flow, partitions themselves define the search
time ranges. Each partition search runs on the exact aligned time
boundaries returned by the partition step. There is no separate
histogram query that needs an externally provided interval — alignment
is "baked into" the partition boundaries.
### 3. Histogram queries are already covered by `is_complex_query`
For histogram queries (which have `GROUP BY`), `is_complex_query`
returns `true`. Combined with `has_ts_column = true`, this triggers
`calculate_partition_settings()` to align `min_step` to the histogram
interval — the same alignment behavior that `enable_align_histogram`
previously provided, but without the extra flag.
### 4. All backend callers pass `false`
Of the 5 callers of `search_partition()`:
| Caller | Value |
|--------|-------|
| gRPC handler | hardcoded `false` |
| streaming execution | hardcoded `false` |
| traces handler | hardcoded `false` |
| HTTP `_search_partition` | from query param (frontend always passed
`true`, but frontend no longer calls this) |
| HTTP `_search_partition_multi` | from query param (frontend always
passed `true`, but frontend no longer calls this) |
No active code path ever sets `enable_align_histogram = true`.
## What Was Removed
- `get_enable_align_histogram_from_request()` — the HTTP query param
parser
- The `enable_align_histogram` parameter from:
- `search_partition()` function signature
- `search_partition_multi()` function signature
- `calculate_partition_settings()` function signature
- The alignment logic in `search_partition()`:
- `add_mini_partition` (always `false` without the flag)
- `PartitionGenerator` align parameter (now driven by `is_histogram`
alone)
- `resp.histogram_interval` override block
- The `|| enable_align_histogram` branch in
`calculate_partition_settings`
- The query parameter from both HTTP handler `utoipa` specs
- The `false` argument from all 5 call sites
- fix incidents not correlating for alert ID
- SRE SA no more visible in other screens than main SA list
- SreAgent role cannot be assigned to any other user now
- regular user now working for MCP
When validating access token, we should use the email returned from db
query to do `check_permissions` rbac function call. Reason is, in
openfga db, we store the same user email as stored in the openobserve
db. Hence, all the permission is supposed to be evaluated against the
same email stored in the db.
The email decoded from token can have any case (upper/lower) as per
user's sso config, so it is important to follow the case stored in the
db created when the user logged in for the first time.
## Summary
Implement automatic provisioning and lifecycle management of
per-organization system service accounts for the AI-powered SRE Agent.
These accounts serve as the identity for RCA (Root Cause Analysis) agent
API interactions and are created automatically for each organization.
## Key Features
### Service Account Provisioning
- Database migration creates SRE Agent accounts for all existing
organizations
- Deterministic email format:
`o2-sre-agent.org-<ORG_ID>@openobserve.internal`
- System accounts are automatically created on organization creation
- Idempotent operations ensure safe re-runs and self-healing
### System Account Protection
- API endpoints prevent modification/deletion of system service accounts
(403 Forbidden)
- User responses include `is_system` flag to identify system accounts
- Protection enforced at HTTP handler layer
### RCA Agent Client Refactoring
- Singleton pattern: global client initialized once at startup
- Per-request user credential forwarding for better audit trails and
scoping
- Lazy initialization: returns None if agent service not configured
- Updated all query methods to accept and forward auth credentials
### Incident Correlation Enhancement
- Query `ServiceIdentityConfig` to get configured semantic dimension
groups
- Extract only `distinguish_by` fields from matched identity sets
- Use parallel correlation execution (Service Discovery + semantic
extraction)
- Improved accuracy by using configured groups instead of fallback label
extraction
### Improved Token Redaction
- Consistent 12-character format: first 4 visible characters + 8
asterisks
- Better security for token display in UI
- Updated tests for new redaction format
## Technical Details
### Database Changes
- New migration:
`m20260331_000001_create_sys_rca_agent_service_accounts`
- Creates accounts with organization and RUM tokens
- Safe pagination (100 orgs at a time) with idempotency checks
- System accounts use role type 6 (SreAgent, distinct from
ServiceAccount)
### Authorization
- System accounts added to OpenFGA with standard org membership tuples
- SRE-specific read-only role for permission control
- Self-healing: runtime ensures OpenFGA state is consistent with
database state
### API Changes
- GET /users: Returns `is_system` and `description` fields
- PUT/DELETE /users: Reject modification/deletion of system accounts
## Test Coverage
- Token redaction tests updated for new format
- System service account protection verified
- User list pagination tested with system accounts
## Feature Flag
This is an enterprise feature gated behind `#[cfg(feature =
"enterprise")]`. System accounts are always created, but RCA integration
only works if the agent service is configured.
## Deployment
- Zero-downtime: migration is additive
- Opt-in for OSS deployments
- Automatic provisioning on upgrade and org creation
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Summary
Design at:
https://github.com/openobserve/designs/blob/main/infrastructure/work-group/work_group_v2_impl.md
This PR adds slot-based distributed admission control for search queries
and fixes file dispatch consistency when a subset of querier nodes is
selected per query.
To enable this feature we need to set:
```
O2_WORK_GROUP_MAX_NODES_PER_QUERY > 0
O2_WORK_GROUP_NODE_SELECTION_STRATEGY = org | stream
```
### Slot-based admission
- Add `TryAcquire / StartQuery / ReleaseQuery` gRPC RPCs — Follow nodes
reserve slots before execution starts and self-release via
`FlightEncoderStream::drop`
- Wire slot admission in both Arrow Flight search and PromQL paths,
co-located with the existing org/user concurrency queue check
- Delegate DataFusion `target_partitions` / `memory_size` to enterprise
`get_resource_limit()`, which applies per-query slot fraction with
optional dynamic expansion
- TTL sweep interval derived from `slot_reserved_ttl_ms / 4` (min 100
ms) instead of a hardcoded 500 ms
- Move `QUERY_PENDING_NUMS` / `QUERY_RUNNING_NUMS` metric updates to
after slot admission completes (a blocked request is no longer counted
as "running")
### Consistent-hash file dispatch fix (`max_nodes_per_query > 0`)
- Add `get_node_from_consistent_hash_within(key, role, group, allowed)`
to `infra::cluster`: walks the global ring clockwise from `hash(key)`
but skips nodes outside `allowed`. Preserves all consistent-hash
properties (even distribution via vnodes, minimal remapping on
membership change) while confining dispatch to the selected subset
- `partition_file_by_hash` now passes only the selected querier nodes as
the `allowed` set, so every file maps to one of the N selected nodes —
no more fallback-to-index-0 imbalance when `strategy=org/stream`
- File-list broadcast (enterprise) pre-computes per-item target nodes
using the same filtered ring walk via `broadcast_item_targets()` in
`o2-enterprise`, so broadcast and dispatch always agree on which node
owns each file
### Proto fix
- Reserve field 10 in `QueryStatus` (was `WorkGroup` enum / varint wire
type) and move `work_group` string to field 13 — prevents wire-type
mismatch during rolling upgrades
### Other fixes from code review
- `get_first_stream_key()` now returns `"{stream_type}/"` sentinel
instead of `""` for multi-stream / cross-index queries, so
`select_nodes` always receives a non-empty deterministic key
- PromQL node selection uses `"metrics/"` stream key (consistent with
the sentinel format) instead of bare `"metrics"`
- Move `_orig_trace_id` binding inside `#[cfg(feature = "enterprise")]`
block in `FlightEncoderStream::drop`
## Key config (all default to safe/off values)
| Env var | Default | Description |
|---|---|---|
| `O2_WORK_GROUP_MAX_NODES_PER_QUERY` | `0` | Nodes per query; 0 = all
nodes (disables slot admission) |
| `O2_WORK_GROUP_NODE_SELECTION_STRATEGY` | `all` | `all` / `org` /
`stream` |
| `O2_WORK_GROUP_SHORT_SLOT_RATIO` | `0.8` | Max fraction of node slots
for short queries |
| `O2_WORK_GROUP_DYNAMIC_RESOURCE` | `true` | Expand DataFusion
allocation on idle nodes |
## Test plan
- [x] Default (`max_nodes_per_query=0`): behavior identical to before,
zero overhead
- [x] `strategy=org`, `max_nodes_per_query=3`, 10-node cluster: each org
consistently uses the same 3 nodes; files distributed evenly across
those 3 nodes (no node-0 pile-up)
- [x] Broadcast routes each file to the same node that
`partition_file_by_hash` selects at query time → cache hits on
re-queries
- [x] Slot admission: 3rd request waits when budget=2 and 2 queries
running; slots released on completion
- [x] PromQL queries respect slot budget
- [x] Rolling upgrade: old node sends `QueryStatus` without field 10
conflict
## Related
Requires companion PR in `o2-enterprise`:
https://github.com/openobserve/o2-enterprise/pull/1378🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
So far, service account token could be viewed and copied even after an
SA was created. This was bad design and posed risk of impersonation.
The changes prevent this from happening by returning only the redacted
token in the API response. Full token with create API itself, and
removing GET API to get the service account details (which returned raw
token).
The same affects token rotation as well.
udev
Hengfei Yang
Subhra264
Huaijin
Loakesh Indiran
Ashish Kolhe
Harsh Mahajan
ByteBaker
Sai prasanna sastry
漂流