Changes
- Switch service graph job from ingester to alert_manager nodes
- Add UUID-based leader election to prevent duplicate writes across
multiple alert_manager instances
- Make query time range configurable via o2_config instead of a
hardcoded constant
- Skip processing streams with no new data since the last interval
(doc_time_max check)
- Downgrade no-trace-streams log from warn to debug
## Summary
Implement automatic provisioning and lifecycle management of
per-organization system service accounts for the AI-powered SRE Agent.
These accounts serve as the identity for RCA (Root Cause Analysis) agent
API interactions and are created automatically for each organization.
## Key Features
### Service Account Provisioning
- Database migration creates SRE Agent accounts for all existing
organizations
- Deterministic email format:
`o2-sre-agent.org-<ORG_ID>@openobserve.internal`
- System accounts are automatically created on organization creation
- Idempotent operations ensure safe re-runs and self-healing
### System Account Protection
- API endpoints prevent modification/deletion of system service accounts
(403 Forbidden)
- User responses include `is_system` flag to identify system accounts
- Protection enforced at HTTP handler layer
### RCA Agent Client Refactoring
- Singleton pattern: global client initialized once at startup
- Per-request user credential forwarding for better audit trails and
scoping
- Lazy initialization: returns None if agent service not configured
- Updated all query methods to accept and forward auth credentials
### Incident Correlation Enhancement
- Query `ServiceIdentityConfig` to get configured semantic dimension
groups
- Extract only `distinguish_by` fields from matched identity sets
- Use parallel correlation execution (Service Discovery + semantic
extraction)
- Improved accuracy by using configured groups instead of fallback label
extraction
### Improved Token Redaction
- Consistent 12-character format: first 4 visible characters + 8
asterisks
- Better security for token display in UI
- Updated tests for new redaction format
## Technical Details
### Database Changes
- New migration:
`m20260331_000001_create_sys_rca_agent_service_accounts`
- Creates accounts with organization and RUM tokens
- Safe pagination (100 orgs at a time) with idempotency checks
- System accounts use role type 6 (SreAgent, distinct from
ServiceAccount)
### Authorization
- System accounts added to OpenFGA with standard org membership tuples
- SRE-specific read-only role for permission control
- Self-healing: runtime ensures OpenFGA state is consistent with
database state
### API Changes
- GET /users: Returns `is_system` and `description` fields
- PUT/DELETE /users: Reject modification/deletion of system accounts
## Test Coverage
- Token redaction tests updated for new format
- System service account protection verified
- User list pagination tested with system accounts
## Feature Flag
This is an enterprise feature gated behind `#[cfg(feature =
"enterprise")]`. System accounts are always created, but RCA integration
only works if the agent service is configured.
## Deployment
- Zero-downtime: migration is additive
- Opt-in for OSS deployments
- Automatic provisioning on upgrade and org creation
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
**Issue**
Logs query generation used raw field names in SELECT [FIELD_LIST] and
non-SQL WHERE filters. If a field name is a SQL reserved word (for
example user, role), the generated SQL could be invalid or fail to
parse.
**Fix**
Field names are now consistently wrapped in double quotes when building
SQL:
- Quick mode field list ([FIELD_LIST])
- Non-SQL filter tokens converted into WHERE clauses
Tests were updated to assert quoted-field behavior for both quick mode
and filter conversion.
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Summary
Design at:
https://github.com/openobserve/designs/blob/main/infrastructure/work-group/work_group_v2_impl.md
This PR adds slot-based distributed admission control for search queries
and fixes file dispatch consistency when a subset of querier nodes is
selected per query.
To enable this feature we need to set:
```
O2_WORK_GROUP_MAX_NODES_PER_QUERY > 0
O2_WORK_GROUP_NODE_SELECTION_STRATEGY = org | stream
```
### Slot-based admission
- Add `TryAcquire / StartQuery / ReleaseQuery` gRPC RPCs — Follow nodes
reserve slots before execution starts and self-release via
`FlightEncoderStream::drop`
- Wire slot admission in both Arrow Flight search and PromQL paths,
co-located with the existing org/user concurrency queue check
- Delegate DataFusion `target_partitions` / `memory_size` to enterprise
`get_resource_limit()`, which applies per-query slot fraction with
optional dynamic expansion
- TTL sweep interval derived from `slot_reserved_ttl_ms / 4` (min 100
ms) instead of a hardcoded 500 ms
- Move `QUERY_PENDING_NUMS` / `QUERY_RUNNING_NUMS` metric updates to
after slot admission completes (a blocked request is no longer counted
as "running")
### Consistent-hash file dispatch fix (`max_nodes_per_query > 0`)
- Add `get_node_from_consistent_hash_within(key, role, group, allowed)`
to `infra::cluster`: walks the global ring clockwise from `hash(key)`
but skips nodes outside `allowed`. Preserves all consistent-hash
properties (even distribution via vnodes, minimal remapping on
membership change) while confining dispatch to the selected subset
- `partition_file_by_hash` now passes only the selected querier nodes as
the `allowed` set, so every file maps to one of the N selected nodes —
no more fallback-to-index-0 imbalance when `strategy=org/stream`
- File-list broadcast (enterprise) pre-computes per-item target nodes
using the same filtered ring walk via `broadcast_item_targets()` in
`o2-enterprise`, so broadcast and dispatch always agree on which node
owns each file
### Proto fix
- Reserve field 10 in `QueryStatus` (was `WorkGroup` enum / varint wire
type) and move `work_group` string to field 13 — prevents wire-type
mismatch during rolling upgrades
### Other fixes from code review
- `get_first_stream_key()` now returns `"{stream_type}/"` sentinel
instead of `""` for multi-stream / cross-index queries, so
`select_nodes` always receives a non-empty deterministic key
- PromQL node selection uses `"metrics/"` stream key (consistent with
the sentinel format) instead of bare `"metrics"`
- Move `_orig_trace_id` binding inside `#[cfg(feature = "enterprise")]`
block in `FlightEncoderStream::drop`
## Key config (all default to safe/off values)
| Env var | Default | Description |
|---|---|---|
| `O2_WORK_GROUP_MAX_NODES_PER_QUERY` | `0` | Nodes per query; 0 = all
nodes (disables slot admission) |
| `O2_WORK_GROUP_NODE_SELECTION_STRATEGY` | `all` | `all` / `org` /
`stream` |
| `O2_WORK_GROUP_SHORT_SLOT_RATIO` | `0.8` | Max fraction of node slots
for short queries |
| `O2_WORK_GROUP_DYNAMIC_RESOURCE` | `true` | Expand DataFusion
allocation on idle nodes |
## Test plan
- [x] Default (`max_nodes_per_query=0`): behavior identical to before,
zero overhead
- [x] `strategy=org`, `max_nodes_per_query=3`, 10-node cluster: each org
consistently uses the same 3 nodes; files distributed evenly across
those 3 nodes (no node-0 pile-up)
- [x] Broadcast routes each file to the same node that
`partition_file_by_hash` selects at query time → cache hits on
re-queries
- [x] Slot admission: 3rd request waits when budget=2 and 2 queries
running; slots released on completion
- [x] PromQL queries respect slot budget
- [x] Rolling upgrade: old node sends `QueryStatus` without field 10
conflict
## Related
Requires companion PR in `o2-enterprise`:
https://github.com/openobserve/o2-enterprise/pull/1378🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
So far, service account token could be viewed and copied even after an
SA was created. This was bad design and posed risk of impersonation.
The changes prevent this from happening by returning only the redacted
token in the API response. Full token with create API itself, and
removing GET API to get the service account details (which returned raw
token).
The same affects token rotation as well.
We need to have open fga enabled to support non-admin roles. However
currently the create user api allows creating non-admin users even when
ofga is not enabled. Those are treated as admin users, which causes
further confusion.
This PR adds a check so that non-admin roles are blocked at creation if
ofga is not enabled.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Currently, when a pipeline is disabled, the scheduled job we don't
delete, but we check after 5mins. 5mins is too frequent to check this,
instead we should follow the same like alerts and reports that checks
again after a week.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yashodhan@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Remove O2_SERVICE_GRAPH_ENABLED and O2_SERVICE_GRAPH_QUERY_TIME_RANGE_MINUTES.
Service graph now runs unconditionally on enterprise ingester nodes.
Query window defaults to 60m constant; UI time range picker controls actual queries.
Also remove checks for this on the UI. Keep only enterprise level checks.
Currently dashboard with Multi-window set up does not really run the
result array vrl functions. This pr adds support for vrl function when
`/_search_multi_stream` is used. When `per_query_response` is true, and
the vrl function is result array vrl function, the function is applied
on the whole combined response (including both the time windows). The
input of the result array vrl function becomes a 2d array where each
child array represent a time window.
Otherwise, if `per_query_response` is false, or the vrl function is not
result array vrl function, then the function is applied individually on
every result of every time-window queries.
ByteBaker
漂流
HuaijinandHengfei Yang
GobinathALandHengfei Yang
Subhra264
Yashodhan JoshiandYashodhan Joshi
udev
Sai Nikhil
vnghia
Franklyn Dsouza
Ashish Kolhe