## Summary
Design at:
https://github.com/openobserve/designs/blob/main/infrastructure/work-group/work_group_v2_impl.md
This PR adds slot-based distributed admission control for search queries
and fixes file dispatch consistency when a subset of querier nodes is
selected per query.
To enable this feature we need to set:
```
O2_WORK_GROUP_MAX_NODES_PER_QUERY > 0
O2_WORK_GROUP_NODE_SELECTION_STRATEGY = org | stream
```
### Slot-based admission
- Add `TryAcquire / StartQuery / ReleaseQuery` gRPC RPCs — Follow nodes
reserve slots before execution starts and self-release via
`FlightEncoderStream::drop`
- Wire slot admission in both Arrow Flight search and PromQL paths,
co-located with the existing org/user concurrency queue check
- Delegate DataFusion `target_partitions` / `memory_size` to enterprise
`get_resource_limit()`, which applies per-query slot fraction with
optional dynamic expansion
- TTL sweep interval derived from `slot_reserved_ttl_ms / 4` (min 100
ms) instead of a hardcoded 500 ms
- Move `QUERY_PENDING_NUMS` / `QUERY_RUNNING_NUMS` metric updates to
after slot admission completes (a blocked request is no longer counted
as "running")
### Consistent-hash file dispatch fix (`max_nodes_per_query > 0`)
- Add `get_node_from_consistent_hash_within(key, role, group, allowed)`
to `infra::cluster`: walks the global ring clockwise from `hash(key)`
but skips nodes outside `allowed`. Preserves all consistent-hash
properties (even distribution via vnodes, minimal remapping on
membership change) while confining dispatch to the selected subset
- `partition_file_by_hash` now passes only the selected querier nodes as
the `allowed` set, so every file maps to one of the N selected nodes —
no more fallback-to-index-0 imbalance when `strategy=org/stream`
- File-list broadcast (enterprise) pre-computes per-item target nodes
using the same filtered ring walk via `broadcast_item_targets()` in
`o2-enterprise`, so broadcast and dispatch always agree on which node
owns each file
### Proto fix
- Reserve field 10 in `QueryStatus` (was `WorkGroup` enum / varint wire
type) and move `work_group` string to field 13 — prevents wire-type
mismatch during rolling upgrades
### Other fixes from code review
- `get_first_stream_key()` now returns `"{stream_type}/"` sentinel
instead of `""` for multi-stream / cross-index queries, so
`select_nodes` always receives a non-empty deterministic key
- PromQL node selection uses `"metrics/"` stream key (consistent with
the sentinel format) instead of bare `"metrics"`
- Move `_orig_trace_id` binding inside `#[cfg(feature = "enterprise")]`
block in `FlightEncoderStream::drop`
## Key config (all default to safe/off values)
| Env var | Default | Description |
|---|---|---|
| `O2_WORK_GROUP_MAX_NODES_PER_QUERY` | `0` | Nodes per query; 0 = all
nodes (disables slot admission) |
| `O2_WORK_GROUP_NODE_SELECTION_STRATEGY` | `all` | `all` / `org` /
`stream` |
| `O2_WORK_GROUP_SHORT_SLOT_RATIO` | `0.8` | Max fraction of node slots
for short queries |
| `O2_WORK_GROUP_DYNAMIC_RESOURCE` | `true` | Expand DataFusion
allocation on idle nodes |
## Test plan
- [x] Default (`max_nodes_per_query=0`): behavior identical to before,
zero overhead
- [x] `strategy=org`, `max_nodes_per_query=3`, 10-node cluster: each org
consistently uses the same 3 nodes; files distributed evenly across
those 3 nodes (no node-0 pile-up)
- [x] Broadcast routes each file to the same node that
`partition_file_by_hash` selects at query time → cache hits on
re-queries
- [x] Slot admission: 3rd request waits when budget=2 and 2 queries
running; slots released on completion
- [x] PromQL queries respect slot budget
- [x] Rolling upgrade: old node sends `QueryStatus` without field 10
conflict
## Related
Requires companion PR in `o2-enterprise`:
https://github.com/openobserve/o2-enterprise/pull/1378🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
So far, service account token could be viewed and copied even after an
SA was created. This was bad design and posed risk of impersonation.
The changes prevent this from happening by returning only the redacted
token in the API response. Full token with create API itself, and
removing GET API to get the service account details (which returned raw
token).
The same affects token rotation as well.
Remove O2_SERVICE_GRAPH_ENABLED and O2_SERVICE_GRAPH_QUERY_TIME_RANGE_MINUTES.
Service graph now runs unconditionally on enterprise ingester nodes.
Query window defaults to 60m constant; UI time range picker controls actual queries.
Also remove checks for this on the UI. Keep only enterprise level checks.
Currently dashboard with Multi-window set up does not really run the
result array vrl functions. This pr adds support for vrl function when
`/_search_multi_stream` is used. When `per_query_response` is true, and
the vrl function is result array vrl function, the function is applied
on the whole combined response (including both the time windows). The
input of the result array vrl function becomes a 2d array where each
child array represent a time window.
Otherwise, if `per_query_response` is false, or the vrl function is not
result array vrl function, then the function is applied individually on
every result of every time-window queries.
Backend:
- New enterprise-only endpoint: GET /{org_id}/traces/service_graph/edge/history
- Returns time-series P50/P95/P99 latency with request-count-weighted baselines
- OSS returns 403; empty result returned gracefully if stream doesn't exist
Frontend — Interaction Model:
- Click-to-panel replaced with hover tooltip for edge interactions
- Tooltip shows compact stats (request count, error rate, latencies) on hover
- "Show trend" button fetches and renders a live P50/P95/P99 sparkline inline
- Tooltip stays open when moused-over; results cached per edge key
- Edge panel component retained, now includes its own latency trend chart section
Frontend — Edge Coloring:
- Edges now colored relative to 24h weighted-average P95 baseline when available
- Falls back to absolute threshold coloring when no baseline data exists
Frontend — Chart Rendering:
- Switched to SVG render mode for sharper output
- devicePixelRatio passed to chart init to fix blurry Retina display rendering
- View type switch no longer recreates chart (no chartKey increment); prevents replay of expand animation
- Tooltip handlers re-registered after every view switch for both tree and graph modes
Tests:
- Updated to reflect removed edge panel flow
- chartKey stability on view switch now asserted (was: increment expected)
- Wrap append() and record_alert() in DB transactions to fix
read-modify-write race conditions on concurrent alert correlation
- Add length/blank validation on comment endpoint (max 10,000 chars)
- Guard update_severity from emitting SeverityOverride on no-op changes
- Add TitleChanged event type with from/to/user_id fields, wire into
update_title, render in activity timeline
Incident Events:
- New `incident_events` table (JSONB array per incident, PK on `(org_id, incident_id)`)
- `IncidentEvent` struct with 12 typed variants: `Created`, `Alert`, `SeverityUpgrade`,
`SeverityOverride`, `Acknowledged`, `Resolved`, `Reopened`, `DimensionsUpgraded`,
`AssignmentChanged`, `Comment`, `AIAnalysisBegin`, `AIAnalysisComplete`
- Constructors on `IncidentEvent` (e.g. `::alert()`, `::resolved()`, `::comment()`)
- `increment_alert(&mut self)` for compaction: same alert only compacts if it's the
last event in the array, preserving interleaving (A x5, B x3, A x2)
- Severity fields use `IncidentSeverity` enum, not strings
Backend:
- Migration: `m20260214_000001_create_incident_events_table`
- DB ops: `init`, `get`, `append`, `record_alert` (with compaction), `delete`
- Emit events at all mutation points: incident creation, alert correlation,
dimensions upgrade, status change, severity change, auto-resolve
- `user_id` passed from HTTP auth headers through to all events
- `auto_resolve_stale()` now returns resolved IDs for event emission
- API: `GET /v2/{org_id}/alerts/incidents/{id}/events`
- API: `POST /v2/{org_id}/alerts/incidents/{id}/events/comment`
Frontend:
- `IncidentTimeline.vue`: vertical timeline with icon/color per event type,
alert compaction display, comment input, refresh button
- Activity tab added to `IncidentDetailDrawer.vue`
- `incidents.ts`: `getEvents()` and `postComment()` API methods
Currently, on the UI when the test destination button is clicked for a
prebuilt email destination, it does not do anything at all. Instead, it
should call the test destination api. Also, this destination api does
not handle email destination testing, this pr fixes this as well.
ByteBaker
Sai Nikhil
udev
Subhra264
Huaijin
Ashish Kolhe
Omkar Kesarkhane
Maxime Grenu
