sktrpathi and sai nikhil kethe
e52d000217
feat: add incident correlation system ( #9507 )
...
## Summary
Introduces comprehensive incident management system to reduce alert
fatigue through intelligent correlation and AI-powered root cause
analysis.
**Design at:**
https://github.com/openobserve/designs/tree/main/alert-correlation
---
## Incident Correlation
**3-Tier Correlation Priority:**
1. **Trace-based**: Groups alerts from same distributed trace
(cross-service)
2. **Service Discovery**: Uses semantic service matching with
normalization
3. **Dimension-based**: Blake3 hash from stable alert dimensions
(fallback)
**Features:**
- Automatic alert grouping into incidents
- Real-time incident status tracking
(open/acknowledged/resolved/suppressed)
- Topology enrichment via Service Graph integration
- Cross-environment correlation prevention via dimension matching
## Incident Analysis (RCA)
**Automated Analysis:**
- Background job queries open incidents and triggers AI agent
- Periodic analysis via configurable interval
- Results displayed in incident detail view with markdown support
**On-Demand Analysis:**
- Manual trigger via API: `POST /api/v2/{org}/alerts/incidents/{id}/rca`
- Real-time results with loading indicators
- One-click "Analyze" button in UI
## SRE Chat Assistant
**Conversational AI for Alerts:**
- Generic `/api/{org}/agent/chat` endpoint with history support
- Context-aware assistance for alerts and incidents
- Sparkle icons on alert/incident rows for quick access
- Persistent chat history via IndexedDB
- Includes RCA analysis in conversation context
---
## Configuration
```bash
# Incident Management
O2_INCIDENTS_AUTO_RESOLVE_AFTER_MINUTES=-1 # -1=disabled (default), >0=auto-resolve
# RCA Agent Integration
O2_INCIDENTS_RCA_ENABLED=true
O2_INCIDENTS_RCA_AGENT_URL=http://localhost:8000
O2_INCIDENTS_RCA_INTERVAL_SECS=600
```
## API Endpoints
- `GET /api/v2/{org}/alerts/incidents` - List incidents with pagination
- `GET /api/v2/{org}/alerts/incidents/{id}` - Get incident details
- `PATCH /api/v2/{org}/alerts/incidents/{id}` - Update incident status
- `GET /api/v2/{org}/alerts/incidents/stats` - Get incident statistics
- `POST /api/v2/{org}/alerts/incidents/{id}/rca` - Trigger RCA analysis
- `POST /api/{org}/agent/chat` - Generic agent chat endpoint
- `POST /api/{org}/agent/chat/stream` - Streaming agent chat
## Database
- **New tables**: `alert_incidents`, `alert_incident_alerts`
- **Migration**: `m20251204_000001_create_alert_incidents_table`
- **ORM**: SeaORM entities with full CRUD support
## UI Components
- **IncidentList.vue**: Status filtering, pagination, statistics
dashboard
- **IncidentDetailDrawer.vue**: Full incident view with
triggers/alerts/analysis
- **SREChat.vue**: AI assistant panel with markdown rendering
- Incidents tab in AlertList with unified view
- AI sparkle icons for contextual chat access
---
## Enterprise Feature
This is an **enterprise-only** feature with OSS stubs for compilation
compatibility.
- Non-enterprise builds: Handlers return `403 Forbidden`
- Enterprise builds: Full functionality enabled
## Test Plan
- [x] Build passes without `--features enterprise`
- [x] Build passes with `--features enterprise`
- [x] Create alerts that fire and verify incidents are created
- [x] Verify incident list/detail APIs work correctly
- [x] Verify status updates (open → acknowledged → resolved)
- [x] Verify correlation logic (trace-based, service discovery,
dimension-based)
- [x] Verify RCA analysis triggers and displays correctly
- [x] Verify SRE chat integration with alert/incident context
---------
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai >
2025-12-12 22:14:18 +05:30
Hengfei Yang
sktrpathiandsai nikhil kethe
Ashish Kolheandgithub-actions[bot]
