mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 08:09:27 +00:00
44011fbebc
## Summary
Implements org-level alert deduplication with semantic field groups for
intelligent alert suppression and batched notifications. This feature
prevents alert storms by deduplicating alerts based on configurable
fingerprints and grouping related alerts together.
### Core Capabilities
- **Semantic Field Groups**: Map field name variations (`hostname`,
`host`, `node`) to canonical dimensions for consistent deduplication
across different data sources
- **Organization-Level Configuration**: Global deduplication settings
with cross-alert support to suppress alerts from different rules sharing
semantic dimensions
- **Alert Grouping & Batching**: Wait-and-collect logic with three send
strategies (`FirstWithCount`, `Summary`, `All`) for batched
notifications
- **Per-Alert Control**: Individual alert deduplication settings with
optional fingerprint field customization
### Features
**Deduplication Modes:**
- Per-alert exact matching using fingerprint-based TTL expiration
- Cross-alert semantic matching using org-defined dimension groups
- Configurable time windows (per-alert override or org-level default)
**Alert Grouping:**
- Batch alerts with same fingerprint before sending notifications
- Three strategies: first alert with count, summary with names, or all
details
- Configurable wait time (1-300 seconds) and max batch size (1-100
alerts)
- Background processor runs every 1 second for expired batches
**Semantic Field Groups:**
- Built-in presets for common fields (host, IP address, service)
- Kubernetes resource groups (cluster, namespace, pod, node, container)
- Field normalization support (lowercase + trim)
- Overlap handling with first-defined precedence
### Configuration Types
- `SemanticFieldGroup`: Field name equivalence definitions with
normalization
- `GlobalDeduplicationConfig`: Org-level settings with semantic groups
and cross-alert dedup
- `DeduplicationConfig`: Per-alert fingerprint fields and grouping
options
### API Endpoints
- `GET /{org_id}/alerts/deduplication/config` - Retrieve org-level dedup
config
- `POST /{org_id}/alerts/deduplication/config` - Set org-level dedup
config
- `DELETE /{org_id}/alerts/deduplication/config` - Delete org-level
dedup config
### Implementation Details
- Pure business logic algorithms in enterprise layer (fingerprinting,
matching, calculations)
- OSS service layer orchestrates database operations with enterprise
delegation
- Feature-gated with `#[cfg(feature = "enterprise")]` for
enterprise-only functionality
- Background job architecture for async batch processing
- Storage: Org config in KV DB at `/alert_config/{org_id}/deduplication`
### Architecture
```
Alert Triggered
↓
Query Evaluation
↓
Deduplication Check (fingerprint + TTL)
↓
Grouping Enabled? → Add to Batch → Wait/Collect → Send Grouped
↓ No
Send Individual Notification
```
---------
Co-authored-by: Ashish Kolhe <ashish.j.kolhe@gmail.com>
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>