feat: add organization-level alert deduplication with semantic field groups (#9209)

## Summary

Implements org-level alert deduplication with semantic field groups for
intelligent alert suppression and batched notifications. This feature
prevents alert storms by deduplicating alerts based on configurable
fingerprints and grouping related alerts together.

### Core Capabilities

- **Semantic Field Groups**: Map field name variations (`hostname`,
`host`, `node`) to canonical dimensions for consistent deduplication
across different data sources
- **Organization-Level Configuration**: Global deduplication settings
with cross-alert support to suppress alerts from different rules sharing
semantic dimensions
- **Alert Grouping & Batching**: Wait-and-collect logic with three send
strategies (`FirstWithCount`, `Summary`, `All`) for batched
notifications
- **Per-Alert Control**: Individual alert deduplication settings with
optional fingerprint field customization

### Features

**Deduplication Modes:**
- Per-alert exact matching using fingerprint-based TTL expiration
- Cross-alert semantic matching using org-defined dimension groups
- Configurable time windows (per-alert override or org-level default)

**Alert Grouping:**
- Batch alerts with same fingerprint before sending notifications
- Three strategies: first alert with count, summary with names, or all
details
- Configurable wait time (1-300 seconds) and max batch size (1-100
alerts)
- Background processor runs every 1 second for expired batches

**Semantic Field Groups:**
- Built-in presets for common fields (host, IP address, service)
- Kubernetes resource groups (cluster, namespace, pod, node, container)
- Field normalization support (lowercase + trim)
- Overlap handling with first-defined precedence

### Configuration Types

- `SemanticFieldGroup`: Field name equivalence definitions with
normalization
- `GlobalDeduplicationConfig`: Org-level settings with semantic groups
and cross-alert dedup
- `DeduplicationConfig`: Per-alert fingerprint fields and grouping
options

### API Endpoints

- `GET /{org_id}/alerts/deduplication/config` - Retrieve org-level dedup
config
- `POST /{org_id}/alerts/deduplication/config` - Set org-level dedup
config
- `DELETE /{org_id}/alerts/deduplication/config` - Delete org-level
dedup config

### Implementation Details

- Pure business logic algorithms in enterprise layer (fingerprinting,
matching, calculations)
- OSS service layer orchestrates database operations with enterprise
delegation
- Feature-gated with `#[cfg(feature = "enterprise")]` for
enterprise-only functionality
- Background job architecture for async batch processing
- Storage: Org config in KV DB at `/alert_config/{org_id}/deduplication`

### Architecture

```
Alert Triggered
     ↓
Query Evaluation
     ↓
Deduplication Check (fingerprint + TTL)
     ↓
Grouping Enabled? → Add to Batch → Wait/Collect → Send Grouped
     ↓ No
Send Individual Notification
```

---------

Co-authored-by: Ashish Kolhe <ashish.j.kolhe@gmail.com>
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
This commit is contained in:
sktrpathi
2025-11-26 12:09:10 +05:30
committed by GitHub
co-authored by Ashish Kolhe Shrinath Rao
parent b3c63ed35b
commit 44011fbebc
33 changed files with 3541 additions and 103 deletions
+2 -2
View File
@@ -327,8 +327,8 @@ jobs:
# Try to get last failed tests from TestDino
# Redirect stderr to /dev/null to avoid capturing npm warnings
if LAST_FAILED_FLAGS="$(npx tdpw last-failed --cache-id "$CACHE_ID" 2>/dev/null)"; then
# Trim whitespace (original working logic - no filtering)
LAST_FAILED_FLAGS="$(echo "$LAST_FAILED_FLAGS" | xargs)"
# Trim whitespace using parameter expansion (safer than xargs for quoted strings)
LAST_FAILED_FLAGS="$(echo "$LAST_FAILED_FLAGS" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')"
if [ -n "$LAST_FAILED_FLAGS" ] && [ "$LAST_FAILED_FLAGS" != "No failed tests found" ]; then
echo "::notice::✅ Found failed tests from previous run."