mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-05 16:10:43 +00:00
Ashish Kolhe
58c7dafa55
## Summary
Two coupled alerting features:
**1. Notification Templates v2 ("content" templates)**
- New `kind: content` template type: a structured spec (title, body
markdown, fields, links, rows, per-channel title overrides,
severity-filtered visibility) rendered natively per channel — existing
`custom` (raw body) templates are untouched and byte-identical on the
send path.
- Per-channel renderers: Slack Block Kit, Teams (Adaptive Card +
MessageCard), Discord embeds, PagerDuty Events v2, Opsgenie, ServiceNow,
SNS, HTML/text email, and a stable machine `webhook` envelope.
- Escaping is renderer-owned (mrkdwn entities, HTML encoding, markdown
stripping) with a shared URL-scheme allowlist; golden-fixture tests per
channel.
- Template editor UI: guided content form (fields/links row tables,
compact rows config, channel title overrides), live preview panel with
per-channel preview + payload tab, test-send (rate-limited,
`[TEST]`-marked).
- Migration `m20260802_000001_add_template_kind`; prebuilt default
template + org backfill.
**2. Alert chart images (stateless, storage-free)**
- Alerts can attach a metric-history chart PNG rendered from the
triggers self-reporting stream (`actual_value` per evaluation), with
warn/crit threshold lines and the firing point appended.
- Zero storage: the chart URL *is* the data — deflate+base64url payload
signed with truncated HMAC-SHA256 (org-bound, expiry embedded), rendered
on fetch by the HTTP-terminating node (`GET
/api/v2/{org}/alerts/charts/render`). Signature verified before inflate
(decompression-bomb guard), uniform 404s, per-node render semaphore +
per-minute query budget + small LRU.
- Renderer: plotters + embedded DejaVu Sans (Bitstream Vera license
included), gap-broken line segments so sparse evaluation history doesn't
draw false plateaus.
- Channel delivery: Slack image block, email inline CID (no remote
fetch), Discord multipart attachment, Teams/PagerDuty image URLs;
per-destination opt-in via template chart flag.
- Slack layout live-A/B'd on a real channel: content + buttons/chart
split across two same-colored attachments so the severity stripe spans
everything while staying under Slack's "Show more" fold; clickable link
title; "View in OpenObserve" keeps a reserved primary-button slot.
- Slack image-delivery fallback: Slack validates image URLs server-side
and rejects the ENTIRE message with `400 invalid_attachments` when its
proxy can't fetch the chart (VPN-only/private `ZO_WEB_URL`) —
live-verified. On that rejection the send strips the image blocks and
resends once (alert always delivers), sets a process-wide flag so later
sends pre-strip (reachability is deployment-wide, not per-destination),
and re-probes hourly. Zero config; observable via `slack_image_rejected`
/ `slack_image_pre_stripped` metric events.
## Testing
- 147 notification unit tests (resolve/render/preview/test-send incl.
per-channel goldens and hostile-input/injection coverage), 8 chart
payload tests, renderer PNG tests.
- Live-verified end to end: Slack (Slackbot chart fetches ~100ms,
cache-immutable semantics), SES email inline image, multi-severity
firing scenario (warning→critical) with correct threshold lines.
- FE: AddTemplate/ContentTemplateForm/preview specs; UX audit pass with
fixes.
- `cargo clippy -D warnings` clean in both OSS and enterprise
arrangements.
## Enterprise pairing
Must merge together with openobserve/o2-enterprise#2353 (workspace
twin-manifest mirror for the new OSS deps).
325 lines
15 KiB
JSON
325 lines
15 KiB
JSON
{
|
|
"revision": 1,
|
|
"destinations": [
|
|
{
|
|
"id": "slack",
|
|
"name": "Slack",
|
|
"description": "Send notifications to Slack channels",
|
|
"type": "http",
|
|
"icon": "slack-icon",
|
|
"popular": true,
|
|
"category": "messaging",
|
|
"endpoint": {
|
|
"url": "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK",
|
|
"method": "POST",
|
|
"skip_tls_verify": false,
|
|
"headers": {
|
|
"Content-Type": "application/json"
|
|
},
|
|
"output_format": "JSON",
|
|
"destination_type": "slack"
|
|
},
|
|
"template": {
|
|
"name": "prebuilt_slack",
|
|
"body": "{\n \"text\": \"🚨 Alert: {alert_name}\",\n \"blocks\": [\n {\n \"type\": \"header\",\n \"text\": {\n \"type\": \"plain_text\",\n \"text\": \"🚨 Alert: {alert_name}\"\n }\n },\n {\n \"type\": \"section\",\n \"fields\": [\n {\n \"type\": \"mrkdwn\",\n \"text\": \"*Stream:*\\n{stream_name}\"\n },\n {\n \"type\": \"mrkdwn\",\n \"text\": \"*Type:*\\n{stream_type}\"\n },\n {\n \"type\": \"mrkdwn\",\n \"text\": \"*Count:*\\n{alert_count}\"\n },\n {\n \"type\": \"mrkdwn\",\n \"text\": \"*Threshold:*\\n{alert_operator} {alert_threshold}\"\n }\n ]\n },\n {\n \"type\": \"section\",\n \"text\": {\n \"type\": \"mrkdwn\",\n \"text\": \"*Status:* 🔴 Firing\\n*Time:* {alert_trigger_time_str}\"\n }\n },\n {\n \"type\": \"actions\",\n \"elements\": [\n {\n \"type\": \"button\",\n \"text\": {\n \"type\": \"plain_text\",\n \"text\": \"View in OpenObserve\"\n },\n \"url\": \"{alert_url}\"\n }\n ]\n }\n ]\n}"
|
|
},
|
|
"credential_fields": [
|
|
{
|
|
"key": "webhookUrl",
|
|
"label": "Slack Webhook URL",
|
|
"type": "text",
|
|
"required": true,
|
|
"hint": "Get your webhook URL from Slack app configuration"
|
|
}
|
|
],
|
|
"metadata": {
|
|
"service": "slack",
|
|
"description": "Slack webhook for team notifications"
|
|
}
|
|
},
|
|
{
|
|
"id": "msteams",
|
|
"name": "Microsoft Teams",
|
|
"description": "Send notifications to Teams channels",
|
|
"type": "http",
|
|
"icon": "teams-icon",
|
|
"popular": true,
|
|
"category": "messaging",
|
|
"endpoint": {
|
|
"url": "https://your-tenant.webhook.office.com/webhookb2/YOUR-TEAMS-WEBHOOK-URL",
|
|
"method": "POST",
|
|
"skip_tls_verify": false,
|
|
"headers": {
|
|
"Content-Type": "application/json"
|
|
},
|
|
"output_format": "JSON",
|
|
"destination_type": "teams"
|
|
},
|
|
"template": {
|
|
"name": "prebuilt_msteams",
|
|
"body": "{\n \"@type\": \"MessageCard\",\n \"@context\": \"http://schema.org/extensions\",\n \"themeColor\": \"D63638\",\n \"summary\": \"Alert: {alert_name}\",\n \"sections\": [\n {\n \"activityTitle\": \"🚨 Alert: {alert_name}\",\n \"activitySubtitle\": \"OpenObserve Alert Notification\",\n \"activityImage\": \"https://openobserve.ai/favicon.ico\",\n \"facts\": [\n {\n \"name\": \"Stream\",\n \"value\": \"{stream_name}\"\n },\n {\n \"name\": \"Type\",\n \"value\": \"{stream_type}\"\n },\n {\n \"name\": \"Status\",\n \"value\": \"🔴 Firing\"\n },\n {\n \"name\": \"Count\",\n \"value\": \"{alert_count}\"\n },\n {\n \"name\": \"Threshold\",\n \"value\": \"{alert_operator} {alert_threshold}\"\n },\n {\n \"name\": \"Time\",\n \"value\": \"{alert_trigger_time_str}\"\n }\n ],\n \"markdown\": true\n }\n ],\n \"potentialActions\": [\n {\n \"@type\": \"OpenUri\",\n \"name\": \"View in OpenObserve\",\n \"targets\": [\n {\n \"os\": \"default\",\n \"uri\": \"{alert_url}\"\n }\n ]\n }\n ]\n}"
|
|
},
|
|
"credential_fields": [
|
|
{
|
|
"key": "webhookUrl",
|
|
"label": "Microsoft Teams Webhook URL",
|
|
"type": "text",
|
|
"required": true,
|
|
"hint": "Get your webhook URL from Teams channel connectors"
|
|
}
|
|
],
|
|
"metadata": {
|
|
"service": "teams",
|
|
"description": "Microsoft Teams webhook for team notifications"
|
|
}
|
|
},
|
|
{
|
|
"id": "pagerduty",
|
|
"name": "PagerDuty",
|
|
"description": "Create incidents in PagerDuty",
|
|
"type": "http",
|
|
"icon": "pagerduty-icon",
|
|
"popular": true,
|
|
"category": "incident",
|
|
"endpoint": {
|
|
"url": "https://events.pagerduty.com/v2/enqueue",
|
|
"method": "POST",
|
|
"skip_tls_verify": false,
|
|
"headers": {
|
|
"Content-Type": "application/json",
|
|
"Authorization": "Token YOUR_INTEGRATION_KEY"
|
|
},
|
|
"output_format": "JSON",
|
|
"destination_type": "pagerduty"
|
|
},
|
|
"template": {
|
|
"name": "prebuilt_pagerduty",
|
|
"body": "{\n \"payload\": {\n \"summary\": \"OpenObserve Alert: {alert_name}\",\n \"severity\": \"{severity}\",\n \"source\": \"{source}\",\n \"component\": \"{stream_name}\",\n \"group\": \"{stream_type}\",\n \"class\": \"alert\",\n \"custom_details\": {\n \"alert_name\": \"{alert_name}\",\n \"stream_name\": \"{stream_name}\",\n \"stream_type\": \"{stream_type}\",\n \"alert_count\": \"{alert_count}\",\n \"alert_threshold\": \"{alert_operator} {alert_threshold}\",\n \"alert_time\": \"{alert_trigger_time_str}\",\n \"alert_url\": \"{alert_url}\"\n }\n },\n \"routing_key\": \"{routing_key}\",\n \"event_action\": \"trigger\",\n \"links\": [\n {\n \"href\": \"{alert_url}\",\n \"text\": \"View in OpenObserve\"\n }\n ]\n}"
|
|
},
|
|
"credential_fields": [
|
|
{
|
|
"key": "integrationKey",
|
|
"label": "Integration Key",
|
|
"type": "text",
|
|
"required": true,
|
|
"hint": "Your PagerDuty Events API v2 integration key"
|
|
}
|
|
],
|
|
"metadata": {
|
|
"service": "pagerduty",
|
|
"description": "PagerDuty incident management"
|
|
}
|
|
},
|
|
{
|
|
"id": "discord",
|
|
"name": "Discord",
|
|
"description": "Send notifications to Discord channels",
|
|
"type": "http",
|
|
"icon": "discord-icon",
|
|
"popular": true,
|
|
"category": "messaging",
|
|
"endpoint": {
|
|
"url": "https://discord.com/api/webhooks/YOUR/DISCORD/WEBHOOK",
|
|
"method": "POST",
|
|
"skip_tls_verify": false,
|
|
"headers": {
|
|
"Content-Type": "application/json"
|
|
},
|
|
"output_format": "JSON",
|
|
"destination_type": "discord"
|
|
},
|
|
"template": {
|
|
"name": "prebuilt_discord",
|
|
"body": "{\n \"content\": \"🚨 Alert: {alert_name}\",\n \"embeds\": [\n {\n \"title\": \"Alert Details\",\n \"color\": 15158332,\n \"fields\": [\n {\n \"name\": \"Stream\",\n \"value\": \"{stream_name}\",\n \"inline\": true\n },\n {\n \"name\": \"Type\",\n \"value\": \"{stream_type}\",\n \"inline\": true\n },\n {\n \"name\": \"Count\",\n \"value\": \"{alert_count}\",\n \"inline\": true\n },\n {\n \"name\": \"Threshold\",\n \"value\": \"{alert_operator} {alert_threshold}\",\n \"inline\": true\n }\n ],\n \"footer\": {\n \"text\": \"OpenObserve Alert\",\n \"icon_url\": \"https://openobserve.ai/favicon.ico\"\n },\n \"timestamp\": \"{alert_trigger_time_str}\"\n }\n ]\n}"
|
|
},
|
|
"credential_fields": [
|
|
{
|
|
"key": "webhookUrl",
|
|
"label": "Discord Webhook URL",
|
|
"type": "text",
|
|
"required": true,
|
|
"hint": "Get your webhook URL from Discord server settings"
|
|
}
|
|
],
|
|
"metadata": {
|
|
"service": "discord",
|
|
"description": "Discord webhook for community notifications"
|
|
}
|
|
},
|
|
{
|
|
"id": "webhook",
|
|
"name": "Generic Webhook",
|
|
"description": "Generic webhook for any HTTP endpoint",
|
|
"type": "http",
|
|
"icon": "webhook-icon",
|
|
"popular": false,
|
|
"category": "generic",
|
|
"endpoint": {
|
|
"url": "https://your-service.com/webhook",
|
|
"method": "POST",
|
|
"skip_tls_verify": false,
|
|
"headers": {
|
|
"Content-Type": "application/json"
|
|
},
|
|
"output_format": "JSON",
|
|
"destination_type": "webhook"
|
|
},
|
|
"template": {
|
|
"name": "prebuilt_webhook",
|
|
"body": "{\n \"alert\": {\n \"name\": \"{alert_name}\",\n \"stream\": {\n \"name\": \"{stream_name}\",\n \"type\": \"{stream_type}\"\n },\n \"status\": \"firing\",\n \"count\": \"{alert_count}\",\n \"threshold\": {\n \"operator\": \"{alert_operator}\",\n \"value\": \"{alert_threshold}\"\n },\n \"time\": \"{alert_trigger_time_str}\",\n \"url\": \"{alert_url}\"\n }\n}"
|
|
},
|
|
"credential_fields": [
|
|
{
|
|
"key": "webhookUrl",
|
|
"label": "Webhook URL",
|
|
"type": "text",
|
|
"required": true,
|
|
"hint": "Your webhook endpoint URL"
|
|
}
|
|
],
|
|
"metadata": {
|
|
"service": "webhook",
|
|
"description": "Generic webhook destination"
|
|
}
|
|
},
|
|
{
|
|
"id": "opsgenie",
|
|
"name": "Opsgenie",
|
|
"description": "Create alerts in Opsgenie",
|
|
"type": "http",
|
|
"icon": "opsgenie-icon",
|
|
"popular": true,
|
|
"category": "incident",
|
|
"endpoint": {
|
|
"url": "https://api.opsgenie.com/v2/alerts",
|
|
"method": "POST",
|
|
"skip_tls_verify": false,
|
|
"headers": {
|
|
"Content-Type": "application/json",
|
|
"Authorization": "GenieKey YOUR_API_KEY"
|
|
},
|
|
"output_format": "JSON",
|
|
"destination_type": "opsgenie"
|
|
},
|
|
"template": {
|
|
"name": "prebuilt_opsgenie",
|
|
"body": "{\n \"message\": \"OpenObserve Alert: {alert_name}\",\n \"alias\": \"{alert_name}_{stream_name}\",\n \"description\": \"Alert triggered from OpenObserve\",\n \"priority\": \"{credential_priority}\",\n \"details\": {\n \"alert_name\": \"{alert_name}\",\n \"stream_name\": \"{stream_name}\",\n \"stream_type\": \"{stream_type}\",\n \"alert_count\": \"{alert_count}\",\n \"alert_threshold\": \"{alert_operator} {alert_threshold}\",\n \"alert_time\": \"{alert_trigger_time_str}\",\n \"alert_url\": \"{alert_url}\"\n },\n \"source\": \"OpenObserve\",\n \"tags\": [\"openobserve\", \"{stream_type}\"]\n}"
|
|
},
|
|
"credential_fields": [
|
|
{
|
|
"key": "apiKey",
|
|
"label": "Opsgenie API Key",
|
|
"type": "password",
|
|
"required": true,
|
|
"hint": "Get your API key from Opsgenie integration settings"
|
|
},
|
|
{
|
|
"key": "euRegion",
|
|
"label": "EU Region",
|
|
"type": "toggle",
|
|
"required": false,
|
|
"hint": "Enable for EU-based Opsgenie instances"
|
|
},
|
|
{
|
|
"key": "priority",
|
|
"label": "Default Priority",
|
|
"type": "select",
|
|
"required": false,
|
|
"hint": "Select the default priority for Opsgenie alerts"
|
|
}
|
|
],
|
|
"metadata": {
|
|
"service": "opsgenie",
|
|
"description": "Opsgenie incident management and alerting"
|
|
}
|
|
},
|
|
{
|
|
"id": "servicenow",
|
|
"name": "ServiceNow",
|
|
"description": "Create incidents in ServiceNow",
|
|
"type": "http",
|
|
"icon": "servicenow-icon",
|
|
"popular": true,
|
|
"category": "incident",
|
|
"endpoint": {
|
|
"url": "https://YOUR_INSTANCE.service-now.com/api/now/table/incident",
|
|
"method": "POST",
|
|
"skip_tls_verify": false,
|
|
"headers": {
|
|
"Content-Type": "application/json",
|
|
"Authorization": "Basic YOUR_AUTH_TOKEN"
|
|
},
|
|
"output_format": "JSON",
|
|
"destination_type": "servicenow"
|
|
},
|
|
"template": {
|
|
"name": "prebuilt_servicenow",
|
|
"body": "{\n \"short_description\": \"OpenObserve Alert: {alert_name}\",\n \"description\": \"Alert triggered from OpenObserve\\n\\nStream: {stream_name}\\nType: {stream_type}\\nCount: {alert_count}\\nThreshold: {alert_operator} {alert_threshold}\\nTime: {alert_trigger_time_str}\\nURL: {alert_url}\",\n \"urgency\": \"2\",\n \"impact\": \"2\",\n \"category\": \"monitoring\",\n \"subcategory\": \"alert\",\n \"assignment_group\": \"{credential_assignmentGroup}\",\n \"caller_id\": \"openobserve\"\n}"
|
|
},
|
|
"credential_fields": [
|
|
{
|
|
"key": "instanceUrl",
|
|
"label": "ServiceNow Instance URL",
|
|
"type": "text",
|
|
"required": true,
|
|
"hint": "https://your-instance.service-now.com/api/now/table/incident"
|
|
},
|
|
{
|
|
"key": "username",
|
|
"label": "Username",
|
|
"type": "text",
|
|
"required": true,
|
|
"hint": "ServiceNow username with incident creation permissions"
|
|
},
|
|
{
|
|
"key": "password",
|
|
"label": "Password",
|
|
"type": "password",
|
|
"required": true,
|
|
"hint": "ServiceNow password or API token"
|
|
},
|
|
{
|
|
"key": "assignmentGroup",
|
|
"label": "Assignment Group (optional)",
|
|
"type": "text",
|
|
"required": false,
|
|
"hint": "Group to assign incidents to (e.g., IT Operations)"
|
|
}
|
|
],
|
|
"metadata": {
|
|
"service": "servicenow",
|
|
"description": "ServiceNow incident management"
|
|
}
|
|
},
|
|
{
|
|
"id": "email",
|
|
"name": "Email",
|
|
"description": "Send HTML formatted email notifications",
|
|
"type": "email",
|
|
"icon": "email-icon",
|
|
"popular": true,
|
|
"category": "email",
|
|
"endpoint": null,
|
|
"template": {
|
|
"name": "prebuilt_email",
|
|
"title": "OpenObserve Alert: {alert_name}",
|
|
"body": "<h2>🚨 Alert Notification: {alert_name}</h2><p><strong>Stream:</strong> {stream_name}<br><strong>Type:</strong> {stream_type}<br><strong>Status:</strong> 🔴 Firing<br><strong>Count:</strong> {alert_count}<br><strong>Threshold:</strong> {alert_operator} {alert_threshold}<br><strong>Time:</strong> {alert_trigger_time_str}</p><p><a href='{alert_url}'>View in OpenObserve</a></p>"
|
|
},
|
|
"credential_fields": [
|
|
{
|
|
"key": "recipients",
|
|
"label": "Recipient Email Addresses",
|
|
"type": "text",
|
|
"required": true,
|
|
"hint": "Comma-separated email addresses"
|
|
}
|
|
],
|
|
"metadata": {
|
|
"service": "email",
|
|
"description": "Email notification",
|
|
"recipients": ["admin@your-domain.com"]
|
|
}
|
|
}
|
|
]
|
|
}
|