This PR make necessary changing for using new sessions table for storing
user sessions and also removes the existing sessions from meta table
NEED TO BE MERGED POST RELEASE
- Move service graph processor from compactor to dedicated job module
- Create `src/job/service_graph.rs` with independent lifecycle
- Remove service graph code from `src/job/compactor.rs`
- Update `src/job/mod.rs` to spawn service_graph job
- Support INTERNAL spans (span_kind='1') in addition to CLIENT/SERVER
- Remove early exit when peer attributes are missing
- Use 'internal' as fallback for peer identification
- Update SQL to include `span_kind IN ('1', '2', '3')`
This enables service graph to work with OpenObserve's internal tracing
data which uses INTERNAL spans without explicit peer.service attributes.
The peer.service attribute can be stored in:
- peer_service column (flattened)
- attributes['peer.service'] map
- attributes_string['peer.service'] map
Use COALESCE to try all locations. Filter NULL results in WHERE clause.
How I run my query
1. nov 26 10:30 - 11:30 - interval 30 mins
2. nov 26 10:30 - 16:30 - interval 60 mins
3. nov 26 10:30 - 16:30 - interval 60 mins -- refresh cache
Cache dir
1. on first run
- has 2 caches 30 mins
2. on 2nd run
- will havve mixed 30mins cache and 60 mins cache
TODO:
- [x] Problem 1:
on step 3
- the interval for that duration is 60 mins
- but the discovery of cache will also include the 30 mins cache
[@mod.rs](file:///Users/lin/o2/openobserve/src/service/search/mod.rs)
`discover_cache_for_query`
- the overwrite will over write the 60 mins cache only. How to address
this problem?
- [x] Problem 2:
- fix: add rbac control only user with permussion to edit stream can
`refresh cache`
Proposed Solution
- On `refresh_cache` run streaming aggs without using cache
- at `streaming/execution.rs` , at the end of steaming aggs delete all
the cache files for the given duration when there are miss matched cache
interval files
- then remove the `streaming_id` from the `GLOBAL_CACHE`
- eg: nov 26 10:30 - 16:30 - interval 60 mins -- refresh cache
- Cache consists of 30mins or 4hr interval files all of them will be
deleted
- 60 mins cache interval files will be overwritten or created a new.
- This way on the next run there will be no query results mismatch when
the query runs again with the newly updated cache, especially delayed
data for the same duration.
- add rbac control to only allow users with edit permissions to stream
to use this feature
In super cluster, currently we broadcast the trigger update events
individually. This pr allows bulk updates through super cluster.
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
This fixes two issues for reporting triggers to own org :
- sometimes triggers of different orgs would get reporting to different
orgs, this fixes it by grouping triggers by org and then ingesting them
- for cloud, previously it would cause issues for expired orgs and block
alert manager, now this skips the expired orgs altogether. With a
previous change of disabling trigger causing stuff (alerts, reports etc)
for expired orgs completely, this change should fix the minor remaining
problem as well.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
## Summary
Implements org-level alert deduplication with semantic field groups for
intelligent alert suppression and batched notifications. This feature
prevents alert storms by deduplicating alerts based on configurable
fingerprints and grouping related alerts together.
### Core Capabilities
- **Semantic Field Groups**: Map field name variations (`hostname`,
`host`, `node`) to canonical dimensions for consistent deduplication
across different data sources
- **Organization-Level Configuration**: Global deduplication settings
with cross-alert support to suppress alerts from different rules sharing
semantic dimensions
- **Alert Grouping & Batching**: Wait-and-collect logic with three send
strategies (`FirstWithCount`, `Summary`, `All`) for batched
notifications
- **Per-Alert Control**: Individual alert deduplication settings with
optional fingerprint field customization
### Features
**Deduplication Modes:**
- Per-alert exact matching using fingerprint-based TTL expiration
- Cross-alert semantic matching using org-defined dimension groups
- Configurable time windows (per-alert override or org-level default)
**Alert Grouping:**
- Batch alerts with same fingerprint before sending notifications
- Three strategies: first alert with count, summary with names, or all
details
- Configurable wait time (1-300 seconds) and max batch size (1-100
alerts)
- Background processor runs every 1 second for expired batches
**Semantic Field Groups:**
- Built-in presets for common fields (host, IP address, service)
- Kubernetes resource groups (cluster, namespace, pod, node, container)
- Field normalization support (lowercase + trim)
- Overlap handling with first-defined precedence
### Configuration Types
- `SemanticFieldGroup`: Field name equivalence definitions with
normalization
- `GlobalDeduplicationConfig`: Org-level settings with semantic groups
and cross-alert dedup
- `DeduplicationConfig`: Per-alert fingerprint fields and grouping
options
### API Endpoints
- `GET /{org_id}/alerts/deduplication/config` - Retrieve org-level dedup
config
- `POST /{org_id}/alerts/deduplication/config` - Set org-level dedup
config
- `DELETE /{org_id}/alerts/deduplication/config` - Delete org-level
dedup config
### Implementation Details
- Pure business logic algorithms in enterprise layer (fingerprinting,
matching, calculations)
- OSS service layer orchestrates database operations with enterprise
delegation
- Feature-gated with `#[cfg(feature = "enterprise")]` for
enterprise-only functionality
- Background job architecture for async batch processing
- Storage: Org config in KV DB at `/alert_config/{org_id}/deduplication`
### Architecture
```
Alert Triggered
↓
Query Evaluation
↓
Deduplication Check (fingerprint + TTL)
↓
Grouping Enabled? → Add to Batch → Wait/Collect → Send Grouped
↓ No
Send Individual Notification
```
---------
Co-authored-by: Ashish Kolhe <ashish.j.kolhe@gmail.com>
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
# Add Row Template Type Support for Alerts
## Overview
This PR adds support for configurable row template types in alerts,
allowing users to choose between **String** and **JSON** formats for row
templates. This provides flexibility in how alert data is formatted and
sent to destinations.
## Features
### Row Template Type Selection
- Added a new `row_template_type` field to alerts with two options:
- **String**: Traditional string-based templates with variable
interpolation
- **JSON**: Structured JSON templates for programmatic consumption
### Database Layer
- Uses efficient `smallint` (i16) storage: `0` for String, `1` for JSON
- Added migration `m20251118_000001_add_alert_row_template_type` with
backward compatibility (defaults to String)
- Includes support for PostgreSQL, MySQL, and SQLite
### API Layer
- Returns enum string representation (`"String"` or `"Json"`) in API
responses
- Full support in GET, CREATE, and UPDATE alert endpoints
### UI Changes
- Added toggle button in alert creation/editing form to switch between
String and JSON template types
- Dynamic placeholder text based on selected template type
- Persists user's template type selection in the database
## Usage Example
### String Template (Default)
When creating an alert with row template type set to **String**:
```
Template Type: [String] JSON
Row Template: Alert triggered: {alert_name} at {timestamp} Affected user: {user_id} Error count: {error_count}
```
### JSON Template
When creating an alert with row template type set to **JSON**:
```
Template Type: String [JSON]
Row Template: { "alert": "{alert_name}", "timestamp": "{timestamp}", "user": { "id": "{user_id}", "name": "{user_name}" }, "metrics": { "error_count": {error_count}, "severity": "{severity}" } }
```
The JSON template type automatically parses and validates JSON
structure, making it ideal for:
- Webhook integrations that expect JSON payloads
- Structured logging systems
- API endpoints requiring specific JSON schemas
- Downstream systems that need programmatic access to alert data
## Technical Details
### Architecture
- **Database**: Stores as `smallint` (0 or 1) for efficiency
- **Service Layer**: Uses `RowTemplateType` enum throughout application
logic
- **API**: Serializes enum as strings (`"String"`, `"Json"`)
- **Frontend**: TypeScript interface uses string type with toggle UI
component
### Backward Compatibility
- Existing alerts without `row_template_type` default to `String`
- Migration adds column with `DEFAULT 0` ensuring zero downtime
- API handles both old and new alert formats seamlessly
## Testing
- Added unit tests for database migration (PostgreSQL, MySQL, SQLite)
- Tested enum serialization/deserialization
- Verified UI toggle behavior and data persistence
- Confirmed backward compatibility with existing alerts
<img width="1070" height="770" alt="Screenshot 2025-11-19 at 5 46 34 PM"
src="https://github.com/user-attachments/assets/900edb1c-815e-4240-91e2-50a1a582a9b0"
/>
1. we have created header component which will seperate out the header
part from mainlayout
2. introduced custom dark logo feature , it will fallback to light mode
logo if dark mode logo is not there
udev
Hengfei Yang
Ashish Kolheandgithub-actions[bot]
Shrinath Rao
ktx-vaidehi
Harsh Mahajan
ByteBaker
Loakesh Indiran
Subhra264andgithub-actions[bot]
Huaijin
Yashodhan JoshiandYashodhan Joshi
Sai Nikhil