Commit Graph
1985 Commits
Author SHA1 Message Date
ByteBaker 14212b7064 fix: schema-based peer attribute detection for service graph
- Check schema for peer attributes before querying
- COALESCE priority: peer_service > server_address > db_name > db_system > http_url
- Fallback to 'internal' if no peer attributes
- Support INTERNAL spans with exclude config
- Improved UI empty state with troubleshooting
2025-12-03 16:40:17 +05:30
Sai Nikhil 060525e200 fix: console error fix for realtime alert (#9435) 2025-12-03 13:20:16 +05:30
Abhay Padamani 80a6c457b2 fix: skip local storage cache for print dashboard (#8997) 2025-12-03 12:12:31 +05:30
Sai Nikhil 26625e7b8f fix: convert severity to integer before checking for status color (#9438)
1. converting severity to integer before checking status 
2. match for exact word instead of shortforms to match the color
2025-12-03 11:04:13 +05:30
Ashish Kolhe 44c8820866 chore: remove patterns-extractions settings (#9440) 2025-12-02 17:23:05 +05:30
Sai Nikhil 6e857ea51a fix: oss open issues (#9413)
#9381 #9339 #9337
2025-12-01 18:28:29 +05:30
ByteBaker ada80858e8 feat: migrate service graph to SQL-based daemon architecture
Replace in-memory EdgeStore with daemon-based architecture that periodically
queries trace streams using SQL aggregation in DataFusion.

Architecture:
- Daemon runs in compactor job every 1 hour (default, configurable)
- Queries last 60 minutes of trace data (configurable window)
- Uses DataFusion SQL with CTE for efficient aggregation
- Calculates p50, p95, p99 percentiles using approx functions
- Writes aggregated edges to internal `_o2_service_graph` stream
- Zero impact on trace ingestion performance

Key Changes:
- Add `processor.rs` with SQL-based daemon that queries trace streams
- Add `aggregator.rs` to write SQL results to `_o2_service_graph` stream (uses json! macro)
- Refactor `api.rs` to query pre-aggregated data from stream
- Move business logic to enterprise repo (`build_topology()`, `span_to_graph_span()`)
- Remove ALL inline processing during trace ingestion
- Delete in-memory EdgeStore, worker threads, and buffering code
- Feature-gated for enterprise builds with non-enterprise stubs

SQL Query:
- CTE extracts client/server from span_kind ('2'=SERVER, '3'=CLIENT)
- Aggregates by (client, server) with COUNT, percentiles, error rate
- Filters: span_kind IN ('2','3') AND peer_service IS NOT NULL

Configuration:
- O2_SERVICE_GRAPH_ENABLED (default: false)
- O2_SERVICE_GRAPH_PROCESSING_INTERVAL_SECS (default: 3600) - Daemon run frequency
- O2_SERVICE_GRAPH_QUERY_TIME_RANGE_MINUTES (default: 60) - Query window size

Technical Details:
- Stream name: `_o2_service_graph` (internal stream)
- Edge timestamp: Uses query end time for API compatibility
- Span kinds: '2' (SERVER), '3' (CLIENT) as VARCHAR in parquet
- Percentiles: approx_median(), approx_percentile_cont(0.95), approx_percentile_cont(0.99)
2025-12-01 17:06:53 +05:30
Sai Nikhil 750ca502d0 fix: logs highlighting (#9371)
1. adds quotes to all values when it is source column
2. syntax highliting for individual section (ip adress , status )
2025-12-01 12:15:50 +05:30
Sai Nikhil d8dc6d8d1c fix: cloud sync with main (#9382)
alert details in dark mode
https://github.com/openobserve/openobserve/issues/9379
include or exclude search term in traces
https://github.com/openobserve/openobserve/issues/9378 9378
fixes query explain histogram title not showing issue
2025-12-01 11:30:24 +05:30
Ashish Kolhe e0aef9677e chore:move-service-graph-config (#9370) 2025-11-28 22:19:32 +05:30
Sai Nikhil c58714fca2 fix: license fixes sync to main (#9352)
don't load license info for cloud in about us page
show license section for meta orgs in cloud as well
show default ingestion limit when license expired
2025-11-28 19:45:09 +05:30
Ashish Kolheandgithub-actions[bot] 0b4e66c8f6 feat: sampling for search apis and patterns (#9207)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-11-28 16:13:04 +05:30
Sai Nikhil c1d37569f9 fix: UI unit test cases (#9349)
1. Header component ui unit test cases added
2. MainLayout component ui unit test cases updated
2025-11-27 19:53:57 +05:30
Huaijin 2d1cec8c9c perf: use datafusion native listingtable (#9259) 2025-11-27 16:46:43 +08:00
ktx-vaidehi 2df0134331 fix: dashboard override camcelCase issue (#9283) 2025-11-26 19:14:15 +05:30
Bhargav 90590dfa6c fix: kuebernetes helm command change (#9312) 2025-11-26 18:35:02 +05:30
Sai Nikhil 9af078907b fix: invalid absolute time text change (#9293)
1. This PR changes the text in search history , scheduled pipeline
This PR fixes the issues of
https://github.com/openobserve/openobserve/issues/9317
https://github.com/openobserve/openobserve/issues/9314
https://github.com/openobserve/openobserve/issues/9311 #9308
https://github.com/openobserve/openobserve/issues/9315 license server
fixes (ent #949) custom logo fixes #9329
2025-11-26 17:30:36 +05:30
7f55832426 ci: added workflow to udpate the language conversion (#9282)
added workflow to udpate the language conversion

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
2025-11-26 16:30:24 +05:30
ktx-vaidehiandktx-abhay bd9211afda fix: dashboard restrict save when alias as timestamp (#9294)
- #9290

---------

Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
2025-11-26 15:58:49 +05:30
Ashish Kolhe 811246bd52 feat: latency analysis (#9301)
Design at : red-drill-down-analysis

Traces Page

- Navigate to Traces section
- Run a trace query (standard or SQL mode)
- Click the "Analyze" button in the results toolbar
- Analysis modal opens with three tabs: Rate, Latency, Errors

Logs Page

1. Navigate to Logs section
2. Run a log query (standard mode only - SQL mode does not support
analysis)
3. Click the "Analyze" button in the results toolbar
4. Analysis modal opens with Volume tab (logs only support volume
analysis)

Note: The Analyze button is automatically hidden when in SQL mode on the
logs page.
2025-11-26 12:50:21 +05:30
44011fbebc feat: add organization-level alert deduplication with semantic field groups (#9209)
## Summary

Implements org-level alert deduplication with semantic field groups for
intelligent alert suppression and batched notifications. This feature
prevents alert storms by deduplicating alerts based on configurable
fingerprints and grouping related alerts together.

### Core Capabilities

- **Semantic Field Groups**: Map field name variations (`hostname`,
`host`, `node`) to canonical dimensions for consistent deduplication
across different data sources
- **Organization-Level Configuration**: Global deduplication settings
with cross-alert support to suppress alerts from different rules sharing
semantic dimensions
- **Alert Grouping & Batching**: Wait-and-collect logic with three send
strategies (`FirstWithCount`, `Summary`, `All`) for batched
notifications
- **Per-Alert Control**: Individual alert deduplication settings with
optional fingerprint field customization

### Features

**Deduplication Modes:**
- Per-alert exact matching using fingerprint-based TTL expiration
- Cross-alert semantic matching using org-defined dimension groups
- Configurable time windows (per-alert override or org-level default)

**Alert Grouping:**
- Batch alerts with same fingerprint before sending notifications
- Three strategies: first alert with count, summary with names, or all
details
- Configurable wait time (1-300 seconds) and max batch size (1-100
alerts)
- Background processor runs every 1 second for expired batches

**Semantic Field Groups:**
- Built-in presets for common fields (host, IP address, service)
- Kubernetes resource groups (cluster, namespace, pod, node, container)
- Field normalization support (lowercase + trim)
- Overlap handling with first-defined precedence

### Configuration Types

- `SemanticFieldGroup`: Field name equivalence definitions with
normalization
- `GlobalDeduplicationConfig`: Org-level settings with semantic groups
and cross-alert dedup
- `DeduplicationConfig`: Per-alert fingerprint fields and grouping
options

### API Endpoints

- `GET /{org_id}/alerts/deduplication/config` - Retrieve org-level dedup
config
- `POST /{org_id}/alerts/deduplication/config` - Set org-level dedup
config
- `DELETE /{org_id}/alerts/deduplication/config` - Delete org-level
dedup config

### Implementation Details

- Pure business logic algorithms in enterprise layer (fingerprinting,
matching, calculations)
- OSS service layer orchestrates database operations with enterprise
delegation
- Feature-gated with `#[cfg(feature = "enterprise")]` for
enterprise-only functionality
- Background job architecture for async batch processing
- Storage: Org config in KV DB at `/alert_config/{org_id}/deduplication`

### Architecture

```
Alert Triggered
     ↓
Query Evaluation
     ↓
Deduplication Check (fingerprint + TTL)
     ↓
Grouping Enabled? → Add to Batch → Wait/Collect → Send Grouped
     ↓ No
Send Individual Notification
```

---------

Co-authored-by: Ashish Kolhe <ashish.j.kolhe@gmail.com>
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
2025-11-26 12:09:10 +05:30
Bhargav 6d318cf7ea fix: added language translation (#9276) 2025-11-25 18:37:10 +05:30
Subhra264 96fa553966 fix: mysql deprecation warning banner inconsistency (#9289)
- Only Home page should show the warning
- The styles should be aligned with openobserve styling

Fixes #9286
2025-11-25 14:39:13 +05:30
de5edadb9e feat: added destination type for remote destination (#9080)
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: mmosarafO2 <mosraf@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yashodhan@openobserve.ai>
2025-11-25 14:07:37 +05:30
ByteBaker ec567b45ff fix: highlight sidebar in pipeline history screen 2025-11-25 11:39:38 +05:30
ByteBaker 9fee4b576f feat: add info tooltip to pipeline history page
Added tooltip explaining that history is only available for scheduled
and manually triggered pipelines. Real-time pipelines do not generate
history records.
2025-11-25 11:39:38 +05:30
ByteBaker e072b7a863 fix: use alert_id instead of alert_name in alerts history API call
The alerts history endpoint expects alert_id parameter (KSUID format)
not alert_name. This was causing 404 errors when fetching alert history
in the AlertList component.
2025-11-25 11:39:38 +05:30
BhargavandmmosarafO2 d56950ced1 fix: updated helm command (#9277)
Co-authored-by: mmosarafO2 <mosraf@openobserve.ai>
2025-11-25 10:06:49 +05:30
Sai Nikhil 5d59b30b5e fix: feature table (#9275) 2025-11-24 17:54:53 +05:30
Sai Nikhil 7ac0c627ba fix: ingestion quota formatting (#9273)
1. This PR formats the ingestion quota to percentage without decimals
2025-11-24 17:12:30 +05:30
Subhra264 5c465c1fc5 feat: add a warning about future deprecation of mysql support (#9229) 2025-11-24 16:28:05 +05:30
Sai Nikhil f55a76c443 fix: issues 20 release (#9261)
1. change text when user selects invalid time 
2. expiry increased to 400 days
2025-11-24 12:44:28 +05:30
Loakesh Indiran 1e7af6372a fix: add use_cache & clear_cache for _multi_search_stream (#9249) 2025-11-23 19:38:25 +05:30
Omkar Kesarkhane 19140b8e7b fix: Reset pagination on search (#9244) 2025-11-21 21:59:40 +05:30
Sai Nikhil 60e21d004f fix: issues 0.20 release (#9226)
the below issues are fixed 
#9205 
#9193 
#7468 (we now show ellipsis and on hover we will show the tooltip with
all the selected streams
#6248 (to fix this previously whenever we have set the cookie expiry
time as session that means as soon as user closes the chrome window it
will be expired now we are having 365 days expiry time)
2025-11-21 19:09:09 +05:30
Sai Nikhil db30551ffe fix: date time invalid if not enter correctly (#9235)
This PR fixes the issue in #1975
2025-11-21 18:33:37 +05:30
ktx-vaidehi 58d294f1a1 fix: dashboard when switching chart type wrong stream selection (#9233) 2025-11-21 16:42:12 +05:30
ktx-vaidehi f5462c88f9 fix: dashboard scroll for tooltip (#9224) 2025-11-21 15:20:12 +05:30
Sai Nikhil 8e8db702ed fix: change primary buttons text (#9189)
1. Now we will have Add module name we will change it to New module name
at module level and also in sidebars
2025-11-21 11:02:46 +05:30
udev a765af49bc feat: bill data retention (#9132)
- [x] Ingest to data_retention_usage fn
- [x] UI for data retention
2025-11-21 10:20:22 +05:30
ByteBaker d1ba8862ca fix: pipeline history search 2025-11-20 19:37:14 +05:30
76ffb13f31 fix: check for timestamp alias in logs page and throw an error (#9212)
Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech>
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
2025-11-20 18:53:00 +05:30
Sai Nikhil 2ad43e1648 fix: theme issues #9213 (#9214)
This PR fixes the issues in #9213
2025-11-20 17:22:38 +05:30
Subhra264 de734bd5d8 feat: Allow Json values in alert row templates (#9137)
# Add Row Template Type Support for Alerts

## Overview
This PR adds support for configurable row template types in alerts,
allowing users to choose between **String** and **JSON** formats for row
templates. This provides flexibility in how alert data is formatted and
sent to destinations.

## Features

### Row Template Type Selection
- Added a new `row_template_type` field to alerts with two options:
- **String**: Traditional string-based templates with variable
interpolation
  - **JSON**: Structured JSON templates for programmatic consumption

### Database Layer
- Uses efficient `smallint` (i16) storage: `0` for String, `1` for JSON
- Added migration `m20251118_000001_add_alert_row_template_type` with
backward compatibility (defaults to String)
- Includes support for PostgreSQL, MySQL, and SQLite

### API Layer
- Returns enum string representation (`"String"` or `"Json"`) in API
responses
- Full support in GET, CREATE, and UPDATE alert endpoints

### UI Changes
- Added toggle button in alert creation/editing form to switch between
String and JSON template types
- Dynamic placeholder text based on selected template type
- Persists user's template type selection in the database

## Usage Example

### String Template (Default)
When creating an alert with row template type set to **String**:

```
Template Type: [String] JSON
Row Template: Alert triggered: {alert_name} at {timestamp} Affected user: {user_id} Error count: {error_count}
```

### JSON Template
When creating an alert with row template type set to **JSON**:

```
Template Type: String [JSON]
Row Template: { "alert": "{alert_name}", "timestamp": "{timestamp}", "user": { "id": "{user_id}", "name": "{user_name}" }, "metrics": { "error_count": {error_count}, "severity": "{severity}" } }
```

The JSON template type automatically parses and validates JSON
structure, making it ideal for:
- Webhook integrations that expect JSON payloads
- Structured logging systems
- API endpoints requiring specific JSON schemas
- Downstream systems that need programmatic access to alert data

## Technical Details

### Architecture
- **Database**: Stores as `smallint` (0 or 1) for efficiency
- **Service Layer**: Uses `RowTemplateType` enum throughout application
logic
- **API**: Serializes enum as strings (`"String"`, `"Json"`)
- **Frontend**: TypeScript interface uses string type with toggle UI
component

### Backward Compatibility
- Existing alerts without `row_template_type` default to `String`
- Migration adds column with `DEFAULT 0` ensuring zero downtime
- API handles both old and new alert formats seamlessly

## Testing
- Added unit tests for database migration (PostgreSQL, MySQL, SQLite)
- Tested enum serialization/deserialization
- Verified UI toggle behavior and data persistence
- Confirmed backward compatibility with existing alerts

<img width="1070" height="770" alt="Screenshot 2025-11-19 at 5 46 34 PM"
src="https://github.com/user-attachments/assets/900edb1c-815e-4240-91e2-50a1a582a9b0"
/>
2025-11-20 14:44:08 +05:30
Hengfei Yang f3f5b29aa2 feat: introduce query_values_default_num configuration. (#9206)
This PR add a new ENV that allow you to change the default num of field
values.
```
ZO_QUERY_VALUES_DEFAULT_NUM=10
```
2025-11-20 16:25:12 +08:00
Omkar Kesarkhane 80cd29ef58 fix: Logs interesting field icon getting disabled (#9204) 2025-11-20 10:09:06 +05:30
Sai Nikhil 33e04588a9 fix: watch for theme changes at app level to sync with other pages (#9198)
This PR fixes 4th issue in
https://github.com/openobserve/openobserve/issues/9138
2025-11-19 20:01:25 +05:30
Harsh Mahajan 18fa55636e fix: SDR pattern import broken (#9180)
resolves https://github.com/openobserve/o2-enterprise/issues/916

**Issue**: Built-in regex pattern imports failed with 0 successful
imports because `jsonArrayOfObj` computed property referenced a null
`baseImportRef` when the BaseImport component wasn't rendered on the
built-in patterns tab.

**Solution**: Added a local ref `localJsonArrayOfObj` and updated the
computed property to use the local ref for built-in patterns tab and
BaseImport's ref for file/URL tabs, along with fixes for partial import
feedback and validation logic.
2025-11-19 18:04:57 +05:30
dependabot[bot] 042b21e35b chore(deps): bump glob from 10.4.5 to 10.5.0 in /web (#9190) 2025-11-19 19:38:33 +08:00
Omkar Kesarkhane 0d2a3e2bf8 fix: fixed traces RED metric queries (#9175) 2025-11-19 17:03:32 +05:30