Commit Graph
4719 Commits
Author SHA1 Message Date
Harsh Mahajan 32cd68e331 feat: user session migration from meta to new table (#9183)
This PR make necessary changing for using new sessions table for storing
user sessions and also removes the existing sessions from meta table

NEED TO BE MERGED POST RELEASE
2025-12-03 17:58:45 +05:30
ByteBaker 14212b7064 fix: schema-based peer attribute detection for service graph
- Check schema for peer attributes before querying
- COALESCE priority: peer_service > server_address > db_name > db_system > http_url
- Fallback to 'internal' if no peer attributes
- Support INTERNAL spans with exclude config
- Improved UI empty state with troubleshooting
2025-12-03 16:40:17 +05:30
ByteBaker 1fafad5dcd refactor: extract service graph into independent job and support INTERNAL spans
- Move service graph processor from compactor to dedicated job module
- Create `src/job/service_graph.rs` with independent lifecycle
- Remove service graph code from `src/job/compactor.rs`
- Update `src/job/mod.rs` to spawn service_graph job

- Support INTERNAL spans (span_kind='1') in addition to CLIENT/SERVER
- Remove early exit when peer attributes are missing
- Use 'internal' as fallback for peer identification
- Update SQL to include `span_kind IN ('1', '2', '3')`

This enables service graph to work with OpenObserve's internal tracing
data which uses INTERNAL spans without explicit peer.service attributes.
2025-12-03 16:40:17 +05:30
ByteBaker 9bb1576b1d fix: handle peer.service attribute in nested structures
The peer.service attribute can be stored in:
- peer_service column (flattened)
- attributes['peer.service'] map
- attributes_string['peer.service'] map

Use COALESCE to try all locations. Filter NULL results in WHERE clause.
2025-12-03 16:40:17 +05:30
udev 99c3391575 fix: audit (#9417)
This brings changes to the MMDB parser logic. Need to test via VRL
functions.
2025-12-03 15:33:04 +05:30
Sai Nikhil 060525e200 fix: console error fix for realtime alert (#9435) 2025-12-03 13:20:16 +05:30
Abhay Padamani 80a6c457b2 fix: skip local storage cache for print dashboard (#8997) 2025-12-03 12:12:31 +05:30
Sai Nikhil 26625e7b8f fix: convert severity to integer before checking for status color (#9438)
1. converting severity to integer before checking status 
2. match for exact word instead of shortforms to match the color
2025-12-03 11:04:13 +05:30
Shrinath Rao 9e3d950c7c test: reports test stabilisation for env (#9446) 2025-12-03 10:02:24 +05:30
Hengfei Yang 9c249f2c3c fix: rollback scc (#9442) 2025-12-03 11:08:23 +08:00
udev 25c3b1d454 fix: null check in conditions (#9449)
fixes https://github.com/openobserve/openobserve/issues/9443
2025-12-03 00:34:36 +05:30
9b9e99b34c test: Stabilize Dashboard Tests for Deployed Environment (#9437)
…d filtering

---------

Co-authored-by: Abhay Padamani <141122205+ktx-abhay@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2025-12-02 19:26:18 +05:30
Ashish Kolhe 44c8820866 chore: remove patterns-extractions settings (#9440) 2025-12-02 17:23:05 +05:30
Yashodhan JoshiandYashodhan Joshi e45b7708a9 ci: auto label and milestone (#9384)
See working run here
https://github.com/openobserve/openobserve/actions/runs/19761555623/job/56624451400

Also check the timeline items that mention gh actions bot added the
label and milestone to it.

This adds a CI which will automatically label every PR on approve and
add it to the lowest open milestone (ignoring backlog)
So if current open milestones are backlog, v0.20.0, v0.20.1 , it will
auto add this on approval to v0.20.0 and add `Needs-Testing` label.

---------

Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
2025-12-02 16:03:24 +05:30
Sai Nikhil 6e857ea51a fix: oss open issues (#9413)
#9381 #9339 #9337
2025-12-01 18:28:29 +05:30
ByteBaker ada80858e8 feat: migrate service graph to SQL-based daemon architecture
Replace in-memory EdgeStore with daemon-based architecture that periodically
queries trace streams using SQL aggregation in DataFusion.

Architecture:
- Daemon runs in compactor job every 1 hour (default, configurable)
- Queries last 60 minutes of trace data (configurable window)
- Uses DataFusion SQL with CTE for efficient aggregation
- Calculates p50, p95, p99 percentiles using approx functions
- Writes aggregated edges to internal `_o2_service_graph` stream
- Zero impact on trace ingestion performance

Key Changes:
- Add `processor.rs` with SQL-based daemon that queries trace streams
- Add `aggregator.rs` to write SQL results to `_o2_service_graph` stream (uses json! macro)
- Refactor `api.rs` to query pre-aggregated data from stream
- Move business logic to enterprise repo (`build_topology()`, `span_to_graph_span()`)
- Remove ALL inline processing during trace ingestion
- Delete in-memory EdgeStore, worker threads, and buffering code
- Feature-gated for enterprise builds with non-enterprise stubs

SQL Query:
- CTE extracts client/server from span_kind ('2'=SERVER, '3'=CLIENT)
- Aggregates by (client, server) with COUNT, percentiles, error rate
- Filters: span_kind IN ('2','3') AND peer_service IS NOT NULL

Configuration:
- O2_SERVICE_GRAPH_ENABLED (default: false)
- O2_SERVICE_GRAPH_PROCESSING_INTERVAL_SECS (default: 3600) - Daemon run frequency
- O2_SERVICE_GRAPH_QUERY_TIME_RANGE_MINUTES (default: 60) - Query window size

Technical Details:
- Stream name: `_o2_service_graph` (internal stream)
- Edge timestamp: Uses query end time for API compatibility
- Span kinds: '2' (SERVER), '3' (CLIENT) as VARCHAR in parquet
- Percentiles: approx_median(), approx_percentile_cont(0.95), approx_percentile_cont(0.99)
2025-12-01 17:06:53 +05:30
4ecc5509d5 test: Added Shard for Regression tests and logs spec file in it (#9394)
Co-authored-by: mmosarafO2 <mosraf@openobserve.ai>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Co-authored-by: Sai Nikhil <nikhil@openobserve.ai>
2025-12-01 14:38:03 +05:30
Sai Nikhil 750ca502d0 fix: logs highlighting (#9371)
1. adds quotes to all values when it is source column
2. syntax highliting for individual section (ip adress , status )
2025-12-01 12:15:50 +05:30
Sai Nikhil d8dc6d8d1c fix: cloud sync with main (#9382)
alert details in dark mode
https://github.com/openobserve/openobserve/issues/9379
include or exclude search term in traces
https://github.com/openobserve/openobserve/issues/9378 9378
fixes query explain histogram title not showing issue
2025-12-01 11:30:24 +05:30
Hengfei Yang 1734f63b3c fix: search on wal panic (#9402) 2025-12-01 09:06:48 +05:30
mmosarafO2 51924fa063 ci: fix for github rate limit issue (#9399)
fix for github rate limit issue
2025-11-30 12:29:34 +05:30
Loakesh Indiran 5ebcb845d7 fix: streaming aggs complete cache hit (#9397) 2025-11-29 20:36:59 +05:30
Loakesh Indiran baa7520d90 fix: overwrite aggs cache instead of delete and rewrite cache (#9390)
How I run my query 
1. nov 26 10:30 - 11:30 - interval 30 mins
2. nov 26 10:30 - 16:30 - interval 60 mins
3. nov 26 10:30 - 16:30 - interval 60 mins -- refresh cache

Cache dir
1. on first run 
- has 2 caches 30 mins 
2. on 2nd run 
- will havve mixed 30mins cache and 60 mins cache 

TODO: 
-  [x] Problem 1: 
on step 3 
- the interval for that duration is 60 mins
- but the discovery of cache will also include the 30 mins cache
[@mod.rs](file:///Users/lin/o2/openobserve/src/service/search/mod.rs)
`discover_cache_for_query`
- the overwrite will over write the 60 mins cache only. How to address
this problem?
- [x] Problem 2:
- fix: add rbac control only user with permussion to edit stream can
`refresh cache`

Proposed Solution
- On `refresh_cache` run streaming aggs without using cache
- at `streaming/execution.rs` , at the end of steaming aggs delete all
the cache files for the given duration when there are miss matched cache
interval files
- then remove the `streaming_id` from the `GLOBAL_CACHE`
- eg: nov 26 10:30 - 16:30 - interval 60 mins -- refresh cache 
- Cache consists of 30mins or 4hr interval files all of them will be
deleted
- 60 mins cache interval files will be overwritten or created a new.
- This way on the next run there will be no query results mismatch when
the query runs again with the newly updated cache, especially delayed
data for the same duration.
- add rbac control to only allow users with edit permissions to stream
to use this feature
2025-11-29 12:49:39 +05:30
Ashish Kolhe e0aef9677e chore:move-service-graph-config (#9370) 2025-11-28 22:19:32 +05:30
Sai Nikhil c58714fca2 fix: license fixes sync to main (#9352)
don't load license info for cloud in about us page
show license section for meta orgs in cloud as well
show default ingestion limit when license expired
2025-11-28 19:45:09 +05:30
Subhra264andgithub-actions[bot] 999d25c88e fix: use bulk update for scheduled job updates in super cluster sync (#9211)
In super cluster, currently we broadcast the trigger update events
individually. This pr allows bulk updates through super cluster.

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-11-28 18:30:29 +05:30
Subhra264 0d956ebdde test: Add tests to directly validate db records (#9356)
pytest to directly access db, and validate test cases using db queries.
2025-11-28 17:20:10 +05:30
Ashish Kolheandgithub-actions[bot] 0b4e66c8f6 feat: sampling for search apis and patterns (#9207)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-11-28 16:13:04 +05:30
ktx-akshay dd0206bad7 test: Dashboard tests stable deployed (#9360) 2025-11-28 15:11:57 +05:30
Huaijin dd6df5443f fix: RecordBatch create with empty value[main] (#9374) 2025-11-28 16:30:42 +08:00
Loakesh Indiran 3a690bd6fb fix: result cache & aggs cache max size needs to honor for entire dir (#9376)
fixes: https://github.com/openobserve/openobserve/issues/9344
2025-11-28 13:08:58 +05:30
Shrinath Rao d65232a2ed test: Fixed failing pipeline import tests (#9367) 2025-11-28 12:24:15 +05:30
udev e5284b82aa fix: add email headers to metrics and traces ep (#9357)
adding headers to multiple endpoints for usage reporting
2025-11-28 10:57:17 +05:30
Hengfei Yang 467a42d23c perf: optimize get_opts for object store (#9363)
Refactor `get_opts` function that improve `5x` for vortex search.
2025-11-28 08:42:29 +08:00
Hengfei Yang 2c67254f8e ci: update vrl version (#9365) 2025-11-28 07:50:41 +08:00
Hengfei Yang 0c00876f10 fix: no process-level CryptoProvider available (#9361)
fixed #9359
2025-11-27 23:53:49 +08:00
Sai Nikhil c1d37569f9 fix: UI unit test cases (#9349)
1. Header component ui unit test cases added
2. MainLayout component ui unit test cases updated
2025-11-27 19:53:57 +05:30
Hengfei Yang fa0672a663 feat: add super cluster support for metrics (#9225) 2025-11-27 20:39:09 +08:00
Yashodhan JoshiandYashodhan Joshi b75aa57d83 fix: self-org-trigger issues (#9348)
This fixes two issues for reporting triggers to own org :
- sometimes triggers of different orgs would get reporting to different
orgs, this fixes it by grouping triggers by org and then ingesting them
- for cloud, previously it would cause issues for expired orgs and block
alert manager, now this skips the expired orgs altogether. With a
previous change of disabling trigger causing stuff (alerts, reports etc)
for expired orgs completely, this change should fix the minor remaining
problem as well.

---------

Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
2025-11-27 16:12:49 +05:30
Neha P 8a158884c8 test: match_all_camelcase_tests (#9345)
- Verify full camel case token search: DbException should be found when
searching for complete camel case identifiers
- Verify atomic token search: UserAccountService should be searchable
and return matching results for full service class names
- Verify XML acronym tokenization: XMLHttpRequest should be properly
tokenized and discoverable through match_all queries
- Verify camel case with numbers: OAuth2TokenHandlershould handle mixed
alphanumeric camel case tokens correctly
- Verify multi-word service search:UserManagementService should support
complex multi-word camel case service names
2025-11-27 15:04:13 +05:30
Huaijin 2d1cec8c9c perf: use datafusion native listingtable (#9259) 2025-11-27 16:46:43 +08:00
dependabot[bot] 5543c130c5 chore(deps): bump node-forge from 1.3.1 to 1.3.2 in /tests/ui-testing (#9347) 2025-11-27 16:46:19 +08:00
ktx-akshay e4e0bb6983 test: enhance stability of dashboard creation process in UI tests (#9342) 2025-11-27 13:27:48 +05:30
Shrinath Rao c51434f937 test: process improvements (#9336) 2025-11-27 11:45:32 +05:30
Subhra264 ef8c797b4c test: use base branch as before commit in case of force push (#9332)
In case the before commit is not found in the history (in case of force
push) inside the api testing ci job, it compares with the base branch
commit.
2025-11-27 10:31:49 +05:30
ktx-vaidehi 2df0134331 fix: dashboard override camcelCase issue (#9283) 2025-11-26 19:14:15 +05:30
Bhargav 90590dfa6c fix: kuebernetes helm command change (#9312) 2025-11-26 18:35:02 +05:30
Sai Nikhil 9af078907b fix: invalid absolute time text change (#9293)
1. This PR changes the text in search history , scheduled pipeline
This PR fixes the issues of
https://github.com/openobserve/openobserve/issues/9317
https://github.com/openobserve/openobserve/issues/9314
https://github.com/openobserve/openobserve/issues/9311 #9308
https://github.com/openobserve/openobserve/issues/9315 license server
fixes (ent #949) custom logo fixes #9329
2025-11-26 17:30:36 +05:30
7f55832426 ci: added workflow to udpate the language conversion (#9282)
added workflow to udpate the language conversion

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
2025-11-26 16:30:24 +05:30
ktx-vaidehiandktx-abhay bd9211afda fix: dashboard restrict save when alias as timestamp (#9294)
- #9290

---------

Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
2025-11-26 15:58:49 +05:30