41f330af67 feat: re_patterns at ingestion (#7323)
### **User description**

This PR adds support for adding and applying regexp patterns at
ingestion time to redact or drop the values from fields if matched. This
is an ent-only feature, and this PR contains the required table
migrations, along with nats code for super cluster.

This also has some additions for musl build for ent repo.

There are basic crud APIs for the pattersn as well as an test api which
can be used to test a specific pattern on given input.

___

### **PR Type**
Enhancement


___

### **Description**
- Add HTTP CRUD and test endpoints for regex patterns

- Implement DB models, migrations for patterns associations

- Integrate pattern manager into ingestion and streams

- Spawn watchers and super-cluster queue for patterns


___

### **Changes diagram**

```mermaid
flowchart LR
  HTTP["CRUD & test HTTP handlers"]
  DBService["Service DB layer"]
  DB["re_patterns & associations tables"]
  Cluster["Cluster Coordinator"]
  PatternMgr["Pattern Manager"]
  Ingest["Ingestion pipeline"]
  Stream["Stream settings update"]
  HTTP -- "invoke add/update/delete/test" --> DBService
  DBService -- "persist & notify" --> DB
  DBService -- "trigger" --> Cluster
  Cluster -- "watch events" --> PatternMgr
  Ingest -- "redact at ingestion" --> PatternMgr
  Stream -- "update associations" --> PatternMgr
```


___



### **Changes walkthrough** 📝
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>14
files</summary><table>
<tr>
<td><strong>mod.rs</strong><dd><code>Add HTTP handlers for re_patterns
endpoints</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-27d37a8738de89c81b540daa978e684caf2cf28803e1208d90cb4d42528ed28b">+459/-0</a>&nbsp;
</td>

</tr>

<tr>
<td><strong>re_pattern.rs</strong><dd><code>Implement DB service for
re_patterns</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-0cb2bf6d550d07b66db7e325c6d44c6a598f2ac8e315bf9e168f4a1d9d6e745d">+384/-0</a>&nbsp;
</td>

</tr>

<tr>
<td><strong>re_pattern_stream_map.rs</strong><dd><code>Add pattern
association table ORM</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-13798240832b9110426267b3e0ebb862bd6c2ac03a3462721f5e85b5bfa88f51">+257/-0</a>&nbsp;
</td>

</tr>

<tr>
<td><strong>re_pattern.rs</strong><dd><code>Add re_patterns table ORM
definitions</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-f113ed92667f5e352ac7a0c1c8b5e48ea858992b92be5817c9b38f65843ef4a2">+185/-0</a>&nbsp;
</td>

</tr>

<tr>
<td><strong>stream.rs</strong><dd><code>Integrate pattern associations
in streams</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-41452a7fb29b4634437174fd6aecf6a9050dbfdef41b9077a21d59af3d79e9ed">+62/-2</a>&nbsp;
&nbsp; </td>

</tr>

<tr>
<td><strong>re_pattern.rs</strong><dd><code>Handle super-cluster queue
for patterns</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-9e64341dc2d7d50cc5acffd0340b983f37b46a9aa34c86fb5a25978f9eb761dc">+166/-0</a>&nbsp;
</td>

</tr>

<tr>
<td><strong>ingest.rs</strong><dd><code>Redact data via pattern manager
at ingestion</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-e62118e45be183266d39ffa08491f255017954235ff0f3d6937683e314da406f">+23/-0</a>&nbsp;
&nbsp; </td>

</tr>

<tr>
<td><strong>stream.rs</strong><dd><code>Add pattern associations to
Stream metadata</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-6f5ee19bbcc9cc7fb699fc1c0f43fc96d42698ae1168a2827b5c65f7948d5dbc">+4/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
<td><strong>stream.rs</strong><dd><code>Define PatternAssociation and
settings wrapper</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-2438d04c3fe97bc5452e5b74a303163b8723d93455cda32c95f0525435ad46bd">+14/-1</a>&nbsp;
&nbsp; </td>

</tr>

<tr>
<td><strong>mod.rs</strong><dd><code>Spawn pattern manager watchers on
init</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+13/-3</a>&nbsp;
&nbsp; </td>

</tr>

<tr>
<td><strong>re_pattern_stream_map.rs</strong><dd><code>Define ORM entity
for stream pattern map</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-3c32b9e32bea40b63ed43542aa136cc9000132c4049ed7b6310baabadc55cd79">+38/-0</a>&nbsp;
&nbsp; </td>

</tr>

<tr>
<td><strong>re_patterns.rs</strong><dd><code>Define ORM entity for
re_patterns table</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-9c4ac843741290c1b999f3101fae6f46d71d1893d7740de50c71e2c29fe1636c">+33/-0</a>&nbsp;
&nbsp; </td>

</tr>

<tr>
<td><strong>mod.rs</strong><dd><code>Register re_pattern processor in
super queue</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-6ed0f1dd36b7216fffde573232a49869f247330ecb03b50f94b2ddd723742e46">+2/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
<td><strong>prelude.rs</strong><dd><code>Expose re_patterns entities in
prelude</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-a818656b9fb01945155f28450270dd2ae20b67c8248c851ffa0f25e859d263ed">+2/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>
</table></details></td></tr><tr><td><strong>Configuration
changes</strong></td><td><details><summary>5 files</summary><table>
<tr>

<td><strong>m20250612_000001_create_re_pattern_table.rs</strong><dd><code>Create
migration for re_patterns table</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-db969ee5510136d5b3c712cdea999c151ab04efe55f1b090670c1eed2b36bcfc">+173/-0</a>&nbsp;
</td>

</tr>

<tr>

<td><strong>m20250612_000002_create_re_pattern_stream_map_table.rs</strong><dd><code>Create
migration for association stream map table</code>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-f1ec25aa2150a4fc857fdb17e45b8e43e7c31e884df77e8c91fd07636a6931ca">+168/-0</a>&nbsp;
</td>

</tr>

<tr>
<td><strong>mod.rs</strong><dd><code>Register new pattern
migrations</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-7ee42a714d3d56c663da0412d15514613222055aebb5b50dafbc051a656a36f6">+5/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
<td><strong>mod.rs</strong><dd><code>Add re_patterns services to HTTP
router</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+7/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
<td><strong>config.rs</strong><dd><code>Bump DB schema version to
support patterns</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+1/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>
</table></details></td></tr><tr><td><strong>Error
handling</strong></td><td><details><summary>1 files</summary><table>
<tr>
<td><strong>auth.rs</strong><dd><code>Add auth rules for re_patterns
test endpoint</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+11/-0</a>&nbsp;
&nbsp; </td>

</tr>
</table></details></td></tr><tr><td><strong>Additional
files</strong></td><td><details><summary>23 files</summary><table>
<tr>
  <td><strong>Cross.toml</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-3abca7e5bab5b140e44643dade038033b8a6f6e50771f76c7d48e7bd0b338815">+5/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>Dockerfile.musl</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-f47eb18f838cbf757cd4e34ed5357c767b969d4551dcea6a4ab3b64d384a5614">+16/-0</a>&nbsp;
&nbsp; </td>

</tr>

<tr>
  <td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-a1cfd40da1ab6cd4db0181fd8664965034298047b3b7c3320c5385897a7c4aed">+2/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-c539d7557d1425986d5c09849d8dbca42d1b309c2bc33d49db641d6a783098f6">+2/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-af023301203a27a1ef34f698d5a329ad7bfced65ac65e3829b1c0905a0c74fb2">+2/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-b7e3bf7f0ac23dd084c0c817a310a4d5e7a64ff532746d182986ef940fa03fd0">+2/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>O2AIChat.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-a6a125719a7b32b31b53fcfe05c12081d82c139a401699affdda0db8e73887e3">+2/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>TemplateList.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-7df66364d7d69a26cfa25cbc555d0e090c2f73e51c8386d9b0435bae4412564c">+1/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>FullViewContainer.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-c0722446274c88c250042e38b051201b81450cb5da1ee827cd5a4a927424a357">+1/-6</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>AssociatedRegexPatterns.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-c6fce5e1c83798991b6ff19effb23c67af59f38fac0b9e56dcab7a09d139f550">+940/-0</a>&nbsp;
</td>

</tr>

<tr>
  <td><strong>schema.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-9196be2ccae4054a00b0b1aa9d1e87b366c10920d6eaf9ef8c6e6293d50bcf46">+134/-4</a>&nbsp;
</td>

</tr>

<tr>
  <td><strong>AddRegexPattern.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-a3783cfbdd32609bfff45ecfa23542854cad7ae60c3d29fedb832312a4de03dc">+689/-0</a>&nbsp;
</td>

</tr>

<tr>
  <td><strong>ImportRegexPattern.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-e8e1bcb5a02311504d7d571090eab27a270aaa71cae185ad2703023e2e70a31e">+771/-0</a>&nbsp;
</td>

</tr>

<tr>
  <td><strong>NoRegexPatterns.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-279b923f20042948c87e907d2453e9d5a4c8c73672b23df951d7d7d823ff3a88">+103/-0</a>&nbsp;
</td>

</tr>

<tr>
  <td><strong>RegexPatternList.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-524927acd9f5bb07b71bfac489e747470108574e21a6dde63ea7c1f219cea853">+506/-0</a>&nbsp;
</td>

</tr>

<tr>
  <td><strong>index.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-699c63979f851c48ad440a078e93282fd797a18fc065e5331b2e41122ddb8b69">+27/-2</a>&nbsp;
&nbsp; </td>

</tr>

<tr>
  <td><strong>useManagementRoutes.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-b71a1d39b1fab0749f0668d5f16382536fbda6e2f0c92c56a972fb5d8b5835f5">+12/-0</a>&nbsp;
&nbsp; </td>

</tr>

<tr>
  <td><strong>useStreams.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-a083369344f022cdf8d4f536719179b1d920369e33e2e26234e70dd36d151a7c">+42/-1</a>&nbsp;
&nbsp; </td>

</tr>

<tr>
  <td><strong>MainLayout.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-91e4134141264c4624d4eb4cfa987b33ed31eb6f551e089c0b26f2610dd54362">+3/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>en.json</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+24/-1</a>&nbsp;
&nbsp; </td>

</tr>

<tr>
  <td><strong>JsonPreview.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-3985073134d35ff074742944b114d0905524038b42e6d2ac887fef2c8436fb16">+272/-3</a>&nbsp;
</td>

</tr>

<tr>
  <td><strong>regex_pattern.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-2a0b0828141d39ab0385e0941314ead1cdd2ec3ec7b3baf5fd0c42f8f010728a">+63/-0</a>&nbsp;
&nbsp; </td>

</tr>

<tr>
  <td><strong>index.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7323/files#diff-955a15c7ee060f1545e82be22911dd0c14307dee5a70071a394e9fd583bd2c55">+9/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>
</table></details></td></tr></tr></tbody></table>

___

> <details> <summary> Need help?</summary><li>Type <code>/help how to
...</code> in the comments thread for any questions about PR-Agent
usage.</li><li>Check out the <a
href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a>
for more information.</li></details>

---------

Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
2025-07-23 14:55:03 +05:30
2024-09-25 11:09:26 +08:00
2025-07-21 18:24:16 +05:30
2025-07-10 11:23:41 -07:00
2024-09-22 17:57:47 +08:00
2025-07-02 21:31:31 +08:00
2024-06-10 13:10:01 -07:00
2023-11-26 11:04:37 -08:00
2024-09-22 17:57:47 +08:00
2025-04-06 13:37:17 -07:00

OpenObserve

🚀 10x easier, 🚀 140x lower storage cost, 🚀 high performance, 🚀 petabyte scale - Elasticsearch/Splunk/Datadog alternative for 🚀 (logs, metrics, traces).

Last Commit GitHub Stars GitHub Issues Contributors GitHub Release

OpenObserve (O2 for short) is a cloud-native observability platform built specifically for logs, metrics, traces, analytics, RUM (Real User Monitoring - Performance, Errors, Session Replay) designed to work at petabyte scale.

It is straightforward and easy to operate, in contrast to Elasticsearch, which requires understanding and tuning numerous settings. Get OpenObserve up and running in under 2 minutes.

OpenObserve serves as a seamless replacement for Elasticsearch for users who ingest data using APIs and perform searches. OpenObserve comes with its own user interface, eliminating the need for separate installation.

You can reduce your log storage costs by ~140x compared to Elasticsearch by using OpenObserve. Below, we present the results from pushing logs from our production Kubernetes cluster to both Elasticsearch and OpenObserve using Fluent Bit.

OpenObserve Vs Elasticsearch

🎥 Introduction Video

OpenObserve Introduction

🌟 Features:

  • Logs, Metrics, Traces: Comprehensive support for various data types.
  • OpenTelemetry Support: Full compatibility with OTLP for logs, metrics, and traces.
  • Real User Monitoring (RUM): Includes performance tracking, error logging, and session replay.
  • Dashboards, Reports, Alerts: Features over 18 different chart types for comprehensive data visualization for on-the-fly analysis and reporting along with alerting.
  • Pipelines: Enrich, redact, reduce, normalize data on the fly. Stream processing for logs to metrics and more.
  • Advanced Embedded GUI: Intuitive and user-friendly interface.
  • SQL and PromQL Support: Query logs and traces with SQL, and metrics with SQL and PromQL.
  • Single Binary or HA Installation: Install using a single binary for small deployments or in HA mode for large deployments.
  • Versatile Storage Options: Supports local disk, S3, MinIO, GCS, Azure Blob Storage.
  • High Availability and Clustering: Ensures reliable and scalable performance.
  • Dynamic Schema: Adapts to your data structure seamlessly.
  • Built-in Authentication: Secure and ready to use.
  • Ease of Operation: Designed for simplicity and efficiency.
  • Seamless Upgrades: Hassle-free updates.
  • Multilingual UI: Supports 11 languages, including English, Spanish, German, French, Chinese, and more.

For a full list of features, check the documentation.

⚡️ Quick start

🐳 Docker:

docker run -d \
      --name openobserve \
      -v $PWD/data:/data \
      -p 5080:5080 \
      -e ZO_ROOT_USER_EMAIL="root@example.com" \
      -e ZO_ROOT_USER_PASSWORD="Complexpass#123" \
      public.ecr.aws/zinclabs/openobserve:latest

🐙 Docker Compose:

services:
  openobserve:
    image: public.ecr.aws/zinclabs/openobserve:latest
    restart: unless-stopped
    environment:
      ZO_ROOT_USER_EMAIL: "root@example.com"
      ZO_ROOT_USER_PASSWORD: "Complexpass#123"
    ports:
      - "5080:5080"
    volumes:
      - data:/data
volumes:
  data:

For other ways to quickly install OpenObserve or use OpenObserve cloud, check quickstart documentation.

For installing OpenObserve in HA mode, check HA deployment documentation.

📷 Screenshots

Home

Home

Logs

Logs

Traces (OpenTelemetry)

Trace details page Traces using OpenTelemetry

Golden metrics based on traces Traces golden metrics

Visualizations and Dashboards

Dashboard Dashboard Create panel Map

Front end monitoring

Performance analytics Performance

Session replay Session replay

Error tracking Error tracking

Alerts

Alerts

Streams

Streams

Ingestion

Ingestion

Pipeline

Pipeline Pipeline

Function Function

IAM

SSO (Single Sign On) SSO

RBAC (Role Based Access Control) RBAC

SBOM

Software Bill of Materials for OpenObserve

Rust

SBOM can be found here. You can analyze it using dependency track.

In order to generate the SBOM, you can use the following commands:

Install cargo-cyclonedx:

cargo install cargo-cyclonedx

Generate the SBOM:

cargo-cyclonedx cyclonedx

JavaScript

SBOM can be found here. You can analyze it using dependency track.

In order to generate the SBOM, you can use the following commands:

Install cyclonedx-npm:

npm install --global @cyclonedx/cyclonedx-npm

Generate the SBOM:

cd web
cyclonedx-npm > sbom.json         

⚖️ License

OpenObserve is licensed under the AGPL-3.0 license. For more details, see the LICENSE.

🌍 Community

🔗 Join OpenObserve community on Slack

Slack

Easiest way to get support is to join the Slack channel.

📱 Join OpenObserve community on WeChat

S
Description
Open source observability platform for logs, metrics, traces, RUM (web, android, ios), Session replay, pipelines, SLO and LLM observability. A sophisticated, simple and highly performant alternative to Datadog, Splunk, and Elasticsearch with 140x lower storage costs and single binary deployment.
Readme AGPL-3.0
1 GiB
Languages
TypeScript 39.9%
Rust 29.5%
Vue 16.9%
JavaScript 11%
Python 2.3%
Other 0.3%