Patrick Sullivan ebf527b9fc fix: Non-random KSUIDs from hashes in folder, dashboard, and alert migrations (#5726)
### Summary

This PR updates the logic that is used to generate KSUID primary keys
for the `folders`, `dashboards`, and `alerts` tables in existing
migrations which transform data from the `meta` table into the
`folders`, `dashboards`, and `alerts` tables.

The new logic generates 160-bit "KSUIDs" which are actually 160-bit
SHA-1 hashes of the records being transformed rather than being truly
random KSUIDs.

### Motivation

In the existing implementation, the migrations generate random KSUIDs
for the records that are transformed from the `meta` table and into the
different `folders`, `dashboards`, and `alerts` tables. When running the
application in a single cluster this is acceptable behavior.

However when running the application in a multi-cluster environment this
causes problems. In a multi-cluster environment, each cluster will have
it's own copy of the application database, and each database should
contain the exact same data in the `meta` table. When we run the
migration scripts in each cluster, the data from the `meta` table will
be transformed into the `folders`, `dashboards`, and `alerts` tables.
However if the migration generates a random KSUID for each new record
inserted into those new tables, then different clusters will each
generate different KSUID `id`s for rows in the `folders`, `dashboards`,
and `alerts` tables which should actually have the same `id`s.

Our solution to this problem is that during the migration from the
`meta` table to the `folders`, `dashboards`, and `alerts` tables, rather
than generating random 160-bit KSUID `id`s for the new records, we
generate 160-bit SHA-1 hashes for each record and we use that SHA-1 hash
as the `id` of the each new record. This ensure that each record that is
transformed from the `meta` table will have an `id` in the `folders`,
`alerts`, or `dashboards` tables that is consistent across all clusters.

Newly generated records (ie, records generated AFTER the migration) in
the `folders`, `alerts`, and `dashboards` tables will continue to be
generated with random KSUID `id`s since these `id`s can be shared with
other clusters via NATS super-cluster events and then records with those
`id`s can be inserted into databases in other clusters, ensuring that
records have consistent `id`s across clusters.

### Notes about KSUID generation

To generate a KSUID this function generates the 160-bit SHA-1 hash of
the alert's `org`, `stream_type`, `stream_name`, and `name` and
interprets that 160-bit hash as a 160-bit KSUID. Therefore two KSUIDs
generated in this manner will always be equal if the alerts have the
same `org`, `stream_type`, `stream_name`, and `name`.

⚠️ It is important to note that although KSUIDs generated in this manner
can be parsed as KSUIDs since they are 160-bit values, the KSUIDs
generated in this manner will have timestamp bits which are effectively
random, meaning that the timestamp in any KSUID generated with this
function will be random. This will render the timestamp-sortability
property of these KSUIDs useless. This probably isn't a big deal since
we can always add and sort by a `created_at` column if we want to, but
we should be aware of this limitation.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added SHA-1 hashing dependency to generate unique identifiers (KSUIDs)
for alerts, folders, and dashboards.
	- Implemented KSUID generation based on specific entity attributes.

- **Refactor**
- Transitioned database tables from integer-based to KSUID-based primary
keys.
	- Updated `StreamType` enum to support more flexible usage.

- **Chores**
	- Updated project dependencies to include SHA-1 hashing library.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-01-17 09:56:47 +05:30
2025-01-15 08:59:53 -08:00
2025-01-07 12:09:48 +08:00
2024-11-11 09:26:20 +08:00
2025-01-14 20:46:10 +05:30
2024-09-25 11:09:26 +08:00
2024-09-22 17:57:47 +08:00
2024-06-10 13:10:01 -07:00
2023-11-26 11:04:37 -08:00
2024-09-22 17:57:47 +08:00
2024-11-11 09:26:20 +08:00
2024-11-25 19:20:07 +08:00

OpenObserve

🚀 10x easier, 🚀 140x lower storage cost, 🚀 high performance, 🚀 petabyte scale - Elasticsearch/Splunk/Datadog alternative for 🚀 (logs, metrics, traces).

Last Commit GitHub Stars GitHub Issues Contributors GitHub Release

OpenObserve (O2 for short) is a cloud-native observability platform built specifically for logs, metrics, traces, analytics, RUM (Real User Monitoring - Performance, Errors, Session Replay) designed to work at petabyte scale.

It is straightforward and easy to operate, in contrast to Elasticsearch, which requires understanding and tuning numerous settings. Get OpenObserve up and running in under 2 minutes.

OpenObserve serves as a seamless replacement for Elasticsearch for users who ingest data using APIs and perform searches. OpenObserve comes with its own user interface, eliminating the need for separate installation.

You can reduce your log storage costs by ~140x compared to Elasticsearch by using OpenObserve. Below, we present the results from pushing logs from our production Kubernetes cluster to both Elasticsearch and OpenObserve using Fluent Bit.

OpenObserve Vs Elasticsearch

🎥 Introduction Video

OpenObserve Introduction

🌟 Features:

  • Logs, Metrics, Traces: Comprehensive support for various data types.
  • OpenTelemetry Support: Full compatibility with OTLP for logs, metrics, and traces.
  • Real User Monitoring (RUM): Includes performance tracking, error logging, and session replay.
  • Dashboards, Reports, Alerts: Features over 18 different chart types for comprehensive data visualization for on-the-fly analysis and reporting along with alerting.
  • Pipelines: Enrich, redact, reduce, normalize data on the fly. Stream processing for logs to metrics and more.
  • Advanced Embedded GUI: Intuitive and user-friendly interface.
  • SQL and PromQL Support: Query logs and traces with SQL, and metrics with SQL and PromQL.
  • Single Binary or HA Installation: Install using a single binary for small deployments or in HA mode for large deployments.
  • Versatile Storage Options: Supports local disk, S3, MinIO, GCS, Azure Blob Storage.
  • High Availability and Clustering: Ensures reliable and scalable performance.
  • Dynamic Schema: Adapts to your data structure seamlessly.
  • Built-in Authentication: Secure and ready to use.
  • Ease of Operation: Designed for simplicity and efficiency.
  • Seamless Upgrades: Hassle-free updates.
  • Multilingual UI: Supports 11 languages, including English, Spanish, German, French, Chinese, and more.

For a full list of features, check the documentation.

⚡️ Quick start

🐳 Docker:

docker run -d \
      --name openobserve \
      -v $PWD/data:/data \
      -p 5080:5080 \
      -e ZO_ROOT_USER_EMAIL="root@example.com" \
      -e ZO_ROOT_USER_PASSWORD="Complexpass#123" \
      public.ecr.aws/zinclabs/openobserve:latest

🐙 Docker Compose:

services:
  openobserve:
    image: public.ecr.aws/zinclabs/openobserve:latest
    restart: unless-stopped
    environment:
      ZO_ROOT_USER_EMAIL: "root@example.com"
      ZO_ROOT_USER_PASSWORD: "Complexpass#123"
    ports:
      - "5080:5080"
    volumes:
      - data:/data
volumes:
  data:

For other ways to quickly install OpenObserve or use OpenObserve cloud, check quickstart documentation.

For installing OpenObserve in HA mode, check HA deployment documentation.

📷 Screenshots

Home

Home

Logs

Logs

Traces (OpenTelemetry)

Trace details page Traces using OpenTelemetry

Golden metrics based on traces Traces golden metrics

Visualizations and Dashboards

Dashboard Dashboard Create panel Map

Front end monitoring

Performance analytics Performance

Session replay Session replay

Error tracking Error tracking

Alerts

Alerts

Streams

Streams

Ingestion

Ingestion

Ingestion

SBOM

Software Bill of Materials for OpenObserve

Rust

SBOM can be found here. You can analyze it using dependency track.

In order to generate the SBOM, you can use the following commands:

Install cargo-cyclonedx:

cargo install cargo-cyclonedx

Generate the SBOM:

cargo-cyclonedx cyclonedx

JavaScript

SBOM can be found here. You can analyze it using dependency track.

In order to generate the SBOM, you can use the following commands:

Install cyclonedx-npm:

npm install --global @cyclonedx/cyclonedx-npm

Generate the SBOM:

cd web
cyclonedx-npm > sbom.json         

⚖️ License

OpenObserve is licensed under the AGPL-3.0 license. For more details, see the LICENSE.

🌍 Community

🔗 Join OpenObserve community on Slack

Slack

Easiest way to get support is to join the Slack channel.

📱 Join OpenObserve community on WeChat

S
Description
Open source observability platform for logs, metrics, traces, RUM (web, android, ios), Session replay, pipelines, SLO and LLM observability. A sophisticated, simple and highly performant alternative to Datadog, Splunk, and Elasticsearch with 140x lower storage costs and single binary deployment.
Readme AGPL-3.0
1 GiB
Languages
TypeScript 40.1%
Rust 29.4%
Vue 16.9%
JavaScript 10.9%
Python 2.3%
Other 0.3%