25c0fe6382
- #10446 Design at : https://github.com/openobserve/designs/tree/main/auto-query-builder-on-logs-page # Logs Query Builder - Behavior Guide What happens in different scenarios when using the Build tab. --- ## Switching to Build Tab - Scenarios ### Scenario 1: No Query, SQL Mode OFF **Starting state:** - Query editor is empty - SQL Mode is OFF - Stream "default" is selected **What happens when you click Build:** 1. SQL Mode turns ON automatically 2. Query is generated: `SELECT * FROM "default"` 3. Quick Mode turns ON (if enabled in config) 4. Build tab opens in **Custom SQL Mode** 5. Chart type: **Table** (with dynamic columns = true) --- ### Scenario 2: No Query, SQL Mode ON **Starting state:** - Query editor is empty - SQL Mode is ON - Stream "default" is selected **What happens when you click Build:** 1. Query is generated: `SELECT * FROM "default"` 2. Build tab opens in **Builder Mode** 3. Chart type: **Line** (default) --- ### Scenario 3: Simple SELECT * Query **Starting state:** - Query: `SELECT * FROM "logs"` - SQL Mode is ON **What happens when you click Build:** 1. Query is parsed but no explicit fields found (`*` = all columns) 2. Build tab opens in **Custom SQL Mode** 3. Chart type: **Table** (with dynamic columns) 4. Quick Mode turns ON (SELECT * pattern detected) 5. Query editor is **editable** --- ### Scenario 4: Histogram Query (Time Series) **Starting state:** - Query: `SELECT histogram(_timestamp) as time, count(*) as count FROM "logs" GROUP BY time` - SQL Mode is ON **What happens when you click Build:** 1. Query is parsed successfully 2. Build tab opens in **Builder Mode** 3. Chart type: **Bar** (default chart type) 4. Fields populated: X-axis = histogram, Y-axis = count --- ### Scenario 5: Count Only Query (No Grouping) **Starting state:** - Query: `SELECT count(*) as total FROM "logs"` - SQL Mode is ON **What happens when you click Build:** 1. Query is parsed successfully 2. Build tab opens in **Builder Mode** 3. Chart type: **Metric** (big number display - only Y-axis, no X-axis) --- ### Scenario 6: Query with 3+ GROUP BY Fields **Starting state:** - Query: `SELECT status, method, host, count(*) FROM "logs" GROUP BY status, method, host` - SQL Mode is ON **What happens when you click Build:** 1. Query is parsed successfully 2. Build tab opens in **Builder Mode** 3. Chart type: **Table** (too many grouping fields for chart) --- ### Scenario 7: Simple JOIN Query **Starting state:** - Query: `SELECT a.field, b.field FROM "stream1" a JOIN "stream2" b ON a.id = b.id` - SQL Mode is ON **What happens when you click Build:** 1. Query is parsed successfully 2. Build tab opens in **Builder Mode** 3. Chart type: **Bar** (default) 4. JOIN is recognized and shown in builder --- ### Scenario 8: Nested JOIN Query **Starting state:** - Query: `SELECT * FROM "a" JOIN ("b" JOIN "c" ON b.x = c.x) ON a.y = b.y` - SQL Mode is ON **What happens when you click Build:** 1. **Complex pattern detected** (nested JOIN with parentheses) 2. Build tab opens in **Custom SQL Mode** 3. Chart type: **Table** (with dynamic columns) 4. Query editor is **editable** 5. Field selectors are **hidden** --- ### Scenario 9: CTE Query (WITH clause) **Starting state:** - Query: `WITH recent AS (SELECT * FROM "logs" WHERE _timestamp > now() - interval '1 hour') SELECT * FROM recent` - SQL Mode is ON **What happens when you click Build:** 1. **Complex pattern detected** (CTE) 2. Build tab opens in **Custom SQL Mode** 3. Chart type: **Table** (with dynamic columns) 4. Query editor is **editable** --- ### Scenario 10: Subquery **Starting state:** - Query: `SELECT * FROM (SELECT status, count(*) FROM "logs" GROUP BY status) subq` - SQL Mode is ON **What happens when you click Build:** 1. **Complex pattern detected** (subquery) 2. Build tab opens in **Custom SQL Mode** 3. Chart type: **Table** (with dynamic columns) --- ### Scenario 11: UNION Query **Starting state:** - Query: `SELECT * FROM "logs1" UNION SELECT * FROM "logs2"` - SQL Mode is ON **What happens when you click Build:** 1. **Complex pattern detected** (UNION) 2. Build tab opens in **Custom SQL Mode** 3. Chart type: **Table** (with dynamic columns) --- ### Scenario 12: Window Function Query **Starting state:** - Query: `SELECT *, ROW_NUMBER() OVER (PARTITION BY status ORDER BY _timestamp) FROM "logs"` - SQL Mode is ON **What happens when you click Build:** 1. **Complex pattern detected** (window function) 2. Build tab opens in **Custom SQL Mode** 3. Chart type: **Table** (with dynamic columns) --- ### Scenario 13: Nested Function Query **Starting state:** - Query: `SELECT ceil(count(*)) as rounded_count FROM "logs"` - SQL Mode is ON **What happens when you click Build:** 1. **Complex pattern detected** (nested function - function inside function) 2. Build tab opens in **Custom SQL Mode** 3. Chart type: **Table** (with dynamic columns) --- ### Scenario 14: JSON Operations Query **Starting state:** - Query: `SELECT data->>'name' as name FROM "logs"` - SQL Mode is ON **What happens when you click Build:** 1. **Complex pattern detected** (JSON operator) 2. Build tab opens in **Custom SQL Mode** 3. Chart type: **Table** (with dynamic columns) --- ### Scenario 15: CASE/WHEN Query **Starting state:** - Query: `SELECT CASE WHEN status >= 500 THEN 'error' ELSE 'ok' END as status_type, count(*) FROM "logs" GROUP BY status_type` - SQL Mode is ON **What happens when you click Build:** 1. Query is parsed successfully (CASE/WHEN is supported) 2. Build tab opens in **Builder Mode** 3. Chart type: **Bar** (default) 4. CASE/WHEN shown as "raw" field type --- ### Scenario 16: Multiple JOINs (Flat) **Starting state:** - Query: `SELECT a.field as "field1", b.field as "field2" FROM "a" INNER JOIN "b" AS stream_0 ON a.x = b.x INNER JOIN "c" AS stream_1 ON b.y = c.y GROUP BY field1, field2 ORDER BY field1 ASC` - SQL Mode is ON **What happens when you click Build:** 1. Query is parsed successfully (flat JOINs are supported) 2. Build tab opens in **Builder Mode** 3. Chart type: **Bar** (default) --- ### Scenario 17: Opening Shared Link (First Time) **Starting state:** - URL contains `build_data` parameter with chart type "bar" - Query in URL: `SELECT status, count(*) FROM "logs" GROUP BY status` **What happens:** 1. Build tab opens 2. Query is parsed successfully 3. Chart type: **Bar** (restored from URL - first toggle) 4. Build tab opens in **Builder Mode** --- ### Scenario 18: Switching Back to Build Tab (After First Time) **Starting state:** - Previously visited Build tab - URL still has `build_data` with chart type "bar" - Query: `SELECT count(*) FROM "logs"` (metric query) **What happens:** 1. Build tab opens 2. Query is parsed successfully 3. Chart type: **Metric** (auto-selected, URL ignored on subsequent toggles) --- ### Scenario 19: Invalid SQL Syntax **Starting state:** - Query: `SELEC * FORM "logs"` (typos) - SQL Mode is ON **What happens when you click Build:** 1. **Parse error** (invalid SQL) 2. Build tab opens in **Custom SQL Mode** 3. Chart type: **Table** (with dynamic columns) 4. Query editor is **editable** (so user can fix the query) --- ### Scenario 20: Query with WHERE Filters **Starting state:** - Query: `SELECT histogram(_timestamp), count(*) FROM "logs" WHERE status = 500 AND method = 'GET' GROUP BY 1` - SQL Mode is ON **What happens when you click Build:** 1. Query is parsed successfully 2. Build tab opens in **Builder Mode** 3. Chart type: **Bar** (default) 4. Filters populated in filter section --- ## Summary Table | Query Type | Mode | Chart Type | |------------|------|------------| | SELECT * | **Custom** | **Table** | | Histogram + count | Builder | Bar | | Count only (no GROUP BY) | Builder | Metric | | 1-2 GROUP BY fields | Builder | Bar | | 3+ GROUP BY fields | Builder | Table | | Simple JOIN | Builder | Bar | | CASE/WHEN | Builder | Bar | | Multiple flat JOINs | Builder | Bar | | CTE (WITH clause) | **Custom** | **Table** | | Subquery | **Custom** | **Table** | | Nested JOIN | **Custom** | **Table** | | UNION/INTERSECT/EXCEPT | **Custom** | **Table** | | Window functions | **Custom** | **Table** | | Nested functions | **Custom** | **Table** | | JSON operations | **Custom** | **Table** | | Invalid SQL | **Custom** | **Table** | --- ## Quick Mode Auto-Enable Quick Mode is automatically enabled when switching to Build tab if: | Condition | Quick Mode | |-----------|------------| | SQL Mode was OFF | ✅ Enabled | | Query is `SELECT * FROM ...` | ✅ Enabled | | Complex aggregation query | ❌ Not changed | **Requires:** `quick_mode_enabled = true` in config --- ## SQL Mode Auto-Enable | Starting State | After Clicking Build | |----------------|---------------------| | SQL Mode OFF | SQL Mode **ON** | | SQL Mode ON | SQL Mode ON (no change) | **Build tab always requires SQL Mode to be ON.** --- ## Chart Type Priority When auto-selecting chart type: 1. **Custom Mode?** → Table 2. **3+ GROUP BY fields?** → Table 3. **Only Y-axis (no X, no breakdown)?** → Metric 4. **Everything else** → Bar (default) **Exception:** First visit with shared link uses chart type from URL. --- ## Builder Mode vs Custom Mode | Aspect | Builder Mode | Custom Mode | |--------|--------------|-------------| | Query Editor | Read-only | Editable | | Field Selectors | Visible | Hidden | | When Used | Parseable queries | Complex/unparseable queries | | Default Chart | Bar/Metric | Table | --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: ktx-akshay <akshay.patil@kiara.tech>
Open source Datadog alternative for logs, metrics, traces, and frontend monitoring. Modern observability platform: 10x easier, 140x lower storage cost, high performance, petabyte scale.
OpenObserve (O2) is a cloud-native observability tool built for logs, metrics, traces, analytics, and Real User Monitoring (RUM). It's designed as a cost-effective alternative to Datadog, Splunk, and Elasticsearch for teams that need full observability without the complexity or cost.
Why OpenObserve?
OpenObserve is a single observability tool for all your monitoring needs. Here's why teams choose OpenObserve:
| Benefit | Description |
|---|---|
| 140x Lower Storage Cost | Parquet columnar storage + S3-native architecture dramatically reduces costs vs Elasticsearch |
| Single Binary Deployment | Get running in under 2 minutes. No complex cluster setup required |
| OpenTelemetry Native | Built on OpenTelemetry standard—no vendor lock-in |
| Unified Platform | Logs, metrics, traces, RUM, dashboards, alerts in one tool |
| High Performance | Better query performance than Elasticsearch on 1/4 the hardware |
| SQL + PromQL | Query logs/traces with SQL, metrics with SQL or PromQL—no proprietary query language |
| Built in Rust | Memory-safe, high-performance, single binary |
Cost Comparison: OpenObserve Vs Elasticsearch
🎥 Introduction Video
🏗️ Architecture
OpenObserve achieves 140x lower storage costs and high performance through its modern architecture:
- Parquet columnar storage: Efficient compression and query performance
- S3-native design: Leverages inexpensive object storage with intelligent caching
- Built in Rust: Memory-safe, high-performance, single binary deployment
- Partitioning, indexing and smart caching: Reduces search space by up to 99% for most queries
- Native multi-tenancy: Organizations and streams as first-class concepts with complete data isolation
- Stateless architecture: Enables rapid scaling and low RPO/RTO for disaster recovery
This architecture delivers 140x cost savings while providing better performance than Elasticsearch.
Scale & Deployment
- Thousands of concurrent users can query a single cluster simultaneously
- Single binary scales to terabytes - unique in the observability space
- High Availability mode scales to petabytes for the most demanding workloads
- Multi-region deployments with cluster federation via Super Cluster architecture (Enterprise feature)
- Federated search across regions and clusters (Enterprise feature)
- Capacity planning tools to size deployments for your workload
High Availability & Disaster Recovery
Deploy in High Availability mode with clustering for mission-critical workloads requiring maximum uptime and performance.
Low RPO/RTO: OpenObserve's stateless architecture with S3-backed storage enables very low Recovery Point Objective (RPO) and Recovery Time Objective (RTO). Stateless nodes can be rapidly restarted, and data durability is guaranteed by S3's 99.999999999% (11 nines) durability. That's a lot of nines.
Read detailed architecture documentation →
Read enterprise deployment guide →
Features
For a full list of features, check the documentation.
Logs Management
OpenObserve provides a centralized log management solution with full-text search, SQL queries, and powerful filtering. Built on Parquet columnar storage for 140x lower storage costs than Elasticsearch. Instantly search through all your logs using quick filters and a powerful query builder. Create dashboards from log data and set up alerts. Read more →
Distributed Tracing
Distributed tracing is essential to troubleshoot issues in microservices applications. Powered by OpenTelemetry, tracing in OpenObserve helps you track user requests across services to identify performance bottlenecks. See user requests in a detailed breakdown with Flamegraphs and Gantt Charts. Click on any span to see the entire trace and understand where issues occurred. Read more →
Metrics & Dashboards
Ingest metrics from your infrastructure or applications and create customized dashboards. 19+ built-in chart types plus custom charts for 200+ visualization variations. Query metrics with SQL or PromQL. Add multiple queries and combine them with formulae. Read more →
Frontend Monitoring (RUM)
Real User Monitoring with performance tracking, error logging, and session replay. Understand exactly what your users experience. Read more →
Alerts
Get notified when anything unusual happens in your application. Set alerts on any telemetry signal (logs, metrics, traces), create thresholds, and configure notification channels.
Advanced features like alert history and anomaly detection help you identify issues before they reach your customers. Read more →
Pipelines
Enrich, redact, reduce, or normalize data on ingest. Stream processing for logs-to-metrics conversion and more. No external tools required. Read more →
Getting Started
OpenObserve Cloud (Fastest Way)
Get started in minutes without managing infrastructure. Free tier includes up to 200 GB/day ingestion. Get Started Free →
🐳 Docker:
docker run -d \
--name openobserve \
-v $PWD/data:/data \
-p 5080:5080 \
-e ZO_ROOT_USER_EMAIL="root@example.com" \
-e ZO_ROOT_USER_PASSWORD="Complexpass#123" \
public.ecr.aws/zinclabs/openobserve:latest
For other ways to quickly install OpenObserve or use OpenObserve cloud, check quickstart documentation.
For installing OpenObserve in High Availability mode, check High Availability deployment documentation.
Production Ready
OpenObserve is battle-tested in production environments worldwide:
- Thousands of active deployments across diverse industries
- Largest deployment: 2+ PB/day ingestion
- Single binary scales to terabytes—unique in the observability space
Comparisons to Familiar Tools
OpenObserve vs Datadog
| Aspect | OpenObserve | Datadog |
|---|---|---|
| Deployment | Self-hosted or Cloud | SaaS only |
| Pricing model | Per-GB (free up to 200GB/day) | Per-host + per-GB |
| Open source | Yes (AGPL-3.0) | No |
| OpenTelemetry | Native OTLP | Supported |
| Query language | SQL + PromQL | Proprietary |
| Vendor lock-in | None | High |
OpenObserve is a Datadog alternative that gives you full control over your data and costs.
OpenObserve vs Elasticsearch
| Aspect | OpenObserve | Elasticsearch |
|---|---|---|
| Storage cost | 140x lower | High (hot/warm/cold tiers) |
| Setup complexity | Single binary | Complex cluster management |
| Query language | SQL | Lucene/KQL |
| Hardware requirements | 1/4 the resources | High memory/CPU |
OpenObserve achieves 140x lower storage costs through Parquet columnar format and S3-native architecture.
OpenObserve vs Splunk
| Aspect | OpenObserve | Splunk |
|---|---|---|
| Licensing | Open source | Expensive enterprise licensing |
| Deployment | Single binary or HA cluster | Complex |
| Query language | SQL + PromQL | SPL (proprietary) |
| Cost | Predictable, low | Unpredictable, high |
OpenObserve is an open-source Splunk alternative without the licensing complexity.
OpenObserve vs Grafana/Loki/Prometheus Stack
| Aspect | OpenObserve | Grafana Stack |
|---|---|---|
| Components | Single platform | Multiple tools (Grafana + Loki + Prometheus + Tempo) |
| Management | One binary | Multiple deployments |
| High cardinality | Full support | Loki struggles with high cardinality |
| Query performance | Fast on large volumes | Loki slow on large data |
OpenObserve consolidates logs, metrics, and traces in one platform instead of stitching together multiple tools.
📷 Screenshots
OpenObserve includes a powerful web UI for logs, traces, dashboards, alerts, and more.
Logs Search
Distributed Tracing
Trace details page with full request flow visualization:
Dashboards
Frontend Monitoring
Real user monitoring with session replay:
See more screenshots
Home
Golden Metrics from Traces
More Dashboard Examples
Performance Analytics
Error Tracking
Alerts
Streams
Ingestion
Pipeline
Functions
🔐 Security & Compliance
Security Features
- Highly secure architecture with secure container images
- Sensitive Data Redaction (SDR): Automatically redact sensitive data during ingestion and query time (Enterprise feature)
- Data encryption: At rest and in transit
- Single Sign-On (SSO): OIDC, OAuth, SAML, LDAP/AD integration (Enterprise feature)
- Role-Based Access Control (RBAC): Granular permissions management (Enterprise feature) - Learn more →
Compliance Certifications
- ✅ SOC 2 Type II certified
- ✅ ISO 27001 certified
- ✅ GDPR compliant
- ✅ HIPAA ready (BAA available with Enterprise contracts)
OpenObserve meets the stringent security and compliance requirements of regulated industries including finance, healthcare, and government.
⚖️ License
Open Source Edition: Licensed under AGPL-3.0. We chose AGPL to ensure that improvements to OpenObserve remain open source and benefit the entire community. This license protects the commons while still allowing free commercial use.
Enterprise Edition: Licensed under a commercial Enterprise License Agreement, not AGPL. This provides additional flexibility for enterprise deployments and eliminates any concerns about AGPL requirements.
For more details:
💼 Enterprise Support
OpenObserve is built as a true open source project, and we're committed to the community. The open source version is feature-complete and production-ready - it includes logs, metrics, traces, dashboards, alerts, pipelines, and everything you need to run observability at scale. It will always remain actively maintained and free to use without restrictions.
Enterprise Edition
For organizations requiring enterprise-grade features and support, we offer an Enterprise edition with:
Enterprise Features:
- Single Sign-On (SSO): OIDC, OAuth, SAML 2.0, LDAP/AD, and integration with major identity providers (Okta, Azure Entra, Google, GitHub, GitLab, Keycloak)
- Advanced RBAC: Granular role-based access control with custom roles and permissions - Learn more →
- Audit trails: Comprehensive immutable audit logs with configurable retention
- Federated search: Query across multiple clusters and regions with Super Cluster
- Sensitive Data Redaction (SDR): Automatically redact PII and sensitive data during ingestion and queries
- Advanced encryption: AES-256 SIV cipher keys with Google Tink KeySet and Akeyless integration
- Query management: Control query resource usage and priorities
- Workload management (QoS): Quality of Service controls for multi-tenant environments
Enterprise Support & SLAs:
- Dedicated support with contractual SLA guarantees
- Priority response times for critical issues
- Technical account management
- Architecture review and deployment assistance
- Migration support from existing tools
- Training and onboarding programs
Pricing:
- Free tier: Up to 200 GB/day of ingestion (roughly 6 TB/month), including full commercial use
- Registration required at 100 GB/day
- Volume discounts and multi-year contracts available
- View complete feature comparison →
For enterprise inquiries and custom deployments, contact our sales team.
🤝 Contributing
We welcome contributions from the community! Whether you're fixing bugs, adding features, improving documentation, or sharing feedback, your help makes OpenObserve better for everyone.
To get started, please read our Contributing Guide which covers:
- How to set up your development environment
- Code standards and best practices
- How to submit pull requests
- Reporting bugs and requesting features
🌍 Community
The best way to get help, share ideas, and connect with other OpenObserve users is through our community channels. We're a friendly group of developers, operators, and observability enthusiasts.
🔗 Join us on Slack
Our Slack community is the most active place for:
- Getting help with installation and configuration
- Sharing best practices and use cases
- Discussing feature requests and roadmap
- Connecting with the core team and other users
📱 Join OpenObserve community on WeChat
Other ways to connect
- 💬 GitHub Discussions - For longer-form discussions and Q&A
- 🐛 GitHub Issues - Report bugs or request features
- 📖 Documentation - Guides, tutorials, and API references
❓ FAQ
How does OpenObserve achieve 140x lower storage costs?
Through a combination of Parquet columnar storage format (efficient compression), S3-native architecture (leveraging inexpensive object storage). See the detailed comparison chart in the "Why OpenObserve?" section above.
What are the limitations?
All data in OpenObserve is immutable - once ingested, it cannot be modified or deleted (only entire retention periods can be dropped). This is by design and is actually a feature for logs and compliance requirements, ensuring data integrity and audit trails.
Is this production-ready?
Yes. OpenObserve is running in production with thousands of deployments worldwide, including environments processing in excess of 2 PB/day. See our customer stories for real-world examples.
How does query performance compare to Elasticsearch?
OpenObserve delivers better performance than Elasticsearch for most workloads. Users report faster search performance and significantly faster analytics queries, all while using 1/4th the hardware resources. The columnar storage format (Parquet) is particularly effective for complex aggregations and analytics workloads.
Is there a steep learning curve?
No. OpenObserve is designed to be intuitive from day one:
- Familiar query languages: Use SQL for logs and traces, PromQL for metrics - no proprietary query language to learn
- Easy-to-use GUI: Intuitive interface with drag-and-drop dashboard builder
- Helpful community: Active Slack community and comprehensive documentation to help you get started quickly
- No complex tuning: Unlike Elasticsearch, you don't need to understand shards, replicas, heap sizes, or other complex configurations. Just install and go.
Most users are productive within hours, not weeks. Some even claim minutes, but we'll let you be the judge.
🔐 SBOM
Software Bill of Materials for OpenObserve
Rust
SBOM can be found here. You can analyze it using dependency track.
In order to generate the SBOM, you can use the following commands:
Install cargo-cyclonedx:
cargo install cargo-cyclonedx
Generate the SBOM:
cargo-cyclonedx cyclonedx
JavaScript
SBOM can be found here. You can analyze it using dependency track.
In order to generate the SBOM, you can use the following commands:
Install cyclonedx-npm:
npm install --global @cyclonedx/cyclonedx-npm
Generate the SBOM:
cd web
cyclonedx-npm > sbom.json