# Nested Functions and Joins - User Guide
## Overview
This guide covers two powerful features that enable sophisticated data
analysis in OpenObserve:
- **Nested Functions**: Combine multiple functions together to create
complex calculations
- **Joins**: Merge data from multiple streams (data sources) based on
common fields
These features work in the **Auto SQL** mode of dashboard panels,
allowing you to write advanced queries without deep SQL knowledge.
---
## What's New
### Why These Features?
**Before:**
- Limited function composition - you could only use basic, single-level
functions
- No way to combine data from multiple streams in one query
- Complex analytics required exporting data and manual processing
**Now:**
- Compose unlimited nested functions with full DataFusion support
- Join multiple streams directly in the dashboard panel editor
- Create sophisticated analytics queries with a visual interface
### Key Benefits
✓ Calculate complex metrics (error rates, percentiles, conditional
aggregations)
✓ Build advanced dashboards without custom code
✓ Support for all DataFusion aggregate and scalar functions
---
## Nested Functions
### What Are Nested Functions?
Nested functions let you combine multiple functions together - using the
output of one function as input to another. Think of it like a cooking
recipe where each step builds on the previous one.
**Simple Example:**
- Basic: `AVG(response_time)` - calculates average response time
- Nested: `ROUND(AVG(response_time), 2)` - calculates average AND rounds
it to 2 decimal places
### Supported Functions
OpenObserve supports **all DataFusion functions**, giving you powerful
analytics capabilities:
**Aggregate Functions** (work across multiple rows):
- `count`, `sum`, `avg`, `min`, `max` - basic aggregations
- `median`, `stddev`, `variance` - statistical functions
- `percentile_cont` - calculate percentiles (e.g., p95, p99)
- `string_agg` - combine text values
**Mathematical Functions**:
- `round`, `ceil`, `floor` - rounding operations
- `abs`, `power`, `sqrt` - mathematical operations
- `log`, `log10`, `log2` - logarithmic functions
- `sin`, `cos`, `tan` - trigonometric functions
**String Functions**:
- `concat`, `upper`, `lower`, `trim`
- `substring`, `length`, `replace`
**Conditional Functions**:
- `CASE WHEN ... THEN ... ELSE ... END` - conditional logic
- `COALESCE` - return first non-null value
📖 **Full Reference**: [DataFusion Aggregate
Functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html)
| [DataFusion Scalar
Functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html)
---
### Using Nested Functions (Step-by-Step)
#### Step 1: Create or Edit a Dashboard Panel
Open your dashboard and create a new panel or edit an existing one. Make
sure you're in **Auto SQL** mode (not Custom SQL).
#### Step 2: Select a Function
In the Y-Axis or column configuration, click **Select Function** to
choose your base function.
<img width="795" alt="Nested Function - Adding Layers"
src="https://github.com/user-attachments/assets/0b6b134e-59b8-48b1-a4ae-52fb4800d772"
/>
In this example, we're using the `Concat` function to combine multiple
fields (namespace and pod name).
#### Step 3: Add Nested Layers
Click the function icon to add another function layer. You can nest
functions multiple times.
<img width="793" alt="Nested Function - Select Base Function"
src="https://github.com/user-attachments/assets/4bd9e03e-1104-4e82-b2ec-a34e6bb64870"
/>
Here we're building a nested function:
- Inner layer: `Avg(severity)` - calculates average severity
- Outer layer: `Ceil()` - rounds up the average to the nearest integer
#### Step 4: Build Complex Expressions
For advanced analytics, you can combine nested functions with
conditional logic using `CASE WHEN` statements.
<img width="1340" alt="Complex Nested Function with CASE WHEN"
src="https://github.com/user-attachments/assets/4c3e3623-bfad-44c4-a760-e10cc94d4a95"
/>
This example shows a sophisticated calculation:
```sql
ROUND(100.0 * SUM(CASE WHEN COALESCE(severity, 0) >= 4 THEN 1 ELSE 0 END) / COUNT(_timestamp), 2)
```
**What this does:**
1. `COALESCE(severity, 0)` - replaces null severity with 0
2. `CASE WHEN severity >= 4 THEN 1 ELSE 0` - returns 1 for high
severity, 0 otherwise
3. `SUM(...)` - counts high severity events
4. Divide by `COUNT(_timestamp)` - calculates percentage
5. Multiply by `100.0` - converts to percentage
6. `ROUND(..., 2)` - rounds to 2 decimal places
**Result**: Percentage of high-severity events, rounded to 2 decimals
#### Step 5: View the Generated SQL
Switch to the **Raw** tab to see the complete SQL query that was
generated from your nested function configuration.
<img width="1388" alt="View Generated SQL"
src="https://github.com/user-attachments/assets/d7f6111a-26c6-4dd0-9e9f-e26092fe6368"
/>
You can review the auto-generated SQL in the Query section at the bottom
of the panel editor. The chart visualization updates in real-time as you
build your nested functions.
---
## Joins
### What Are Joins?
Joins allow you to combine data from two or more different streams (data
sources) based on common fields between them.
**Example:**
**employee_data_join** stream:
```json
[
{"employee_id": 1, "name": "Alice", "department_id": 1},
{"employee_id": 2, "name": "Bob", "department_id": 2},
{"employee_id": 3, "name": "Charlie", "department_id": null},
{"employee_id": 4, "name": "David", "department_id": 1},
{"employee_id": 5, "name": "Eve", "department_id": 3}
]
```
**dept_data_join** stream:
```json
[
{"department_id": 1, "department_name": "HR"},
{"department_id": 2, "department_name": "Engineering"},
{"department_id": 3, "department_name": "Sales"},
{"department_id": 4, "department_name": "Marketing"}
]
```
A join lets you combine these streams to see the department name
alongside each employee, matched by `department_id`.
### Types of Joins
OpenObserve supports three types of joins:
| Join Type | Description | When to Use |
|-----------|-------------|-------------|
| **INNER JOIN** | Returns only records with matches in both streams |
When you need data that exists in both sources |
| **LEFT JOIN** | Returns all records from left stream + matching
records from right | When you want all data from your primary source,
with supplemental info where available |
| **RIGHT JOIN** | Returns all records from right stream + matching
records from left | When you want all data from your secondary source,
with supplemental info where available |
> **Note:** UNION and CROSS joins are not currently supported.
---
### Using Joins (Step-by-Step)
#### Step 1: Access the Joins Section
In your dashboard panel editor, locate the **Joins** section. Click the
**+** button to add a join.
<img width="1725" alt="Access Joins Interface"
src="https://github.com/user-attachments/assets/35db4f28-c526-43a2-bdd1-470e188f53d7"
/>
In this example:
- **Primary stream**: `employee_data_join` (contains employee names)
- We'll join it with `dept_data_join` (contains department information)
- The interface shows available fields from both streams in the left
panel
#### Step 2: Configure the Join
Click on the **Joins** dropdown and select the stream you want to join.
Then configure:
1. **Joining Stream**: Select the stream to join (e.g.,
`employee_data_join`)
2. **Join Type**: Choose from LEFT, INNER, or RIGHT using the visual
toggle
3. **On Stream**: Select the target stream (e.g., `dept_data_join`)
<img width="1725" alt="Configure Join Type"
src="https://github.com/user-attachments/assets/35db4f28-c526-43a2-bdd1-470e188f53d7"
/>
The interface displays: **"Performing a Left Join between
employee_data_join and dept_data_join"**
Notice the visual toggle showing three join types:
- **Left** (selected in this example)
- **Inner**
- **Right**
#### Step 3: Define the Joining Clause
Set up the join condition by specifying which fields to match between
the two streams:
**Clause 1:**
- **Select Field** (Left): Choose a field from the first stream (e.g.,
`department_id`)
- **Select Operator**: Choose comparison operator (usually `=`)
- **Select Field** (Right): Choose the matching field from the second
stream (e.g., `department_id`)
You can add multiple clauses using the **+** button for complex join
conditions (e.g., joining on multiple fields or adding additional
filters).
#### Step 4: View the Results
After configuring the join, the data table will show combined results
from both streams.
<img width="1724" alt="Join Results"
src="https://github.com/user-attachments/assets/94ca9f33-6cfa-4ad1-9b84-bee9db34f912"
/>
In this example:
- **Name** column from `employee_data_join` (Alice, Bob, Charlie, David,
Eve)
- **Department Name** column from `dept_data_join` (HR, Engineering, HR,
Sales)
- Data successfully joined on `department_id`
#### Step 5: Review the Generated SQL
Check the **Query** section at the bottom to see the generated SQL:
```sql
SELECT employee_data_join.name as "x_axis_1",
stream_0.department_name as "x_axis_2"
FROM "employee_data_join"
LEFT JOIN "dept_data_join" AS stream_0
ON employee_data_join.department_id = stream_0.department_id
```
The SQL query shows exactly how OpenObserve is combining your data
streams.
---
---
## Examples
### Example 1: String Concatenation with Nested Functions
**Use Case**: Combine multiple fields into a single display value
```sql
Concat(k8s_namespace_name, '-', k8s_pod_name)
```
This joins namespace and pod names with a hyphen separator.
### Example 2: Statistical Analysis with Nested Math Functions
**Use Case**: Calculate rounded average of a metric
```sql
Ceil(Avg(severity))
```
This calculates the average severity level and rounds up to the nearest
whole number.
### Example 3: Conditional Percentage Calculation
**Use Case**: Calculate percentage of high-severity events
```sql
ROUND(100.0 * SUM(CASE WHEN COALESCE(severity, 0) >= 4 THEN 1 ELSE 0 END) / COUNT(_timestamp), 2)
```
**Breakdown:**
- Handles null values with `COALESCE`
- Counts events where severity >= 4
- Calculates percentage of total events
- Rounds to 2 decimal places
### Example 4: Employee-Department Join
**Use Case**: Show employee names with their department information
```sql
SELECT employee_data_join.name as "Name",
dept_data_join.department_name as "Department Name"
FROM employee_data_join
LEFT JOIN dept_data_join
ON employee_data_join.department_id = dept_data_join.department_id
```
This displays all employees with their department names, showing null
for employees without a department assignment.
### Example 5: Complex Analytics - Joins + Nested Functions
**Use Case**: Calculate error rate per service with deployment
information
```sql
SELECT
logs.service_name,
deployments.version,
COUNT(*) as total_requests,
ROUND((COUNT(CASE WHEN logs.status_code >= 500 THEN 1 END) * 100.0 / COUNT(*)), 2) as error_rate_percent
FROM service_logs as logs
LEFT JOIN deployments ON logs.service_name = deployments.service_name
GROUP BY logs.service_name, deployments.version
ORDER BY error_rate_percent DESC
```
This combines:
- Join between service logs and deployment data
- Nested functions for conditional counting and percentage calculation
- Grouping and sorting for analysis
---
## Tips and Best Practices
### Nested Functions Best Practices
✅ **Start Simple**: Build your function one layer at a time
- Begin with inner function: `AVG(response_time)`
- Add outer layers: `ROUND(AVG(response_time), 2)`
✅ **Test Each Layer**: Verify results before adding complexity
- Test the inner function first
- Add one layer at a time and check output
✅ **Handle Null Values**: Use `COALESCE` to avoid unexpected nulls
```sql
COALESCE(AVG(response_time), 0)
```
✅ **Use Clear Aliases**: Name your calculated fields descriptively
```sql
ROUND(AVG(response_time), 2) as avg_response_ms
```
✅ **Check for Division by Zero**: Protect against zero denominators
```sql
CASE
WHEN COUNT(*) > 0 THEN (SUM(errors) * 100.0 / COUNT(*))
ELSE 0
END as error_rate
```
### Joins Best Practices
✅ **Choose the Right Join Type**:
- **INNER JOIN**: Both streams must have matching data
- **LEFT JOIN**: Keep all records from primary stream
- **RIGHT JOIN**: Keep all records from secondary stream
✅ **Verify Field Compatibility**:
- Match data types (don't join string to number)
- Ensure field names are correct
- Check that values actually match between streams
✅ **Consider Performance**:
- Join on indexed fields when possible
- Test with limited data first
---
## FAQ
### Q: Can I nest more than 2 functions?
**A**: Yes! You can nest as many functions as needed. Example:
```sql
ROUND(CEIL(AVG(COALESCE(response_time, 0))), 2)
```
### Q: What's the difference between LEFT and RIGHT joins?
**A**:
- **LEFT JOIN**: Keeps all records from the first (left) stream
- **RIGHT JOIN**: Keeps all records from the second (right) stream
- Choose based on which dataset you want to preserve completely
### Q: Can I join more than 2 streams?
**A**: Yes! Click the **+** button in the Joins section to add
additional joins. You can chain multiple joins together.
### Q: Do nested functions impact performance?
**A**: Complex nested functions may take longer to compute, especially
on large datasets. Test with small time ranges first and optimize your
expressions.
### Q: Can I use Custom SQL instead of Auto SQL?
**A**: Yes! If you prefer writing raw SQL, switch to **Custom SQL**
mode. However, Auto SQL provides a visual interface that's easier for
building queries.
---
## Additional Resources
📚 **DataFusion Documentation**:
- [Aggregate
Functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html)
- COUNT, SUM, AVG, etc.
- [Scalar
Functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html)
- ROUND, CONCAT, CASE, etc.
🔗 **OpenObserve Resources**:
- [GitHub Repository](https://github.com/openobserve/openobserve)
- [PR #9343 - Feature
Implementation](https://github.com/openobserve/openobserve/pull/9343)
- [OpenObserve Documentation](https://openobserve.ai/docs/)
💬 **Get Help**:
- [GitHub Issues](https://github.com/openobserve/openobserve/issues) -
Report bugs or request features
- [Community
Discussions](https://github.com/openobserve/openobserve/discussions) -
Ask questions and share ideas
---
Design at: https://github.com/openobserve/designs/tree/main/joins
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Fixes for various p2 bugs and UI issues, some other functionality
changes as well.
NOTE: because of some CLA issues, I have squashed all the original
commits into a single one. There are no code changes compared to before.
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Abhay Padamaniandgreptile-apps[bot]
Sai NikhilandByteBaker
Omkar Kesarkhane
Yashodhan JoshiandYashodhan Joshi
udev
