## Summary
Fast fields are used by search optimizations such as timestamp
filtering, histograms, multi-histograms, and top-N queries. Previously,
`build_footer_cache` only warmed indexed field metadata, so fast field
access could still trigger additional reads later. This change primes
fast field handles during footer cache construction so those reads can
be served from the cached directory.
## Changes
- Warm fast field handles while building the puffin footer cache.
- Refactor `warm_up_terms` to operate on a `SegmentReader`.
- Update the Tantivy search warm-up call site for the new
`warm_up_terms` signature.
- Add a unit test to verify footer cache generation includes fast field
cache entries.
## Testing
- Added `test_build_footer_cache_with_fast_field`.
- Ran formatting and clippy fixes.
## Summary
- Skip OTLP histogram `_sum`, `_min`, and `_max` records when the
corresponding optional stat is absent.
- Preserve existing `_count` and bucket records.
- Add coverage for histograms whose optional stats are missing.
## Root Cause
OTLP `HistogramDataPoint` defines `sum`, `min`, and `max` as optional
values. The previous ingestion path converted missing optional stats
into JSON `null` and still emitted derived metric records such as
`<metric>_min` and `<metric>_max`. The flattening layer drops `null`
fields, so those derived streams could be created without a `value`
field.
Fixes#13019.
## Validation
- `cargo test -p openobserve test_process_hist_data_point --quiet`
- `cargo test -p openobserve
otlp::tests::metric_name_processing_tests::test_histogram_metric_name_suffixes
--quiet`
- `git diff --check`
The two metrics were using an empty string as the label name even though
no label names were registered in the metrics. Hence, whenever this
metric is emitted, there is a runtime panic. This pr fixes that.
## Scheduler: per-module job pullers with dedicated worker pools (A3 +
A4)
### Why
Scheduled jobs are dispatched late at scale. A single shared puller uses
one FIFO, one cadence, and one worker pool across every module, so a
backlog or slow handler in one module (e.g. an alert burst or a bulk
backfill) starves all others. This is Part A of the scheduler
scalability design.
### What
Give **each `TriggerModule` its own pull loop, cadence, LIMIT budget,
channel, and worker pool**, behind a flag. Off by default — zero
behavior change until explicitly enabled.
- **`ZO_SCHEDULER_PER_MODULE_PULLERS`** (default `false`) — master
switch.
- **off** → single shared puller (legacy, byte-identical SQL + same
global advisory lock).
- **on** → one lane per module (Alert, Report, DerivedStream, Backfill,
AnomalyDetection, QueryRecommendations).
- **Module-filtered pull** — `pull(...)` gains `module:
Option<TriggerModule>`; when `Some(m)` it filters `WHERE module = m` and
takes a **per-module advisory lock** (`scheduler_pull_lock:{module}`,
strategy C3) so modules no longer serialize against each other. `None`
keeps the exact legacy query + global lock.
- **Per-module budgets/cadence** — `ZO_SCHEDULER_<MODULE>_CONCURRENCY` /
`_INTERVAL` env vars, `0` = inherit the global alert default. Backfill
defaults to the smallest budget (1) so bulk jobs can't crowd out
latency-sensitive modules.
### Changes
| Area | Files |
|---|---|
| Flag + per-module config | `src/config/src/config.rs` |
| `module` param on `pull` (trait + wrappers) |
`src/infra/src/scheduler/{mod,postgres,sqlite}.rs`,
`src/service/db/scheduler.rs` |
| Per-module lanes (puller + pool + channel) |
`src/service/alerts/scheduler/worker.rs` |
### Notes
- No new SQL statement variants beyond two (module-filtered / legacy);
bind ordinals preserved, `is_realtime/is_silenced` exclusion unchanged.
- The keep-alive task and channel-full back-pressure retry are preserved
verbatim.
- Deferred: per-module Prometheus metrics (A6) and `FOR UPDATE SKIP
LOCKED` benchmark (A2/C2).
### Rollout
Flip the flag **cluster-wide**. During a mixed on/off window the
per-module and global locks don't mutually exclude, but the atomic
`UPDATE ... WHERE status=Waiting ... RETURNING` still claims each row
once (no double-dispatch) — only transient loss of cross-node FIFO. Size
the Postgres/sqlx pool for the extra pullers before enabling at scale.
### Testing
- 11 unit tests in `worker.rs`: legacy single-lane path,
one-lane-per-module, concurrency floor, and full
`resolve_module_configs` coverage (inherit/override/backfill floor).
- `cargo fmt --all -- --check` clean · `cargo clippy -- -D warnings`
clean.
## Summary
This PR adds a metadata count fast path for follower search `count(*)`
queries.
When a query is a simple count with no filters, files that are fully
inside the query time range can be counted from file metadata instead of
being scanned from object storage.
## Changes
- Replaces the Tantivy-only aggregate rewrite with a more general
aggregate optimization rewrite.
- Adds support for combining normal DataFusion scan results, metadata
count results, and Tantivy optimized results in one `UnionExec`.
- Splits storage files into metadata-count files and normal scan files.
- Uses metadata count only when:
- enterprise feature is enabled
- the optimizer selects `IndexOptimizeMode::SimpleCount`
- the index condition is `Condition::All()`
- the file time range is fully inside `[query_start, query_end)`
- Keeps boundary-overlapping files on the normal scan path to avoid
incorrect counts.
- Updates scan stats and logs for the broader index optimization path.
- Adds unit tests for file splitting and aggregate rewrite behavior.
## Notes
- Metadata count is only used for simple `count(*)` queries without
filters.
- Files with `max_ts == query_end` are not counted from metadata because
the query range is right-open.
- `MetadataCountExec` is enterprise-only.
Fixes#13041.
## Root cause
Two broken string comparisons, both caused by Windows path
canonicalization:
**1. `src/config/src/config.rs` — `check_disk_cache_config()`**
`Path::canonicalize()` on Windows returns a verbatim extended-length
path (`\\?\C:\OpenObserve\data\cache`), but `sysinfo` returns plain
mount points (`C:\`). The `starts_with` check
`"\\?\C:\...".starts_with("C:\\")` is always `false` → `disk_total =
disk_free = 0` on every Windows deployment.
**2. `src/job/stats.rs` — `update_node_disk_usage()`**
The comparison was also reversed: `mount_point.starts_with(data_dir)`
checks whether the mount point path begins with the (much longer) data
directory path — `"C:\\".starts_with("C:\\OpenObserve\\data\\")` is
always `false`. The correct direction is
`data_dir.starts_with(mount_point)`.
## Consequences when disk sizes are 0
- Disk cache `max_size` is computed from near-zero `effective_free`,
leaving `result_max_size = 0`
- `disk.rs` GC panics at first tick: `max_size / result_max_size`
evaluates before `max(1, …)` → divide by zero, GC task dies permanently
- `check_disk_circuit_breaker()` always compares `0 > 0` → never trips →
no self-protection against disk exhaustion on Windows
## Fix
1. **`config.rs`**: strip the `\\?\` prefix from the canonicalized cache
dir string before the `starts_with` lookup against sysinfo mount points.
2. **`stats.rs`**: fix the reversed comparison
(`mount_point.starts_with(data_dir)` →
`data_dir.starts_with(mount_point)`) and apply the same `\\?\` prefix
strip so canonicalized data dirs match correctly.
Both fixes are no-ops on Linux/macOS where `canonicalize()` never
produces the `\\?\` prefix.
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Design at: https://github.com/openobserve/o2-enterprise/issues/2112
## What changed
Three changes to cut per-round compactor database load on clusters with
many streams (with `ZO_COMPACT_INTERVAL=2` and 42k streams, each round
fired ~84k db calls):
1. **Retention generation moved to its own schedule.** New config
`ZO_COMPACT_DATA_RETENTION_INTERVAL` (default 3600s). Retention job
generation runs `get_min_date` (`SELECT MIN(date) FROM file_list WHERE
stream = ?`) for every stream, but retention works at day granularity,
so running it every `ZO_COMPACT_INTERVAL` (commonly 2s) wasted enormous
db traffic.
2. **Stream-stats gate before `get_min_date`.** Retention generation now
checks the in-memory stream stats cache first: if the cached
`doc_time_min` is newer than the stream's retention boundary, the stream
has no expired data and is skipped with zero db calls. Since
`delete_by_date` already refreshes `stream_stats` right after deletion,
a stream only pays the `get_min_date` query when it actually crosses the
retention boundary (~once per day). `doc_time_min == 0` (no stats yet)
falls through to the db check for safety.
3. **Unlocked existence probe in `get_pending_jobs` (postgres).** All
compactor nodes serialize on the same `pg_advisory_xact_lock` every
round. A cheap `SELECT id FROM file_list_jobs WHERE status = 0 LIMIT 1`
(index-only via `file_list_jobs_stream_status_idx`) now runs before the
locked transaction, so idle rounds skip the lock and the `GROUP BY
stream` scan entirely.
## Net effect
Steady-state retention traffic drops from ~O(streams) `MIN(date)`
queries every few seconds to ~1 query per stream per day, and idle merge
rounds no longer contend on the global advisory lock.
## Notes for reviewers
- Retention job **execution** (the per-round meta-table pass that picks
up user-triggered stream deletes) moved to the hourly schedule together
with generation, so a user-initiated stream delete can now take up to
`ZO_COMPACT_DATA_RETENTION_INTERVAL` to be physically executed. The
stats gate makes generation cheap, so this interval can be shortened if
that delay matters.
- SQLite backend is unchanged for `get_pending_jobs` — it is single-node
with an in-process mutex, so there is no lock contention to avoid.
## Background
In a super-cluster query, the cluster leader sends the query to region
leaders. Each region leader then uses `RemoteScanExec` to query data
nodes inside its region.
When the source physical plan is `AggregateExec(mode = Partial)`, each
data node returns partial aggregate results. Previously, the region
leader only wrapped the plan with `RemoteScanExec`, so partial results
from multiple data nodes were sent up directly to the cluster leader.
This change adds a region-level `PartialReduce` around `RemoteScanExec`,
so each region can reduce its own partial aggregate results before
returning them.
## Before
For non-sorted aggregate plans:
```text
Cluster Leader
└── ... upper aggregate / merge ...
Region Leader / Follower
└── RemoteScanExec
└── remote plan sent to region data nodes:
└── AggregateExec(mode = Partial)
└── scan / filter / projection ...
```
In this shape, rows with the same group key can still appear multiple
times in one region, because the region leader does not reduce the
remote partial outputs.
## After
With `GROUP BY`:
```text
Cluster Leader
└── ... upper aggregate / merge ...
Region Leader / Follower
└── AggregateExec(mode = PartialReduce)
└── RepartitionExec(Hash(group columns), same partition count)
└── RemoteScanExec
└── remote plan sent to region data nodes:
└── AggregateExec(mode = Partial)
└── scan / filter / projection ...
```
Without `GROUP BY`:
```text
Cluster Leader
└── ... upper aggregate / merge ...
Region Leader / Follower
└── AggregateExec(mode = PartialReduce)
└── CoalescePartitionsExec
└── RemoteScanExec
└── remote plan sent to region data nodes:
└── AggregateExec(mode = Partial)
└── scan / filter / projection ...
```
In short:
```text
Before:
RemoteScanExec(PartialAgg)
After:
PartialReduce(Repartition/Coalesce(RemoteScanExec(PartialAgg)))
```
## Sorted Plans
Plans rooted at `SortPreservingMergeExec` keep the existing shape:
```text
SortPreservingMergeExec
└── RemoteScanExec
└── original ordered physical plan
```
This avoids changing the semantics of ordered query paths such as `ORDER
BY` and
`ORDER BY LIMIT`.
## Implementation
The reusable wrapping logic is in:
```text
src/service/search/datafusion/optimizer/physical_optimizer/remote_scan.rs
```
```rust
pub(crate) fn wrap_partial_reduce_for_aggregate(
partial_reduce_enabled: bool,
agg: &AggregateExec,
input: Arc<dyn ExecutionPlan>,
) -> Result<Arc<dyn ExecutionPlan>>
```
The super-cluster follower integration is in:
```text
src/service/search/super_cluster/follower.rs
```
```text
if physical_plan is SortPreservingMergeExec:
keep SortPreservingMergeExec(RemoteScanExec(...))
else:
remote_scan = RemoteScanExec(physical_plan)
if physical_plan is AggregateExec(mode = Partial or PartialReduce):
return PartialReduce(remote_scan)
else:
return remote_scan
```
## Benefit
Each region now performs a local partial reduce before sending results
to the cluster leader. This reduces intermediate aggregate output across
regions while leaving sorted query plans unchanged.
for metrics data, we only generate vortex file in the compactor, but
others we generate parquet file on ingester and compactor. becuase
metrics have many small files
Currently if map group to role is true, and custom parser is not
enabled, o2 still goes for org mapping (removing any extra org not part
of dex claim) which is not expected behaviour.
This pr fixes this.
Design at: #12995
## Summary
On k8s, when the user only sets a memory limit (no CPU limit), we detect
the whole node's CPU cores via sysinfo — e.g. 64 cores on a node while
the pod only has 8GB memory. All cpu-derived thread pools
(`http_worker_num`, `query_thread_num = cpu * 4`, file
download/move/merge threads, etc.) then get sized for 64 cores against
an 8GB memory budget, causing memory pressure and OOM.
## Changes
- In `check_limit_config`, clamp `limit.cpu_num` by the memory limit:
`cpu_num = real_cpu_num.min(mem_total / 1GB).max(2)`.
- If the memory limit is unknown (0), use the CPU limit directly.
- `real_cpu_num` keeps the true detected value; `get_cpu_limit()` /
cgroup detection is untouched.
Example: 64-core node + 8GB memory limit → `cpu_num = 8`, so
`query_thread_num` becomes 32 instead of 256.
## Summary
Fixes a PostgreSQL type mismatch in compactor fast mode when
`ZO_LOCAL_MODE=true` is used with `ZO_META_STORE=postgres`.
The fast mode query returned `0 as num`, which PostgreSQL infers as
`INT4`. The result is decoded into `MergeJobPendingRecord.num` as `i64`
/ `INT8`, causing sqlx to fail with:
```text
ERROR openobserve::job::compactor: [COMPACTOR::JOB] run data merge
error: SqlxError# error occurred while decoding column "num": mismatched
types; Rust type `i64` (as SQL type `INT8`) is not compatible with SQL
type `INT4`
```
This changes the literal to 0::BIGINT AS num.
## Reproduction
Tested with:
- ZO_META_STORE=postgres
- ZO_LOCAL_MODE=true
- ZO_COMPACT_ENABLED=true
- ZO_COMPACT_INTERVAL=2
On openobserve/openobserve:v0.92.0-rc1, inserting a pending row into
file_list_jobs reproduces the error repeatedly and leaves the job stuck
at status = 0.
I've also created an issue on this #13007
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## What
Fixes several issues in PostgreSQL `file_list` partition maintenance.
- **Fixed daily schedule (P1):** maintenance now runs once per day at a
fixed UTC hour (`ZO_FILE_LIST_MAINTENANCE_HOUR`, default `4`) instead of
`boot_time + N×24h`. The old `retention_allowed_hours` gate — which
almost never matched the random run time, so drop/reindex effectively
never ran — is removed; pre-create + drop + reindex now always run each
cycle.
- **Single-node execution + failover (P2/P3):** the task now runs only
on compactor nodes and uses a distributed lock plus a
`/file_list/maintenance/last_run` marker (replacing the per-run PG
advisory lock). This guarantees exactly-once-per-day even when nodes
wake at slightly different times, with automatic failover if the
previous runner is down.
- **`file_list_dump_stats` gated on dump flag (P5):** the table and its
partitions are no longer created/pre-created when
`ZO_COMPACT_FILE_LIST_DUMP_ENABLED=false`.
## Notes
- **P4 (delayed events → DEFAULT):** confirmed a non-issue in auto mode
— partitions are created on demand and DEFAULT is never dropped. No
change.
- **P6 (manual mode + `retention_allowed_hours`):** out of scope,
tracked separately.
- `retention_allowed_hours` is now decoupled from file_list maintenance
and keeps its existing meaning for the retention job only.
## Config
| Env var | Default | Change |
|---|---|---|
| `ZO_FILE_LIST_MAINTENANCE_HOUR` | `4` | new — UTC hour for daily
maintenance |
| `ZO_COMPACT_FILE_LIST_DUMP_ENABLED` | `false` | now also gates dump
table creation |
## Testing
- `cargo check` / `cargo clippy` on `infra` + `config`: clean.
- Added unit tests for the schedule/day helpers; full non-DB test module
passes.
Design doc: `docs/design/file-list-partition-maintenance.md`
related to https://github.com/openobserve/o2-enterprise/issues/1860
- [x] limit parallel by the target partitions
## Summary
This PR improves Flight `do_get` response throughput by executing and
encoding output partitions in parallel instead of funneling everything
through a single sequential stream/encoder path.
## Changes
- Switches Flight `do_get` from `execute_stream` to
`execute_stream_partitioned`.
- Adds per-partition encoder tasks that coalesce RecordBatches, encode
Arrow IPC, and apply ZSTD compression independently.
- Splits Flight encoding into:
- `ChunkCoalescer` for 8192-row batch coalescing.
- `FlightDataEncoder` for IPC/ZSTD encoding of completed chunks.
- Adds `PartitionEncoderStream` to handle per-partition batch coalescing
and encoding.
- Preserves custom Flight messages behavior:
- early-emitted messages are sent before the first data message;
- stats/metrics/peak-memory messages are sent after data completes.
- Uses a fresh dictionary tracker per encoded chunk so
dictionary-encoded columns remain decodable when chunks are encoded
independently.
- Cleans up stream lifecycle handling by aborting partition tasks before
clearing session resources.
## Performance
100M records
single node optimzier disable(to simulate the distribute query)
### Q1
```sql
SELECT
trace_id,
count(*) AS cnt,
max(_timestamp) AS ts
FROM
"k8s_logs"
GROUP BY
trace_id ORDER BY cnt DESC LIMIT 100
```
12M trace_id unique value
12M records send from follower to leader
main branch: 6s
this branch: 3s
### Q2
```sql
SELECT count(distinct request_id) FROM "k8s_logs"
```
100M request_id unique value
100M records send from follower to leader
main branch: 30s
this branch: 11s
### Q3
```sql
SELECT
kubernetes_namespace_name,
COUNT(DISTINCT trace_id) AS cnt
FROM
k8s_logs
GROUP BY
kubernetes_namespace_name ORDER BY cnt DESC
```
10 kubernetes_namespace_name unique value, 12M trace_id unique value
63M records send from follower to leader
main branch: 21s
this branch: 8.3s
### Q4
```sql
SELECT
host,
COUNT(DISTINCT trace_id) AS cnt
FROM
k8s_logs
GROUP BY
host ORDER BY cnt DESC
```
2160 host unique value, 12M trace_id unique value
99.57M records send from follower to leader
main branch: 56s
this branch: 20s
Design at: #12894
## What changed
- Added `ZO_FEATURE_DEFAULT_INDEX_FIELDS_ENABLED` (default `true`). When
set to `false`, the built-in default fields are disabled for:
- full text search (`log`, `message`, `msg`, `content`, `data`, `body`,
`json`, `error`, `llm_input`, `llm_output`)
- secondary index (`trace_id`, `service_name`, `operation_name`)
- distinct values (`service_name`, `operation_name`)
Only the fields from the `*_EXTRA_FIELDS` ENVs and per-stream settings
are used, so the extra-fields ENVs become full replacements for the
defaults.
- Added `ZO_FEATURE_BLOOM_FILTER_EXTRA_FIELDS` and deprecated
`ZO_BLOOM_FILTER_DEFAULT_FIELDS`. The old ENV is still honored: it is
merged into the new one at startup with a deprecation warning, and will
be removed in v1.0.0.
## Why
Users with high log volume pay ingestion/compaction/storage costs for
tantivy indexes on default FTS fields they never query, with no way to
opt out (see #12894). This is especially bad for CJK content where
single-character tokens make the inverted index a bottleneck instead of
an accelerator.
## Example: fully custom default fields
```
ZO_FEATURE_DEFAULT_INDEX_FIELDS_ENABLED=false
ZO_FEATURE_FULLTEXT_EXTRA_FIELDS=f1,f2
ZO_FEATURE_INDEX_EXTRA_FIELDS=f1,f2
ZO_FEATURE_BLOOM_FILTER_EXTRA_FIELDS=""
```
## Notes for reviewers
- All changes are in `src/config/src/config.rs`: the gating happens
inside the `SQL_FULL_TEXT_SEARCH_FIELDS` /
`SQL_SECONDARY_INDEX_SEARCH_FIELDS` / `DISTINCT_FIELDS` lazy statics, so
every consumer (`get_stream_setting_*` in infra/schema, ingestion,
compaction, UDS auto-enable keep-list, and the `/config` API's
`default_fts_keys` / `default_secondary_index_fields`) follows
automatically.
- `get_stream_setting_index_updated_at_for_fields` also follows: with
defaults disabled, those fields no longer get the `BASE_TIME` "indexed
since forever" treatment.
- Bloom filter has no hardcoded defaults, so the flag does not gate it;
the ENV rename just makes its naming consistent with the other
extra-fields ENVs.
- Added unit tests for the bloom ENV migration (migrate, merge when both
set, no-op).
Closes#12894
## Fix PagerDuty prebuilt destination
### Problem
PagerDuty Events API v2 requires the integration key as `routing_key`
**in the JSON body**, but it never got there:
- Template used `{integration_key}` — not a real server variable → sent
literally → PagerDuty rejects payload. `severity`/`source` hardcoded.
- UI stored integration key only in an `X-Routing-Key` header, which
PagerDuty ignores.
- Severity stored as `credential_severity`, never referenced by the
template.
- "Test" passed only via client-side substitution, diverging from the
real server send path → Test could pass while real alerts failed.
### Fix
Route everything through the alert engine's existing endpoint-metadata
substitution so real sends and Test/Preview render identical, valid
payloads.
- **Template body**: corrected to PagerDuty v2 shape with substitutable
`{routing_key}`, `{severity}`, `{source}` (`prebuilt-destinations.json`,
Rust fallback, web `pagerduty.ts`).
- **Fields now used**: new `generatePrebuiltMetadata()` writes
`routing_key` (=integration key), `severity`, `source` into destination
metadata on save → server fills them at send time. Dropped useless
`X-Routing-Key` header; edit-restore reads key from metadata (legacy
header kept as fallback).
- **Test uses real template**: Test/Preview sample data now supplies
`routing_key`/`source` → client renders the same body the server sends.
- **Stale templates**: `ensure_system_templates()` previously skipped
already-existing prebuilt templates → existing installs kept the broken
body. Now refreshes system-managed templates whose stored body has
drifted from shipped source.
### Verification
- Rust: `config` + `openobserve` binary compile; prebuilt loader tests
pass (10/10). Added `PartialEq`/`Eq` to `TemplateType` for drift check.
- Updated header unit test + added `generatePrebuiltMetadata` tests. JS
suite not runnable locally (pre-existing `ERR_REQUIRE_ESM` in vitest
setup, unrelated) — run in CI.
### Note
Out of scope: Test always uses `prebuilt_pagerduty` template name, not a
per-destination custom template override.
related to https://github.com/openobserve/o2-enterprise/issues/1860
when we use the do_get, the recordbatch already compression by the zstd,
we do not need compression again
## New env
```
ZO_GRPC_HTTP2_ADAPTIVE_WINDOW=false
```
## Performance
100M records
single node optimzier disable(to simulate the distribute query)
### Q1
```sql
SELECT
trace_id,
count(*) AS cnt,
max(_timestamp) AS ts
FROM
"k8s_logs"
GROUP BY
trace_id ORDER BY cnt DESC LIMIT 100
```
12M trace_id unique value
12M records send from follower to leader
main branch: 12s
this branch: 6s
### Q2
```sql
SELECT count(distinct request_id) FROM "k8s_logs"
```
100M request_id unique value
100M records send from follower to leader
main branch: 74s
this branch: 30s
### Q3
```sql
SELECT
kubernetes_namespace_name,
COUNT(DISTINCT trace_id) AS cnt
FROM
k8s_logs
GROUP BY
kubernetes_namespace_name ORDER BY cnt DESC
```
10 kubernetes_namespace_name unique value, 12M trace_id unique value
63M records send from follower to leader
main branch: 48s
this branch: 21s
### Q4
```sql
SELECT
host,
COUNT(DISTINCT trace_id) AS cnt
FROM
k8s_logs
GROUP BY
host ORDER BY cnt DESC
```
2160 host unique value, 12M trace_id unique value
99.57M records send from follower to leader
main branch: 104s
this branch: 56s
related to: https://github.com/openobserve/o2-enterprise/issues/1860
related to: https://github.com/openobserve/openobserve/pull/12418
## Performance
100M records
single node optimzier disable(to simulate the distribute query)
### Q1
```sql
SELECT
trace_id,
count(*) AS cnt,
max(_timestamp) AS ts
FROM
"k8s_logs"
GROUP BY
trace_id ORDER BY cnt DESC LIMIT 100
```
12M trace_id unique value
12M records send from follower to leader
main branch: 20s
this branch: 12s
### Q2
```sql
SELECT count(distinct request_id) FROM "k8s_logs"
```
100M request_id unique value
100M records send from follower to leader
main branch: 77s
this branch: 74s
### Q3
```sql
SELECT
kubernetes_namespace_name,
COUNT(DISTINCT trace_id) AS cnt
FROM
k8s_logs
GROUP BY
kubernetes_namespace_name ORDER BY cnt DESC
```
10 kubernetes_namespace_name unique value, 12M trace_id unique value
63M records send from follower to leader
main branch: 54s
this branch: 48s
### Q4
```sql
SELECT
host,
COUNT(DISTINCT trace_id) AS cnt
FROM
k8s_logs
GROUP BY
host ORDER BY cnt DESC
```
2160 host unique value, 12M trace_id unique value
99.57M records send from follower to leader
main branch: 112s
this branch: 104s
disable partial reduce: 88s
## Summary
Improve `PartialReduce` execution parallelism in remote scan aggregation
plans.
For grouped aggregations, `PartialReduce` now hash repartitions partial
aggregate output by the group keys instead of always coalescing all
partitions into one. This keeps the reduce stage parallel while ensuring
rows with the same group key are routed to the same partition.
## ExecutionPlan change
Before:
```text
AggregateExec: mode=PartialReduce
CoalescePartitionsExec
AggregateExec: mode=Partial
...
```
After, for `GROUP BY`:
```text
AggregateExec: mode=PartialReduce
RepartitionExec: partitioning=Hash([group_keys...], N)
AggregateExec: mode=Partial
...
```
For no `GROUP BY`, the plan still falls back to:
```text
AggregateExec: mode=PartialReduce
CoalescePartitionsExec
AggregateExec: mode=Partial
...
```
## Changes
- Reuse `GROUP BY` column expressions as hash repartition keys before
`PartialReduce`.
- Preserve the partial aggregate output partition count for grouped
partial reduce.
- Keep `CoalescePartitionsExec` behavior for global aggregations without
group keys.
- Add unit tests for grouped hash repartition and no-group-by coalesce
fallback.
## Testing
Added tests in `remote_scan.rs` covering the new `PartialReduce`
planning behavior.
## Problem
Filebeat (and other Elasticsearch clients) can't ingest using an org
ingestion token (`o2oi_…`). On startup Filebeat's `eslegclient` pings
the root URL `GET /api/{org_id}/` to detect the ES version. That request
**403s** (`Unauthorized Access`), so Filebeat never gets past the
handshake and loops in backoff-retry, shipping nothing — even though
`POST /api/{org_id}/_bulk` works fine.
## Root cause
The org-root handler is registered as the slash-mandatory route
`/{org_id}/`, so every request to it carries a trailing slash. In
`AuthExtractor::from_request_parts`, `"default/".split('/')` yields
`["default", ""]` (len 2).
The ES-compat bypass entry in `ROUTE_PERMISSIONS` is `path: &["{org}"]`
(len 1, `bypass: true`), and `path_matches` requires an exact length
match (`si == path.len()`). The trailing empty segment makes the length
2 ≠ 1 → no match → the "route missing" fallback sets `bypass_check =
false` with an empty `o2_type` → a roleless ingestion token hits OpenFGA
→ **403**.
Because the route *requires* the trailing slash, this bypass entry was
effectively **dead code** — no request could ever satisfy both the route
and the matcher. Notably `validate_credentials` already strips the
trailing empty segment; `AuthExtractor` did not. That inconsistency was
the bug.
## Fix
Strip a trailing empty path segment in
`AuthExtractor::from_request_parts` before route resolution, mirroring
`validate_credentials`:
```rust
let mut path_columns = path.split('/').collect::<Vec<&str>>();
if path_columns.len() > 1
&& let Some(v) = path_columns.last()
&& v.is_empty()
{
path_columns.pop();
}
```
## Blast radius
- Only affects requests that arrive **with a trailing slash**; no-slash
paths split identically → no behavior change.
- `/{org_id}/` is the **only** org-scoped handler in the codebase
registered with a trailing slash; every other endpoint 404s at routing
for trailing-slash requests regardless of auth, so no working (200)
request can break.
- Trailing slash never grants more access than no-slash (verified for
both token and root).
- Guarded `len > 1` → no empty-vec / out-of-bounds risk.
## Verification
Local: enterprise build, OpenFGA enabled, org ingestion token on
`default`.
| Request | Before | After |
|---|---|---|
| token `GET /api/default/` (handshake) | 403 | **200** |
| token `POST /api/default/_bulk` | 200 | 200 |
| token `GET /api/default/streams` (real read API) | 401 | 401 (still
blocked) |
| **Filebeat** (token) | `403 Forbidden` → infinite backoff |
**"Connection established"**, logs ingested |
- router should watch org settings changes
- if a sso parser is selected in the settings, the field should show the
parser function selected even after page refresh. Currently, on page
refresh the select button empties the field even though the sso parser
function is already configured.
<img width="1492" height="806" alt="Screenshot 2026-06-26 at 7 10 42 PM"
src="https://github.com/user-attachments/assets/5d40fb8f-779c-4e0e-81f6-f86359b5056a"
/>
- Adds index only to the next_run_at column to improve the pull method
throughput and time.
- Adds a 80% fillfactor for the postgres table to enhance the HOT% for
updates. One minor caveat is that, since the next_run_at is now indexed,
for rescheduling updates (next_run_at update), it will not be able to
utilize Heap Only Tuples and can add to the index/dead tuple bloating in
the postgres db. The pulls, keep alives and watch timeout can still use
the HOT approach because they don't touch the next_run_at column.
Note in the results that, the adv lock wait time reduced drastically
after using the index.
Another thing to note that there is a difference from the file_list
table where the dead tuples caused bad search performance when there
were huge number of records in the table (1TB+ table size). In that
case, the auto vacuum didn't help much because, the amount of dead
tuples required to trigger auto vacuum, itself was very big - resulting
in bad index bloating. In the scheduled_jobs, in most cases, almost all
the rows are updated frequently, resulting in relatively higher
percentage of dead tuples and hence trigger auto vacuum more frequently.
The above is visible from the below test results as well - for baseline
(current main branch approach), the HOT% is very good (91%) because the
updates didn't touch the indexes at all and hence, low no. of dead
tuples. For index_ff100 (index on next_run_at + default 100%
fillfactor), it already triggers 1 autovacuum. Same for 80% fill factor.
Results (100K rows, 8 clients, 60s)
### Tier 1 — pull query plan
| variant | plan | exec time | shared buffers |
|---------------|-----------------------------------|-----------:|---------------:|
| `baseline` | Parallel Seq Scan + top-N Sort | **13.5 ms**| 4424 |
| `index_ff100` | Index Scan `scheduled_jobs_pull_idx` | **0.039 ms** |
7 |
| `optimized` | Index Scan `scheduled_jobs_pull_idx` | **0.030 ms** | 7
|
→ **~350× faster** at 100K, and the gap **grows with table size** (seq
scan is O(rows), index is
O(log n + limit)). Fillfactor has no effect on the read plan, as
expected.
### Tier 2 — throughput, lock-wait, churn health
| metric (per variant) | `baseline` | `index_ff100` | `optimized` |
|----------------------------------|-----------:|--------------:|------------:|
| **total tps** | 75.7 | **930.6** | 875.3 |
| avg txn latency | 105.6 ms | 8.6 ms | 9.1 ms |
| **advisory-lock wait** (pull) | **177 ms** | **2.4 ms** | 1.7 ms |
| pull claim UPDATE | 26.4 ms | 0.93 ms | 0.80 ms |
| updates in 60s (`n_tup_upd`) | 22.5 K | 278.8 K | 261.9 K |
| **HOT %** (`n_tup_hot_upd`) | 91.3 % | 47.0 % | 50.2 % |
| index size | 18 MB | 28 MB | **25 MB** |
| true dead-tuple % (pgstattuple) | 0.10 % | 0.84 % | **0.11 %** |
| free space % (pgstattuple) | 3.3 % | 9.2 % | **19.1 %** |
| autovacuum runs | 0 | 1 | 1 |
# feat: Dashboard Table Chart — per-column formatting
## 1. Entry point — config panel
Column formatting lives in the panel **Config** panel, under a **Column
Formatting**
section. A **"Configure column formatting"** button opens the **Column
Formatting**
dialog.
<img width="1920" height="907" alt="image"
src="https://github.com/user-attachments/assets/21d05294-3d3e-4f84-893f-cf3b235adf0d"
/>
## 2. Column Formatting dialog — empty state
A 3-pane dialog. When nothing is configured yet it shows an empty state
(`OEmptyState`)
with a single **"+ Add field"** action. **Add field** is a dropdown
listing the panel's
columns (each with a **NUM / TEXT** type badge); the menu matches the
trigger width.
<img width="1920" height="907" alt="image"
src="https://github.com/user-attachments/assets/866fb119-76f3-48b2-98b6-50df24660a8c"
/>
## 3. Column Formatting dialog — editing a field
- **Left** — "+ Add field" + the list of added fields (type badge, name,
hover-to-remove ✕).
- **Middle** — the shared controls for the selected field:
- **Field Type** — `Auto` / `Num` / `Text` (Auto detects from the data;
Num/Text force it).
- **Value Formatting** — unit dropdown (numbers, bytes, durations,
percentage, currency, custom) — numeric columns only.
- **Alignment** — `Auto` / `Left` / `Center` / `Right` (Auto =
type-based default).
- **Styling** — Text color, Background color (curated swatches +
custom), and a Unique-value-color toggle.
- **Conditional Styling** — threshold rules (numeric columns only).
- **Right** — a **live preview** of the column rendered with the current
settings.
<img width="1920" height="907" alt="image"
src="https://github.com/user-attachments/assets/276289d8-f92d-46f2-a413-b58e840e2f6b"
/>
## 4. Conditional styling
Per-column rules read as a sentence: **If value** `[operator]` `[value]`
**then text
color** `[…]` **and background** `[…]`. Operators: `<`, `>`, `<=`, `>=`,
`=`, `!=`.
Add multiple rules — **the last matching rule wins**, so later rules
override earlier
ones (e.g. `> 1000` takes precedence over `> 400` for `2301`).
<img width="1920" height="907" alt="image"
src="https://github.com/user-attachments/assets/b47bde1c-11e4-4ec9-8d6b-03f702ae8902"
/>
**Example** — flag `count` when it's low:
> Conditional Styling → **Add rule** → operator `<`, value `50` → Text
`#b91c1c`, Background `#fef2f2`.
Persists in `override_config` as:
```json
{
"field": { "matchBy": "name", "value": "count" },
"config": [
{
"type": "conditional_styles",
"rules": [
{ "operator": "<", "threshold": 50, "textColor": "#b91c1c", "bgColor": "#fef2f2" }
]
}
]
}
```
## 5. Rendered result in the live table
Cells matching a rule render with the configured colors; non-matching
cells stay
normal. Thresholds may be decimals (e.g. `0.5`).
<img width="1920" height="907" alt="image"
src="https://github.com/user-attachments/assets/4c4146f9-b018-465e-81ae-fc04149cdc83"
/>
---
## Data model
Overrides persist as an array at `config.override_config`; each entry
targets one column
and carries typed config items. The discriminants (`tableConfigUtils.ts`
`OVERRIDE_CONFIG_TYPES`) are kept in sync with the Rust v8 schema
(`src/config/src/meta/dashboards/v8/mod.rs`):
```
field_type · unit · alignment · text_color · background_color · unique_value_color · conditional_styles
```
Unset values use `null` (inherit the panel default); `field_type`
defaults to `auto`.
Backward compatible: dashboards saved before this change load and render
unchanged (the
v8 `Config` enum has no `deny_unknown_fields` and no newly-required
field).
## Architecture
| Piece | File |
| --- | --- |
| Dialog (3-pane) |
`web/src/components/dashboards/OverrideConfigPopup.vue` |
| Shared sections |
`web/src/components/dashboards/ColumnFormatControls.vue` |
| Color picker | `web/src/components/dashboards/ColorSwatchPicker.vue` |
| Model + (de)serialize |
`web/src/composables/dashboard/useColumnFormatting.ts` |
| Parse / apply / format | `web/src/utils/dashboard/tableConfigUtils.ts`
|
| Render (cell styling) |
`web/src/components/dashboards/panels/TableRenderer.vue` |
| Config-panel entry |
`web/src/components/dashboards/addPanel/OverrideConfig.vue` |
| Rust v8 schema | `src/config/src/meta/dashboards/v8/mod.rs` |
## Scope changes (vs the original proposal)
- **Cell types removed** — the `Text / Bar / Spark` toggle, in-cell
**progress bars**,
and **sparklines** were prototyped and then dropped (frontend + the v8
`cell_type`
variant). Formatting is type/unit/alignment/color/conditional only.
- **Inline header popover removed** — formatting is reached via the
config-panel dialog
(no per-header tune-icon popover / "Edit all").
- **Conditional precedence** — last matching rule wins (was
first-match).
## Notable correctness fixes
- **Value-mapping ranges** — numeric values arriving as strings (e.g.
`"5"`) again match
range mappings (restored origin behavior; empty/null cells don't
false-match a
`0`-bounded range).
- **Conditional threshold** — decimal thresholds (e.g. `0.5`)
deserialize correctly on the
Rust side (flexible `OrdF64` deserializer).
- **Reset-on-type-change** — forcing a numeric column to `Text` clears
its now-hidden
unit/conditions (keyed by field, so removing a column never wipes a
different one).
---
# Column Filtering
> Detailed design: [`table-filtering.md`](./table-filtering.md)
In-table, multi-select, spreadsheet-style filtering for table panels.
Each column header
gets a filter button; its dropdown lists that column's **distinct
values** as a checkbox
list — tick/untick to narrow the rows. Filtering is **purely
client-side** (no re-query).
## 1. Enable it
A single **Column Filtering** toggle in the panel **Config** panel
(`config.table_filtering`, an `Option<bool>` in the v8 schema). Off by
default; when on,
the table renderers receive `:enable-filtering` and the header filter
buttons appear.
(Filtering is suppressed in print/export mode.)
<img width="1680" height="907" alt="image"
src="https://github.com/user-attachments/assets/d834a916-c7eb-4dce-9984-95276e996e87"
/>
## 2. Filter a column
Click a header's filter button to open the dropdown:
- **Distinct values** of the column as a checkbox list — pick any
subset.
- **Searchable** — a search box narrows the value list.
- **Clear** a column's filter in one click.
- Values display formatted (units/mappings) but match on the raw value.
- Multiple column filters combine with **AND**; sorting preserves active
filters.
<img width="1680" height="907" alt="image"
src="https://github.com/user-attachments/assets/0671b75e-e049-442f-bc56-53e861b9ff41"
/>
## Architecture
| Piece | File |
| --- | --- |
| Config toggle |
`web/src/components/dashboards/addPanel/ConfigPanel.vue`
(`table_filtering`) |
| Renderer wiring |
`web/src/components/dashboards/PanelSchemaRenderer.vue`
(`:enable-filtering`) |
| Filter UI + logic | `web/src/components/TenstackTable.vue` (gated by
`enableColumnFilter`) |
| v8 schema flag | `src/config/src/meta/dashboards/v8/mod.rs`
(`table_filtering: Option<bool>`) |
Implemented on the same branch as the formatting work;
`enableColumnFilter` is a no-op
when false, so non-table charts and filtering-off panels are unaffected.
</content>
Design At: dashboards/table-enhancement
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
## Problem
The enterprise build is broken on `main` (`error[E0061]`):
```
error[E0061]: this function takes 3 arguments but 2 arguments were supplied
--> src/super_cluster_queue/short_urls.rs:33:13
33 | infra::table::short_urls::add(&short_id, &original_url).await?;
| argument #3 of type `&str` is missing
```
#12829 ("fix: org id as mandatory check") made `org_id` a mandatory
third argument of `infra::table::short_urls::add`, but the
**enterprise-gated** super-cluster queue handler
`super_cluster_queue::short_urls::process` was not updated. This module
is `#[cfg(feature = "enterprise")]`, so OSS-only CI compiled fine while
the enterprise build (and every downstream Playwright shard) failed.
## Fix
The super-cluster queue message (`emit_put_event` → `short_url_put`)
only carries the `original_url` over the wire — `org_id` is never
published to the queue. So the receiver passes an empty `org_id`.
This matches the legacy semantics introduced in #12829, where
`short_urls::get` accepts empty-org rows from any org:
```rust
infra::table::short_urls::add(&short_id, &original_url, "").await?;
```
## Note / possible follow-up
With an empty `org_id`, short URLs replicated **across a super cluster**
are stored org-agnostic (accepted from any org on lookup), the same as
pre-existing legacy rows. If org-scoped short URLs need to survive
super-cluster replication, a follow-up should thread `org_id` through
the queue message value (producer `emit_put_event` and the
`short_url_put` payload) — a larger change touching the wire format.
This PR only unblocks the build with behavior consistent with #12829's
design.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This attempts to refactor the process_node fn to make it more readable
and change-able.
The commits should be reviewed individually, first extracts individual
match arms into their own fns, second does cleanup of fn params by
extracting them into structs.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Currently in pipeline execution, if any null record is encountered, it
results in errors. Instead it should just ignore the null records and
continue. It is very common in pipeline vrls to use -`. = null` when
certain condition matches.
Currently if you try to self consume the histogram metrics emitted by
o2, it fails with the below message -
```
ERROR openobserve::job::promql_self_consume: error in ingesting self-metrics : missing value
```
This pr fixes this .
- Add logs to analyze the time taken by pipelines (can be turned on by
`ZO_PRINT_KEY_EVENT=true`)
- Add pipeline processing time to the usage reporting
- Introduce new metrics (`zo_pipeline_exec_batch_size` and
`zo_pipeline_exec_time_ms`) for time taken by pipeline.
## Summary
Eliminates 1 of 3 redundant \SearchService::cache::search\ calls per
trace request by merging the Q2a (per-trace aggregate fields) SQL
directly into Q1.
### Changes
- **\uild_llm_trace_query\**: Added \extra_selects: &str\ param — Q2a
fields injected before \FROM\ for both regular and LLM trace queries
- **\get_latest_traces\**: Moved schema-dependent expression computation
before Q1 SQL building; extended Q1 with Q2a columns (span_count,
error_count, max_duration, service_count, root_service_name,
first_service_name, first_operation_name); removed entire Q2a search
block (~200 lines)
- **\process_latest_traces_stream\**: Same Q1 extension; added
early-break after 3 consecutive empty partitions for absolute time
ranges
- **\ un_q2_for_traces\**: Removed Q2a search call; populates all fields
directly from Q1 hits; keeps Q2b (service breakdown) for multi-service
traces
### Before (3 search calls per request)
1. Q1 — basic trace list
2. Q2a — \ race_id IN (...)\ for span aggregates (removed)
3. Q2b — \ race_id IN (...)\ for service breakdown
### After (2 search calls, or 1 for single-service traces)
1. Q1 — trace list with all aggregates
2. Q2b — service breakdown (multi-service only)
### Risk
Low. The removed Q2a code was entirely internal — column aliases like
\min_start_time\/\max_end_time\ had zero external references.
\TraceResponseItem\ struct unchanged. All fields now populated from Q1
with equivalent values.
---------
Co-authored-by: Giriraj Bidwai <girirajbidwai@users.noreply.github.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Summary
Opening a `.ttv` (puffin) inverted-index file on a footer-cache **miss**
previously issued **two** sequential ranged reads against object
storage:
1. read the last 12 bytes (footer tail) to learn `payload_size`, then
2. read the payload region (`HeadMagic + JSON`).
This mirrors the one-shot tail optimization already used for
bloom-filter (`.bf`) files: pull a single fixed-size suffix that almost
always covers the whole footer region, eliminating the first round-trip.
## Why 4 KB
A `.ttv` is always a **single-segment** tantivy index, so its footer is
a small uncompressed JSON over a **fixed ~6-blob** set (`meta.json`, the
`.term/.idx/.pos/.fast` segment files, and `footer_cache`; at most ~9
even with `.store/.fieldnorm/.del`). The footer size does **not** grow
with the number of indexed fields.
Measured across 195 real `.ttv` files (4 KB – 176 MB):
| metric | min | p50 | p99 | max |
|---|---|---|---|---|
| footer payload JSON | 949 B | 1013 B | 1124 B | 1139 B |
| one-shot tail needed | 965 B | 1029 B | 1140 B | 1155 B |
| blobs / file | 6 | 6 | 7 | 7 |
So **4 KB** covers the entire footer region in one read with ~2×
headroom over the structural ceiling (~2 KB), while being far smaller
than the bloom filter's 16 KB (whose footer scales with group
cardinality, unlike here).
## Behavior
- **Common case:** 1 ranged read instead of 2 → saves ~one RTT per cold
footer open. Queries scanning many partitions save `N × RTT`.
- **Rare larger footer:** re-reads exactly the footer region, so the
worst case is no worse than before (still 2 reads).
- **Cache unchanged:** only the exact footer region (not the full 4 KB
probe) is stored in `FOOTER_DATA_CACHE`, so cache memory is identical to
today.
- **Footer-cache hit path:** untouched.
## Tests
- `cargo test -p tantivy_utils puffin` — 153 passed, incl. new
`test_slice_payload_from_suffix_roundtrip` covering both the single-read
fast path and the short-probe fallback against a real in-memory puffin
file.
- `cargo clippy -p tantivy_utils --tests` — clean.
- Verified working locally on real `.ttv` data.
udev
Huaijin
Hengfei Yang
Subhra264
Ashish Kolhe
tsushanthandHengfei Yang
Rafael MendonçaandHengfei Yang
Abhay Padamaniandgithub-actions[bot]
Harsh Mahajan