## Summary
Removes all `crate::handler::*` dependencies from `src/service` and
`src/common`, in preparation for splitting the codebase into separate
crates. Code that both layers need is moved down into
`common`/`service`, and the old handler locations keep `pub use`
re-exports so call sites stay unchanged.
After this PR, `src/service` and `src/common` contain zero references to
`crate::handler`.
## Move map
| Item | Old location (now a re-export) | New location |
|---|---|---|
| `MetadataMap` (OTel extractor) | `handler/grpc/mod.rs` |
`common/meta/grpc.rs` (new) |
| `CONTENT_TYPE_JSON` / `CONTENT_TYPE_PROTO` |
`handler/http/request/mod.rs` | `common/meta/http.rs` |
| `ERROR_HEADER` | `handler/http/router/mod.rs` | `common/meta/http.rs`
|
| `get_user_email_from_auth_str` / `get_user_details` |
`handler/http/auth/validator.rs` | `common/utils/auth.rs` |
| `get_user_name_from_token` | `handler/http/auth/token.rs` |
`common/utils/jwt.rs` |
| `check_permissions` / `list_objects_for_user` |
`handler/http/auth/validator.rs` | `service/authz.rs` (new) |
| `map_error_to_http_response` |
`handler/http/request/search/error_utils.rs` | `service/http.rs` (new) |
| `Searcher` | `handler/grpc/request/search/mod.rs` |
`service/search/searcher.rs` (new) |
| `get_cluster_metrics` / `get_peak_memory_from_ctx` |
`handler/grpc/flight/visitor.rs` |
`service/search/datafusion/plan_metrics.rs` (new) |
| `GetOrgUsageResponseBody` / `OrgUserData` |
`handler/http/models/billings.rs` | `service/org_usage.rs` |
| `CreateAnomalyConfigRequest` / `UpdateAnomalyConfigRequest` |
`handler/http/request/anomaly_detection/mod.rs` |
`service/anomaly_detection.rs` |
| `BackfillRequest` | `handler/http/request/pipelines/backfill.rs` |
`service/alerts/backfill.rs` |
| LLM schema-compat constants |
`handler/http/request/traces/schema_compat.rs` |
`service/traces/schema_compat.rs` (new) |
## Notes
- Pure code movement; no functional changes intended. Two incidental
cleanups: `strip_prefix(...).unwrap()` replaced with `?` in the moved
auth helpers (removes panic paths), and `list_objects_for_user` now
returns `anyhow::Result` instead of `AuthError` (call sites still
respond 403; only the error message text loses the `Forbidden:` prefix).
- Existing unit tests moved along with the code (`Searcher`,
`error_utils`, `visitor`).
- `common/utils/auth.rs` now calls `service::authz` instead of
`handler::...::validator` — `common → service` dependencies already
existed, so this doesn't add a new layering edge.
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Summary
Improves PromQL query performance and memory usage by reducing metrics
label allocations and avoiding unnecessary global map construction.
- Closes https://github.com/openobserve/o2-enterprise/issues/2151
## Performance
```
histogram_quantile(
0.9,
sum by(le, path) (
rate(codelab_api_request_duration_seconds_bucket[5m])
)
)
```
3.6M series, 10 minute, 30s step
main branch with cache: 3.8s
this branch: 3.3s
## Changes
- Keep samples and labels partitioned by series hash throughout the
DataFusion loading pipeline.
- Attach labels directly to the corresponding metrics partition in
parallel.
- Flatten already-deduplicated partitions on the common single-context
path, while preserving hash-based merging for multi-context and
super-cluster queries.
- Replace the process-wide series label cache with a bounded,
query-local label interner:
- Reuses `Arc<Label>` values for label columns with sufficient
repetition.
- Automatically disables interning for low-reuse/high-cardinality
columns.
- Limits retained values to keep memory usage bounded.
- Build `RangeValue` objects directly while loading samples and
exemplars, avoiding intermediate maps and allocations.
- Compute aggregation grouping signatures directly from projected labels
without cloning and filtering label vectors.
- Use advancing sample-window indices for range functions instead of
repeated binary searches.
- Remove the obsolete label-cache configuration and cache hit/miss
metrics.
## Operational notes
The following are removed because the process-wide label cache no longer
exists:
- `ZO_METRICS_LABEL_CACHE_MAX_SIZE`
- `query_metrics_label_cache_hit_count`
- `query_metrics_label_cache_miss_count`
Deployments, dashboards, or alerts using these settings or metrics
should be updated.
## Testing
Added and updated tests covering:
- Partitioned sample and label loading.
- UInt64 and string series hashes.
- Label deduplication and first-row selection.
- Adaptive label interning behavior.
- Single- and multi-context metric merging.
- Aggregation signature equivalence.
- Advancing range-function sample windows, including duplicate
timestamps and empty windows.
Backend unit, API integration/regression, metrics E2E, and enterprise
build checks pass.
## What
Adds backfill support for pipelines whose only destination is a **remote
destination**.
## Why
Backfilling a remote-only pipeline failed at runtime:
`get_destination_stream_from_pipeline` only recognized local `Stream`
nodes, so a pipeline with only a `RemoteStream` destination returned "No
destination streams found". That error path **deleted the scheduler
trigger** while keeping the `backfill_jobs` config row, which then
surfaced as:
- `GET .../backfill/{job_id}` returning **404** (`DbError# key
<org>/backfill/<job_id> does not exist`)
- the backfill row **disappearing from the UI** on the next refresh
Remote destinations were never actually unsupported by execution —
`ExecutablePipeline::process_batch` (already called by the backfill
handler) routes remote records to the remote WAL. The pipeline was just
being rejected by an over-strict pre-execution gate before it could run.
## Changes
- `get_destination_stream_from_pipeline` now returns an empty list (not
an error) for a remote-only pipeline, and only errors when there is no
destination leaf of either kind. Remote records continue to be handled
inside `process_batch`.
- Added a defensive guard so `delete_before_backfill` is skipped (with a
clear log) when a pipeline has no local destination streams — the
edit-after-create case. Creation already rejects
`delete_before_backfill` for remote pipelines with a user-facing
message.
- Added unit tests for remote-only (→ `Ok(empty)`) and mixed
local+remote (→ locals only) pipelines.
## Testing
- `cargo test -p openobserve get_destination_stream` → 8 passed.
- ⚠️ Enterprise build not compiled locally (private `o2_enterprise`
crate unavailable). The remote-WAL write path should be smoke-tested on
an enterprise build with a real remote-only pipeline backfill before
merge.
## Out of scope
The pre-existing trigger-orphaning inconsistency (get 404s while list
silently skips when a trigger is missing) is not addressed here.
design at: cloud/org%20deletion/org-deletion-design.md
## Summary
Self-service cloud org deletion with a **recoverable grace period**,
followed by complete, retryable, auditable hard cleanup.
**Lifecycle:** `active → pending_deletion → deleting → gone`
- **Delete** soft-deletes into `pending_deletion` — data intact, org
hidden from the switcher + all API access blocked, `deleted_at`
recorded. With `grace=0` it deletes immediately (legacy path).
- A **promotion scheduler** flips expired pending orgs to `deleting`
after the grace window; the background **cleanup worker** then tears
down all data across object store, file_list, DB, OpenFGA, and billing —
per-step tracked and retried (`MAX_ATTEMPTS`).
- **Resurrect** (`_meta`/root only) flips `pending_deletion` back to
`active` — no data was ever touched.
## Key changes
- New `org_cleanup_tasks` ledger table +
`organizations.status`/`deleted_at` columns (2 migrations);
`DB_SCHEMA_VERSION` 47→48.
- `OrgStatus::PendingDeletion` cache + coordinator events + watch;
`is_blocked` middleware hides & blocks pending/deleting orgs.
- Grace window from enterprise `O2_ORG_DELETION_GRACE_PERIOD_DAYS`
(default 30), read from OSS behind `#[cfg]`.
- Cleanup worker + promotion scheduler run on **compactor** nodes;
dist-lock + atomic CAS make execution exactly-once cluster-wide. Task
rows purged immediately on hard-delete completion and on resurrect (no
retention sweep).
- Audit-trail entry to `_meta` on every status transition
(fire-and-forget).
- Resurrect endpoint (dual-impl, cloud-gated) + list surfaces
`deleted_at`/grace days.
- UI: pending-deletion badge + countdown + resurrect action,
deletion-progress dialog, Danger Zone restyle with type-to-confirm.
## RBAC
In-handler role checks (no OFGA middleware): **delete** = root OR org
Admin; **resurrect** = root only via `_meta`.
## Companion PR
Enterprise side: `openobserve/o2-enterprise` #2128 (config + ofga +
cloud billing).
## Testing
Unit suites pass; full flow live-verified end-to-end (soft-delete →
resurrect with data intact, grace=0 immediate delete, audit entries,
RBAC 403s).
> Requires ≥1 compactor node running for deletions to progress (always
true for single-binary / `ZO_NODE_ROLE=all`).
## Summary
Follow-up to #13109. Two aggregation-side bottlenecks showed up at high
cardinality (3.17M series), both single-threaded loops that scale with
series count:
1. **Parallel group signature computation** — `eval_aggregate` computed
each series' grouping signature (clone projected labels + hash)
sequentially: 1.17s of a ~3.8s `sum by(le, path)` query. Now fanned out
with rayon (~95ms); `group_series_by_labels` (used by `topk`/`bottomk`)
gets the same treatment.
2. **Parallel chunked accumulation for huge groups** — an aggregation
without a modifier (`sum(rate(m[5m]))`) puts every series into a single
group, so the "parallel per-group" aggregation ran on one thread: ~250ms
for 66M samples at a 15-min window, and 2-3s at a 1-hour window. Large
groups (≥64k series) are now split into 32k-series chunks aggregated in
parallel and merged via a new `Accumulate::merge` (implemented by all 9
accumulators; equivalence covered by a unit test).
Plus a refactor commit: `eval_aggregate` had reimplemented
`group_series_by_labels` inline just to keep per-series projected labels
it only needed once per group — both now share `projected_labels()`, and
a 3.17M-entry `(hash, Labels)` side vector (and its drop) disappears.
## Benchmarks
Same setup as #13109: single node (M-series Mac, 32GB), 1.34B rows /
3.17M series histogram bucket stream, result cache disabled, warm = mean
of runs 2-4.
**15-minute window:**
| Query | main (#13109) | this PR | speedup |
|---|---|---|---|
| `histogram_quantile(0.9, sum by(le,path)(rate(m[5m])))` | 3.81s |
3.02s | **1.26x** |
| `count by(path) (rate(m[5m]))` | 3.54s | 2.86s | **1.24x** |
| `sum by(region) (rate(m[5m]))` | 3.40s | 2.83s | **1.20x** |
| `max(rate(m[5m]))` | 2.60s | 2.34s | 1.11x |
| `sum(rate(m[5m]))` | 2.69s | 2.46s | 1.09x |
| `topk(5, rate(m[5m]))` | 10.37s | 10.39s | parity |
| selective filter (132k series) | 0.46s | 0.45s | parity |
**1-hour window (1.2B samples):** `sum(rate(m[5m]))` 12.1s → 8.6s
(**1.4x** — the single-threaded accumulation scales with samples, so the
fix matters more at longer ranges), `histogram_quantile` 13.0s → 11.8s,
`sum by(region)` 14.0s → 13.3s. Peak RSS unchanged.
Output verified against main: `max` bitwise identical; `sum`/`avg`
differ by ≤1.7e-11 relative (float summation order over 3.17M values;
run-to-run noise on main alone is ~1e-14).
## Tried and rejected (kept out of this PR)
Each was implemented, benchmarked A/B on the same dataset, and reverted
— recorded here so they aren't retried blindly:
- **Pre-reserving the partition-map merge**: zero effect — the merge is
hidden behind the slowest scan partition.
- **O(1) rate windows via a counter-reset prefix**: only -2~4%
end-to-end; the per-window reset rescan is cheap branch-predictable
compares, and the per-series prefix allocation ate the savings.
- **DataFusion `array_agg` for the sample load**: +17-55% slower — DF's
vectorized grouping is fast, but converting its Arrow list output into
engine structures is a second full materialization.
- **Sharded concurrent series map instead of `RepartitionExec`**:
+10-14% slower — the repartition copy is vectorized and buys each
consumer a purely local, cache-friendly map build.
Conclusion from the last two: the sample load (~60% of a 1-hour query)
is at a local optimum for the current row-at-a-time `Vec<Sample>` model;
further gains there need a columnar (SoA) sample layout, which is out of
scope for this PR.
## Notes
- Query-side only; no storage or config changes, no new env vars.
- `Accumulate` gains `merge`/`into_any`; all merges are
order-insensitive combines (add / min / max / union / concat).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Enrichment table queries (top k values and search) should ignore max
query range because enrichment tables, even stored as a stream in
backend, are not exactly logically a stream. It contains the data used
for enrichment like a csv file.
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Summary
This PR improves the Tantivy `SimpleSelect` path for queries using
`ORDER BY _timestamp ASC|DESC LIMIT n`.
Previously, each index file could return up to `limit` rows, causing
extra Parquet reads. The new path collects `(_timestamp, doc_id)`
candidates, safely stops when remaining file groups cannot produce
better results, and merges the global Top-N before building row
selections. On the fully indexed path, only the global `LIMIT`
candidates reach Parquet.
## Performance
https://github.com/openobserve/o2-ck-benchmark use this repo to generate
100M recores.
```sql
SELECT
*
FROM
k8s_logs
WHERE
kubernetes_pod_name = 'api-gateway-9a2c1e6f4-8923f'
ORDER BY _timestamp DESC
LIMIT 100
```
**main branch: 550ms
this branch: 250ms**
## Changes
- Add `SimpleSelectPruner` for safe early stopping, global Top-N
merging, and file/row pruning.
- Add timestamp-aware candidate results and cache entries.
- Fall back to full row-ID collection and DataFusion filtering when
Tantivy skips any condition.
- Conservatively retain unindexed, overlapping, or timestamp-tied files
unless they are provably dominated.
- Avoid caching results for files only partially covered by the query
time range.
- Separate SimpleSelect, SimpleCount, and generic row-count aggregation,
with minor query and scan-repartition cleanup.
## Correctness
- Supports both ascending and descending timestamp order.
- Uses strict timestamp bounds so ties are not pruned incorrectly.
- Keeps uncertain files when fewer than `limit` exact candidates are
available.
- No public API, configuration, or wire-format changes.
## Testing
- `cargo test --lib service::search::tantivy::` — 84 passed.
## What
Adds an **"Enable usage reporting" call-to-action** to the Cloud Billing
→ Usage tab for organizations that have self-usage reporting turned off,
a **"waiting for usage data" illustrated state** for orgs that have it
on but haven't reported any usage yet, and flips the per-org default for
the setting to **OFF** so the CTA is the entry point.
Previously, when `usage_stream_enabled` was off the Usage tab showed
only the billing-cycle cards and nothing where the daily chart would be.
And when it was on but the org's `usage` stream didn't exist yet, the
panel showed a raw `Search stream not found: usage` error. Now:
- **Reporting OFF** → an illustrated empty state (`OEmptyState`,
`wave-bars`) invites the user to enable reporting. Clicking **Enable**
opens a `ConfirmDialog` explaining the per-org, reversible effect before
the merge-and-POST enable flow runs.
- **Reporting ON, no data yet** → the raw "stream not found" error is
suppressed (on this panel only) and replaced with an illustrated
**"Waiting for usage data"** state. When usage lands, the search
succeeds, the error clears, and the daily chart shows through
automatically.
- **Reporting ON, has data** → unchanged: the daily line chart
(Ingestion + Search + Pipelines) renders.
## Behavior change (please review)
`default_usage_stream_enabled()` now returns **`false` in every build**
(previously cloud builds defaulted to ON when
`ZO_USAGE_REPORT_TO_OWN_ORG` was set). `ZO_USAGE_REPORT_TO_OWN_ORG`
remains the deploy-wide master switch but no longer flips the per-org
default. This is the intentional pairing with the CTA — an "enable"
prompt only makes sense when reporting is off by default. Explicit
stored values (opt-in or opt-out) still win in every build.
## How the waiting state is scoped
The "stream not found" suppression is set via `error_config` **only on
the usage panel's schema** (`usageDailyPanelSchema.ts`) —
`custom_error_handeling: true` with no `custom_error_message` silences
PanelSchemaRenderer's own error blocks. `usage.vue` catches the panel's
`@error`, detects the stream-not-found case specifically (unrelated
query errors still surface), and overlays the graphic. No other chart is
affected.
## Testing
- `web`: 417/417 billing specs pass, incl. new coverage for the enable
CTA, merged-settings POST, the schema silencing the renderer, and the
waiting overlay showing only for stream-not-found (not other errors).
- `cargo check -p openobserve` (non-release): clean.
## Note
These commits were cherry-picked onto a fresh branch from `main` — the
original branch (`chore/usage-ui-cloud-changes`) had already merged its
base work via #13112 and carried divergent, pre-squash history that
would have produced a conflicting diff.
## Summary
Fixes the "pin dashboard to Home" feature going stale when the pinned
dashboard is **moved** or **deleted**.
- **o2-enterprise#2139** (P1) — pin a dashboard, delete it, click the
pin → **404**.
- **o2-enterprise#2140** (P2) — pin a dashboard in Folder B, move it to
Folder A, click the pin → navigates to the **stale folder**.
### Root cause
The pin is stored as an org setting `home_dashboard` holding a static
snapshot `{ dashboardId, folderId, label }`, written once at pin-time
and never reconciled with the dashboard's live state. Both issues are
instances of the same class: a stored reference that drifts out of sync
on move/delete/rename.
## Approach — defense in depth
**L1 — authoritative server-side reconciliation (root fix).**
`move_dashboard` / `delete_dashboard` (OSS
`src/service/dashboards/mod.rs`) now keep the `home_dashboard` org
setting in sync at the mutation source:
- **move** → update the stored `folderId` to the destination folder.
- **delete** → clear the setting.
This is authoritative across every client and API caller (incl. **bulk
delete**, which loops `delete_dashboard`). Reconciliation is
**best-effort** — it runs *after* the primary op succeeds and
logs-and-swallows any error, so it can never fail the move/delete. No
`folder_id` is added to the GET response (no folder-membership info
leak).
**L2 — read-side re-sync (cross-system convergence).** The FE re-reads
the now-authoritative setting when the pinned tab is opened and when the
Dashboards page mounts, so a snapshot that went stale on **another
system** self-corrects before navigation.
**L3 — graceful degradation.** A deleted pin clears + toasts instead of
routing to a 404 (both the Home pinned tab and the Dashboards shortcut
button). Deliberate unpin (× button) stays silent.
## Deferred (follow-ups, out of scope)
- **Super-cluster cross-region reconciliation** — the replication queue
consumer (`src/super_cluster_queue/dashboards.rs`) applies moves/deletes
via table ops directly (bypassing the service layer where L1 lives), and
`system_settings` is not super-cluster-replicated. In a multi-region
deployment the non-originating region's pin isn't reconciled on **move**
(delete still converges via L3). Follow-up: add the same reconcile calls
to the queue consumer.
- **403-on-move (RBAC)** — moving a pin into a folder the current user
can't access. L3 handles 404 only for now; whether to clear the org-wide
setting vs. hide per-user is a separate design.
## Testing
- **Backend:** 6 new unit tests for the reconcile decision logic
(match/no-match, folder-changed, malformed value). `cargo test -p
openobserve home_pin` ✅
- **Frontend:** extended `HomeView.spec.ts` — re-sync on tab activation,
error toast on unavailable, silent deliberate unpin. `HomeView` +
`PinnedDashboardTab` specs: **40 passing** ✅
- **rcheck:** `cargo fmt --all --check` + `cargo clippy -- -D warnings`
clean ✅
- The `openHomeDashboard` shortcut guard is a small guard mirroring the
already-tested `PinnedDashboardTab` unavailable path; it's covered by
manual repro rather than a brittle full-mount of the 1400-line
`Dashboards.vue`.
### Manual reproduction
- **#2140:** pin in B → move to A → pin opens in **A** (correct
URL/breadcrumb); setting shows `folderId: "A"`.
- **#2139:** pin → delete → click pin/shortcut → no 404, toast,
pin/tab/shortcut gone; setting removed.
- **Cross-system:** move on client B while client A sits on Home/AI tab
→ A opens the pinned tab → resolves to the new folder.
## Layer compliance
All changes are OSS and unconditional (no enterprise gating, no
migrations, no new HTTP handlers) — `home_dashboard` is a generic system
setting readable by both builds. Reconciliation lives in the service
layer and reuses `src/service/db/system_settings.rs`.
## What
Adds a **daily usage view** to Cloud Billing → Usage for orgs with usage
reporting enabled, layered on top of the existing billing-cycle cards.
Cloud-only.
### Frontend (cloud billing components)
- **Merged into the existing Usage tab** (no separate tab): the billable
usage cards are always shown; when self-usage reporting is on, a
dashboards **date-range picker** (in the toolbar next to the GB/MB
toggle) drives **both** the cards and a daily chart.
- **Daily line chart** — a single query via `PanelSchemaRenderer` over
the org's own `usage` stream, broken down by event into **Ingestion /
Search / Pipeline / Remote Pipeline** lines (connect-nulls on).
- **Six billable usage tiles** on one row with **automatic unit
scaling** (GB→TB→PB / MB→GB→…) so large numbers stay readable; AI
Credits shown as a count.
- Cards re-fetch over the calendar range (derived `<N>days`, hardened
against bad values).
### Backend (all cloud-gated)
- `usage_stream_enabled` **defaults to `ZO_USAGE_REPORT_TO_OWN_ORG`** in
cloud builds, so orgs report to their own `usage` stream by default; a
per-org opt-out persists an explicit `false`. OSS / self-hosted default
stays `false`.
- **Reserved self-reporting streams** (`usage`, `stats`, `triggers`,
`errors`, `data_retention_usage`) are blocked from user **create /
ingest / delete** — cloud-only; the internal self-reporting writer
(`IngestionRequest::Usage`) is exempt.
- `get_org_usage`: a **missing `usage` / `data_retention_usage` stream
is tolerated** (treated as zero) instead of failing the whole request
with a 400.
## Notes
- No enterprise-repo or DB-migration changes; backend logic stays in OSS
behind `#[cfg(feature = "cloud")]`, per the OSS/Enterprise layer map.
- `Cargo.toml` / `Cargo.lock` are intentionally **not** included (local
`downloadO2.sh` dev-setup).
## Testing
- `cargo check` (OSS) and `cargo check --features cloud` — pass; `cargo
fmt` clean.
- Frontend: **411 billing unit tests** pass; `vue-tsc` typecheck clean;
production build succeeds.
## Summary
Fast fields are used by search optimizations such as timestamp
filtering, histograms, multi-histograms, and top-N queries. Previously,
`build_footer_cache` only warmed indexed field metadata, so fast field
access could still trigger additional reads later. This change primes
fast field handles during footer cache construction so those reads can
be served from the cached directory.
## Changes
- Warm fast field handles while building the puffin footer cache.
- Refactor `warm_up_terms` to operate on a `SegmentReader`.
- Update the Tantivy search warm-up call site for the new
`warm_up_terms` signature.
- Add a unit test to verify footer cache generation includes fast field
cache entries.
## Testing
- Added `test_build_footer_cache_with_fast_field`.
- Ran formatting and clippy fixes.
## Summary
- Skip OTLP histogram `_sum`, `_min`, and `_max` records when the
corresponding optional stat is absent.
- Preserve existing `_count` and bucket records.
- Add coverage for histograms whose optional stats are missing.
## Root Cause
OTLP `HistogramDataPoint` defines `sum`, `min`, and `max` as optional
values. The previous ingestion path converted missing optional stats
into JSON `null` and still emitted derived metric records such as
`<metric>_min` and `<metric>_max`. The flattening layer drops `null`
fields, so those derived streams could be created without a `value`
field.
Fixes#13019.
## Validation
- `cargo test -p openobserve test_process_hist_data_point --quiet`
- `cargo test -p openobserve
otlp::tests::metric_name_processing_tests::test_histogram_metric_name_suffixes
--quiet`
- `git diff --check`
The two metrics were using an empty string as the label name even though
no label names were registered in the metrics. Hence, whenever this
metric is emitted, there is a runtime panic. This pr fixes that.
## Scheduler: per-module job pullers with dedicated worker pools (A3 +
A4)
### Why
Scheduled jobs are dispatched late at scale. A single shared puller uses
one FIFO, one cadence, and one worker pool across every module, so a
backlog or slow handler in one module (e.g. an alert burst or a bulk
backfill) starves all others. This is Part A of the scheduler
scalability design.
### What
Give **each `TriggerModule` its own pull loop, cadence, LIMIT budget,
channel, and worker pool**, behind a flag. Off by default — zero
behavior change until explicitly enabled.
- **`ZO_SCHEDULER_PER_MODULE_PULLERS`** (default `false`) — master
switch.
- **off** → single shared puller (legacy, byte-identical SQL + same
global advisory lock).
- **on** → one lane per module (Alert, Report, DerivedStream, Backfill,
AnomalyDetection, QueryRecommendations).
- **Module-filtered pull** — `pull(...)` gains `module:
Option<TriggerModule>`; when `Some(m)` it filters `WHERE module = m` and
takes a **per-module advisory lock** (`scheduler_pull_lock:{module}`,
strategy C3) so modules no longer serialize against each other. `None`
keeps the exact legacy query + global lock.
- **Per-module budgets/cadence** — `ZO_SCHEDULER_<MODULE>_CONCURRENCY` /
`_INTERVAL` env vars, `0` = inherit the global alert default. Backfill
defaults to the smallest budget (1) so bulk jobs can't crowd out
latency-sensitive modules.
### Changes
| Area | Files |
|---|---|
| Flag + per-module config | `src/config/src/config.rs` |
| `module` param on `pull` (trait + wrappers) |
`src/infra/src/scheduler/{mod,postgres,sqlite}.rs`,
`src/service/db/scheduler.rs` |
| Per-module lanes (puller + pool + channel) |
`src/service/alerts/scheduler/worker.rs` |
### Notes
- No new SQL statement variants beyond two (module-filtered / legacy);
bind ordinals preserved, `is_realtime/is_silenced` exclusion unchanged.
- The keep-alive task and channel-full back-pressure retry are preserved
verbatim.
- Deferred: per-module Prometheus metrics (A6) and `FOR UPDATE SKIP
LOCKED` benchmark (A2/C2).
### Rollout
Flip the flag **cluster-wide**. During a mixed on/off window the
per-module and global locks don't mutually exclude, but the atomic
`UPDATE ... WHERE status=Waiting ... RETURNING` still claims each row
once (no double-dispatch) — only transient loss of cross-node FIFO. Size
the Postgres/sqlx pool for the extra pullers before enabling at scale.
### Testing
- 11 unit tests in `worker.rs`: legacy single-lane path,
one-lane-per-module, concurrency floor, and full
`resolve_module_configs` coverage (inherit/override/backfill floor).
- `cargo fmt --all -- --check` clean · `cargo clippy -- -D warnings`
clean.
## Summary
This PR adds a metadata count fast path for follower search `count(*)`
queries.
When a query is a simple count with no filters, files that are fully
inside the query time range can be counted from file metadata instead of
being scanned from object storage.
## Changes
- Replaces the Tantivy-only aggregate rewrite with a more general
aggregate optimization rewrite.
- Adds support for combining normal DataFusion scan results, metadata
count results, and Tantivy optimized results in one `UnionExec`.
- Splits storage files into metadata-count files and normal scan files.
- Uses metadata count only when:
- enterprise feature is enabled
- the optimizer selects `IndexOptimizeMode::SimpleCount`
- the index condition is `Condition::All()`
- the file time range is fully inside `[query_start, query_end)`
- Keeps boundary-overlapping files on the normal scan path to avoid
incorrect counts.
- Updates scan stats and logs for the broader index optimization path.
- Adds unit tests for file splitting and aggregate rewrite behavior.
## Notes
- Metadata count is only used for simple `count(*)` queries without
filters.
- Files with `max_ts == query_end` are not counted from metadata because
the query range is right-open.
- `MetadataCountExec` is enterprise-only.
Fixes#13041.
## Root cause
Two broken string comparisons, both caused by Windows path
canonicalization:
**1. `src/config/src/config.rs` — `check_disk_cache_config()`**
`Path::canonicalize()` on Windows returns a verbatim extended-length
path (`\\?\C:\OpenObserve\data\cache`), but `sysinfo` returns plain
mount points (`C:\`). The `starts_with` check
`"\\?\C:\...".starts_with("C:\\")` is always `false` → `disk_total =
disk_free = 0` on every Windows deployment.
**2. `src/job/stats.rs` — `update_node_disk_usage()`**
The comparison was also reversed: `mount_point.starts_with(data_dir)`
checks whether the mount point path begins with the (much longer) data
directory path — `"C:\\".starts_with("C:\\OpenObserve\\data\\")` is
always `false`. The correct direction is
`data_dir.starts_with(mount_point)`.
## Consequences when disk sizes are 0
- Disk cache `max_size` is computed from near-zero `effective_free`,
leaving `result_max_size = 0`
- `disk.rs` GC panics at first tick: `max_size / result_max_size`
evaluates before `max(1, …)` → divide by zero, GC task dies permanently
- `check_disk_circuit_breaker()` always compares `0 > 0` → never trips →
no self-protection against disk exhaustion on Windows
## Fix
1. **`config.rs`**: strip the `\\?\` prefix from the canonicalized cache
dir string before the `starts_with` lookup against sysinfo mount points.
2. **`stats.rs`**: fix the reversed comparison
(`mount_point.starts_with(data_dir)` →
`data_dir.starts_with(mount_point)`) and apply the same `\\?\` prefix
strip so canonicalized data dirs match correctly.
Both fixes are no-ops on Linux/macOS where `canonicalize()` never
produces the `\\?\` prefix.
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Design at: https://github.com/openobserve/o2-enterprise/issues/2112
## What changed
Three changes to cut per-round compactor database load on clusters with
many streams (with `ZO_COMPACT_INTERVAL=2` and 42k streams, each round
fired ~84k db calls):
1. **Retention generation moved to its own schedule.** New config
`ZO_COMPACT_DATA_RETENTION_INTERVAL` (default 3600s). Retention job
generation runs `get_min_date` (`SELECT MIN(date) FROM file_list WHERE
stream = ?`) for every stream, but retention works at day granularity,
so running it every `ZO_COMPACT_INTERVAL` (commonly 2s) wasted enormous
db traffic.
2. **Stream-stats gate before `get_min_date`.** Retention generation now
checks the in-memory stream stats cache first: if the cached
`doc_time_min` is newer than the stream's retention boundary, the stream
has no expired data and is skipped with zero db calls. Since
`delete_by_date` already refreshes `stream_stats` right after deletion,
a stream only pays the `get_min_date` query when it actually crosses the
retention boundary (~once per day). `doc_time_min == 0` (no stats yet)
falls through to the db check for safety.
3. **Unlocked existence probe in `get_pending_jobs` (postgres).** All
compactor nodes serialize on the same `pg_advisory_xact_lock` every
round. A cheap `SELECT id FROM file_list_jobs WHERE status = 0 LIMIT 1`
(index-only via `file_list_jobs_stream_status_idx`) now runs before the
locked transaction, so idle rounds skip the lock and the `GROUP BY
stream` scan entirely.
## Net effect
Steady-state retention traffic drops from ~O(streams) `MIN(date)`
queries every few seconds to ~1 query per stream per day, and idle merge
rounds no longer contend on the global advisory lock.
## Notes for reviewers
- Retention job **execution** (the per-round meta-table pass that picks
up user-triggered stream deletes) moved to the hourly schedule together
with generation, so a user-initiated stream delete can now take up to
`ZO_COMPACT_DATA_RETENTION_INTERVAL` to be physically executed. The
stats gate makes generation cheap, so this interval can be shortened if
that delay matters.
- SQLite backend is unchanged for `get_pending_jobs` — it is single-node
with an in-process mutex, so there is no lock contention to avoid.
## Background
In a super-cluster query, the cluster leader sends the query to region
leaders. Each region leader then uses `RemoteScanExec` to query data
nodes inside its region.
When the source physical plan is `AggregateExec(mode = Partial)`, each
data node returns partial aggregate results. Previously, the region
leader only wrapped the plan with `RemoteScanExec`, so partial results
from multiple data nodes were sent up directly to the cluster leader.
This change adds a region-level `PartialReduce` around `RemoteScanExec`,
so each region can reduce its own partial aggregate results before
returning them.
## Before
For non-sorted aggregate plans:
```text
Cluster Leader
└── ... upper aggregate / merge ...
Region Leader / Follower
└── RemoteScanExec
└── remote plan sent to region data nodes:
└── AggregateExec(mode = Partial)
└── scan / filter / projection ...
```
In this shape, rows with the same group key can still appear multiple
times in one region, because the region leader does not reduce the
remote partial outputs.
## After
With `GROUP BY`:
```text
Cluster Leader
└── ... upper aggregate / merge ...
Region Leader / Follower
└── AggregateExec(mode = PartialReduce)
└── RepartitionExec(Hash(group columns), same partition count)
└── RemoteScanExec
└── remote plan sent to region data nodes:
└── AggregateExec(mode = Partial)
└── scan / filter / projection ...
```
Without `GROUP BY`:
```text
Cluster Leader
└── ... upper aggregate / merge ...
Region Leader / Follower
└── AggregateExec(mode = PartialReduce)
└── CoalescePartitionsExec
└── RemoteScanExec
└── remote plan sent to region data nodes:
└── AggregateExec(mode = Partial)
└── scan / filter / projection ...
```
In short:
```text
Before:
RemoteScanExec(PartialAgg)
After:
PartialReduce(Repartition/Coalesce(RemoteScanExec(PartialAgg)))
```
## Sorted Plans
Plans rooted at `SortPreservingMergeExec` keep the existing shape:
```text
SortPreservingMergeExec
└── RemoteScanExec
└── original ordered physical plan
```
This avoids changing the semantics of ordered query paths such as `ORDER
BY` and
`ORDER BY LIMIT`.
## Implementation
The reusable wrapping logic is in:
```text
src/service/search/datafusion/optimizer/physical_optimizer/remote_scan.rs
```
```rust
pub(crate) fn wrap_partial_reduce_for_aggregate(
partial_reduce_enabled: bool,
agg: &AggregateExec,
input: Arc<dyn ExecutionPlan>,
) -> Result<Arc<dyn ExecutionPlan>>
```
The super-cluster follower integration is in:
```text
src/service/search/super_cluster/follower.rs
```
```text
if physical_plan is SortPreservingMergeExec:
keep SortPreservingMergeExec(RemoteScanExec(...))
else:
remote_scan = RemoteScanExec(physical_plan)
if physical_plan is AggregateExec(mode = Partial or PartialReduce):
return PartialReduce(remote_scan)
else:
return remote_scan
```
## Benefit
Each region now performs a local partial reduce before sending results
to the cluster leader. This reduces intermediate aggregate output across
regions while leaving sorted query plans unchanged.
for metrics data, we only generate vortex file in the compactor, but
others we generate parquet file on ingester and compactor. becuase
metrics have many small files
Currently if map group to role is true, and custom parser is not
enabled, o2 still goes for org mapping (removing any extra org not part
of dex claim) which is not expected behaviour.
This pr fixes this.
Design at: #12995
## Summary
On k8s, when the user only sets a memory limit (no CPU limit), we detect
the whole node's CPU cores via sysinfo — e.g. 64 cores on a node while
the pod only has 8GB memory. All cpu-derived thread pools
(`http_worker_num`, `query_thread_num = cpu * 4`, file
download/move/merge threads, etc.) then get sized for 64 cores against
an 8GB memory budget, causing memory pressure and OOM.
## Changes
- In `check_limit_config`, clamp `limit.cpu_num` by the memory limit:
`cpu_num = real_cpu_num.min(mem_total / 1GB).max(2)`.
- If the memory limit is unknown (0), use the CPU limit directly.
- `real_cpu_num` keeps the true detected value; `get_cpu_limit()` /
cgroup detection is untouched.
Example: 64-core node + 8GB memory limit → `cpu_num = 8`, so
`query_thread_num` becomes 32 instead of 256.
## Summary
Fixes a PostgreSQL type mismatch in compactor fast mode when
`ZO_LOCAL_MODE=true` is used with `ZO_META_STORE=postgres`.
The fast mode query returned `0 as num`, which PostgreSQL infers as
`INT4`. The result is decoded into `MergeJobPendingRecord.num` as `i64`
/ `INT8`, causing sqlx to fail with:
```text
ERROR openobserve::job::compactor: [COMPACTOR::JOB] run data merge
error: SqlxError# error occurred while decoding column "num": mismatched
types; Rust type `i64` (as SQL type `INT8`) is not compatible with SQL
type `INT4`
```
This changes the literal to 0::BIGINT AS num.
## Reproduction
Tested with:
- ZO_META_STORE=postgres
- ZO_LOCAL_MODE=true
- ZO_COMPACT_ENABLED=true
- ZO_COMPACT_INTERVAL=2
On openobserve/openobserve:v0.92.0-rc1, inserting a pending row into
file_list_jobs reproduces the error repeatedly and leaves the job stuck
at status = 0.
I've also created an issue on this #13007
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## What
Fixes several issues in PostgreSQL `file_list` partition maintenance.
- **Fixed daily schedule (P1):** maintenance now runs once per day at a
fixed UTC hour (`ZO_FILE_LIST_MAINTENANCE_HOUR`, default `4`) instead of
`boot_time + N×24h`. The old `retention_allowed_hours` gate — which
almost never matched the random run time, so drop/reindex effectively
never ran — is removed; pre-create + drop + reindex now always run each
cycle.
- **Single-node execution + failover (P2/P3):** the task now runs only
on compactor nodes and uses a distributed lock plus a
`/file_list/maintenance/last_run` marker (replacing the per-run PG
advisory lock). This guarantees exactly-once-per-day even when nodes
wake at slightly different times, with automatic failover if the
previous runner is down.
- **`file_list_dump_stats` gated on dump flag (P5):** the table and its
partitions are no longer created/pre-created when
`ZO_COMPACT_FILE_LIST_DUMP_ENABLED=false`.
## Notes
- **P4 (delayed events → DEFAULT):** confirmed a non-issue in auto mode
— partitions are created on demand and DEFAULT is never dropped. No
change.
- **P6 (manual mode + `retention_allowed_hours`):** out of scope,
tracked separately.
- `retention_allowed_hours` is now decoupled from file_list maintenance
and keeps its existing meaning for the retention job only.
## Config
| Env var | Default | Change |
|---|---|---|
| `ZO_FILE_LIST_MAINTENANCE_HOUR` | `4` | new — UTC hour for daily
maintenance |
| `ZO_COMPACT_FILE_LIST_DUMP_ENABLED` | `false` | now also gates dump
table creation |
## Testing
- `cargo check` / `cargo clippy` on `infra` + `config`: clean.
- Added unit tests for the schedule/day helpers; full non-DB test module
passes.
Design doc: `docs/design/file-list-partition-maintenance.md`
related to https://github.com/openobserve/o2-enterprise/issues/1860
- [x] limit parallel by the target partitions
## Summary
This PR improves Flight `do_get` response throughput by executing and
encoding output partitions in parallel instead of funneling everything
through a single sequential stream/encoder path.
## Changes
- Switches Flight `do_get` from `execute_stream` to
`execute_stream_partitioned`.
- Adds per-partition encoder tasks that coalesce RecordBatches, encode
Arrow IPC, and apply ZSTD compression independently.
- Splits Flight encoding into:
- `ChunkCoalescer` for 8192-row batch coalescing.
- `FlightDataEncoder` for IPC/ZSTD encoding of completed chunks.
- Adds `PartitionEncoderStream` to handle per-partition batch coalescing
and encoding.
- Preserves custom Flight messages behavior:
- early-emitted messages are sent before the first data message;
- stats/metrics/peak-memory messages are sent after data completes.
- Uses a fresh dictionary tracker per encoded chunk so
dictionary-encoded columns remain decodable when chunks are encoded
independently.
- Cleans up stream lifecycle handling by aborting partition tasks before
clearing session resources.
## Performance
100M records
single node optimzier disable(to simulate the distribute query)
### Q1
```sql
SELECT
trace_id,
count(*) AS cnt,
max(_timestamp) AS ts
FROM
"k8s_logs"
GROUP BY
trace_id ORDER BY cnt DESC LIMIT 100
```
12M trace_id unique value
12M records send from follower to leader
main branch: 6s
this branch: 3s
### Q2
```sql
SELECT count(distinct request_id) FROM "k8s_logs"
```
100M request_id unique value
100M records send from follower to leader
main branch: 30s
this branch: 11s
### Q3
```sql
SELECT
kubernetes_namespace_name,
COUNT(DISTINCT trace_id) AS cnt
FROM
k8s_logs
GROUP BY
kubernetes_namespace_name ORDER BY cnt DESC
```
10 kubernetes_namespace_name unique value, 12M trace_id unique value
63M records send from follower to leader
main branch: 21s
this branch: 8.3s
### Q4
```sql
SELECT
host,
COUNT(DISTINCT trace_id) AS cnt
FROM
k8s_logs
GROUP BY
host ORDER BY cnt DESC
```
2160 host unique value, 12M trace_id unique value
99.57M records send from follower to leader
main branch: 56s
this branch: 20s
Design at: #12894
## What changed
- Added `ZO_FEATURE_DEFAULT_INDEX_FIELDS_ENABLED` (default `true`). When
set to `false`, the built-in default fields are disabled for:
- full text search (`log`, `message`, `msg`, `content`, `data`, `body`,
`json`, `error`, `llm_input`, `llm_output`)
- secondary index (`trace_id`, `service_name`, `operation_name`)
- distinct values (`service_name`, `operation_name`)
Only the fields from the `*_EXTRA_FIELDS` ENVs and per-stream settings
are used, so the extra-fields ENVs become full replacements for the
defaults.
- Added `ZO_FEATURE_BLOOM_FILTER_EXTRA_FIELDS` and deprecated
`ZO_BLOOM_FILTER_DEFAULT_FIELDS`. The old ENV is still honored: it is
merged into the new one at startup with a deprecation warning, and will
be removed in v1.0.0.
## Why
Users with high log volume pay ingestion/compaction/storage costs for
tantivy indexes on default FTS fields they never query, with no way to
opt out (see #12894). This is especially bad for CJK content where
single-character tokens make the inverted index a bottleneck instead of
an accelerator.
## Example: fully custom default fields
```
ZO_FEATURE_DEFAULT_INDEX_FIELDS_ENABLED=false
ZO_FEATURE_FULLTEXT_EXTRA_FIELDS=f1,f2
ZO_FEATURE_INDEX_EXTRA_FIELDS=f1,f2
ZO_FEATURE_BLOOM_FILTER_EXTRA_FIELDS=""
```
## Notes for reviewers
- All changes are in `src/config/src/config.rs`: the gating happens
inside the `SQL_FULL_TEXT_SEARCH_FIELDS` /
`SQL_SECONDARY_INDEX_SEARCH_FIELDS` / `DISTINCT_FIELDS` lazy statics, so
every consumer (`get_stream_setting_*` in infra/schema, ingestion,
compaction, UDS auto-enable keep-list, and the `/config` API's
`default_fts_keys` / `default_secondary_index_fields`) follows
automatically.
- `get_stream_setting_index_updated_at_for_fields` also follows: with
defaults disabled, those fields no longer get the `BASE_TIME` "indexed
since forever" treatment.
- Bloom filter has no hardcoded defaults, so the flag does not gate it;
the ENV rename just makes its naming consistent with the other
extra-fields ENVs.
- Added unit tests for the bloom ENV migration (migrate, merge when both
set, no-op).
Closes#12894
## Fix PagerDuty prebuilt destination
### Problem
PagerDuty Events API v2 requires the integration key as `routing_key`
**in the JSON body**, but it never got there:
- Template used `{integration_key}` — not a real server variable → sent
literally → PagerDuty rejects payload. `severity`/`source` hardcoded.
- UI stored integration key only in an `X-Routing-Key` header, which
PagerDuty ignores.
- Severity stored as `credential_severity`, never referenced by the
template.
- "Test" passed only via client-side substitution, diverging from the
real server send path → Test could pass while real alerts failed.
### Fix
Route everything through the alert engine's existing endpoint-metadata
substitution so real sends and Test/Preview render identical, valid
payloads.
- **Template body**: corrected to PagerDuty v2 shape with substitutable
`{routing_key}`, `{severity}`, `{source}` (`prebuilt-destinations.json`,
Rust fallback, web `pagerduty.ts`).
- **Fields now used**: new `generatePrebuiltMetadata()` writes
`routing_key` (=integration key), `severity`, `source` into destination
metadata on save → server fills them at send time. Dropped useless
`X-Routing-Key` header; edit-restore reads key from metadata (legacy
header kept as fallback).
- **Test uses real template**: Test/Preview sample data now supplies
`routing_key`/`source` → client renders the same body the server sends.
- **Stale templates**: `ensure_system_templates()` previously skipped
already-existing prebuilt templates → existing installs kept the broken
body. Now refreshes system-managed templates whose stored body has
drifted from shipped source.
### Verification
- Rust: `config` + `openobserve` binary compile; prebuilt loader tests
pass (10/10). Added `PartialEq`/`Eq` to `TemplateType` for drift check.
- Updated header unit test + added `generatePrebuiltMetadata` tests. JS
suite not runnable locally (pre-existing `ERR_REQUIRE_ESM` in vitest
setup, unrelated) — run in CI.
### Note
Out of scope: Test always uses `prebuilt_pagerduty` template name, not a
per-destination custom template override.
related to https://github.com/openobserve/o2-enterprise/issues/1860
when we use the do_get, the recordbatch already compression by the zstd,
we do not need compression again
## New env
```
ZO_GRPC_HTTP2_ADAPTIVE_WINDOW=false
```
## Performance
100M records
single node optimzier disable(to simulate the distribute query)
### Q1
```sql
SELECT
trace_id,
count(*) AS cnt,
max(_timestamp) AS ts
FROM
"k8s_logs"
GROUP BY
trace_id ORDER BY cnt DESC LIMIT 100
```
12M trace_id unique value
12M records send from follower to leader
main branch: 12s
this branch: 6s
### Q2
```sql
SELECT count(distinct request_id) FROM "k8s_logs"
```
100M request_id unique value
100M records send from follower to leader
main branch: 74s
this branch: 30s
### Q3
```sql
SELECT
kubernetes_namespace_name,
COUNT(DISTINCT trace_id) AS cnt
FROM
k8s_logs
GROUP BY
kubernetes_namespace_name ORDER BY cnt DESC
```
10 kubernetes_namespace_name unique value, 12M trace_id unique value
63M records send from follower to leader
main branch: 48s
this branch: 21s
### Q4
```sql
SELECT
host,
COUNT(DISTINCT trace_id) AS cnt
FROM
k8s_logs
GROUP BY
host ORDER BY cnt DESC
```
2160 host unique value, 12M trace_id unique value
99.57M records send from follower to leader
main branch: 104s
this branch: 56s
related to: https://github.com/openobserve/o2-enterprise/issues/1860
related to: https://github.com/openobserve/openobserve/pull/12418
## Performance
100M records
single node optimzier disable(to simulate the distribute query)
### Q1
```sql
SELECT
trace_id,
count(*) AS cnt,
max(_timestamp) AS ts
FROM
"k8s_logs"
GROUP BY
trace_id ORDER BY cnt DESC LIMIT 100
```
12M trace_id unique value
12M records send from follower to leader
main branch: 20s
this branch: 12s
### Q2
```sql
SELECT count(distinct request_id) FROM "k8s_logs"
```
100M request_id unique value
100M records send from follower to leader
main branch: 77s
this branch: 74s
### Q3
```sql
SELECT
kubernetes_namespace_name,
COUNT(DISTINCT trace_id) AS cnt
FROM
k8s_logs
GROUP BY
kubernetes_namespace_name ORDER BY cnt DESC
```
10 kubernetes_namespace_name unique value, 12M trace_id unique value
63M records send from follower to leader
main branch: 54s
this branch: 48s
### Q4
```sql
SELECT
host,
COUNT(DISTINCT trace_id) AS cnt
FROM
k8s_logs
GROUP BY
host ORDER BY cnt DESC
```
2160 host unique value, 12M trace_id unique value
99.57M records send from follower to leader
main branch: 112s
this branch: 104s
disable partial reduce: 88s
## Summary
Improve `PartialReduce` execution parallelism in remote scan aggregation
plans.
For grouped aggregations, `PartialReduce` now hash repartitions partial
aggregate output by the group keys instead of always coalescing all
partitions into one. This keeps the reduce stage parallel while ensuring
rows with the same group key are routed to the same partition.
## ExecutionPlan change
Before:
```text
AggregateExec: mode=PartialReduce
CoalescePartitionsExec
AggregateExec: mode=Partial
...
```
After, for `GROUP BY`:
```text
AggregateExec: mode=PartialReduce
RepartitionExec: partitioning=Hash([group_keys...], N)
AggregateExec: mode=Partial
...
```
For no `GROUP BY`, the plan still falls back to:
```text
AggregateExec: mode=PartialReduce
CoalescePartitionsExec
AggregateExec: mode=Partial
...
```
## Changes
- Reuse `GROUP BY` column expressions as hash repartition keys before
`PartialReduce`.
- Preserve the partial aggregate output partition count for grouped
partial reduce.
- Keep `CoalescePartitionsExec` behavior for global aggregations without
group keys.
- Add unit tests for grouped hash repartition and no-group-by coalesce
fallback.
## Testing
Added tests in `remote_scan.rs` covering the new `PartialReduce`
planning behavior.
## Problem
Filebeat (and other Elasticsearch clients) can't ingest using an org
ingestion token (`o2oi_…`). On startup Filebeat's `eslegclient` pings
the root URL `GET /api/{org_id}/` to detect the ES version. That request
**403s** (`Unauthorized Access`), so Filebeat never gets past the
handshake and loops in backoff-retry, shipping nothing — even though
`POST /api/{org_id}/_bulk` works fine.
## Root cause
The org-root handler is registered as the slash-mandatory route
`/{org_id}/`, so every request to it carries a trailing slash. In
`AuthExtractor::from_request_parts`, `"default/".split('/')` yields
`["default", ""]` (len 2).
The ES-compat bypass entry in `ROUTE_PERMISSIONS` is `path: &["{org}"]`
(len 1, `bypass: true`), and `path_matches` requires an exact length
match (`si == path.len()`). The trailing empty segment makes the length
2 ≠ 1 → no match → the "route missing" fallback sets `bypass_check =
false` with an empty `o2_type` → a roleless ingestion token hits OpenFGA
→ **403**.
Because the route *requires* the trailing slash, this bypass entry was
effectively **dead code** — no request could ever satisfy both the route
and the matcher. Notably `validate_credentials` already strips the
trailing empty segment; `AuthExtractor` did not. That inconsistency was
the bug.
## Fix
Strip a trailing empty path segment in
`AuthExtractor::from_request_parts` before route resolution, mirroring
`validate_credentials`:
```rust
let mut path_columns = path.split('/').collect::<Vec<&str>>();
if path_columns.len() > 1
&& let Some(v) = path_columns.last()
&& v.is_empty()
{
path_columns.pop();
}
```
## Blast radius
- Only affects requests that arrive **with a trailing slash**; no-slash
paths split identically → no behavior change.
- `/{org_id}/` is the **only** org-scoped handler in the codebase
registered with a trailing slash; every other endpoint 404s at routing
for trailing-slash requests regardless of auth, so no working (200)
request can break.
- Trailing slash never grants more access than no-slash (verified for
both token and root).
- Guarded `len > 1` → no empty-vec / out-of-bounds risk.
## Verification
Local: enterprise build, OpenFGA enabled, org ingestion token on
`default`.
| Request | Before | After |
|---|---|---|
| token `GET /api/default/` (handshake) | 403 | **200** |
| token `POST /api/default/_bulk` | 200 | 200 |
| token `GET /api/default/streams` (real read API) | 401 | 401 (still
blocked) |
| **Filebeat** (token) | `403 Forbidden` → infinite backoff |
**"Connection established"**, logs ingested |
- router should watch org settings changes
- if a sso parser is selected in the settings, the field should show the
parser function selected even after page refresh. Currently, on page
refresh the select button empties the field even though the sso parser
function is already configured.
<img width="1492" height="806" alt="Screenshot 2026-06-26 at 7 10 42 PM"
src="https://github.com/user-attachments/assets/5d40fb8f-779c-4e0e-81f6-f86359b5056a"
/>
- Adds index only to the next_run_at column to improve the pull method
throughput and time.
- Adds a 80% fillfactor for the postgres table to enhance the HOT% for
updates. One minor caveat is that, since the next_run_at is now indexed,
for rescheduling updates (next_run_at update), it will not be able to
utilize Heap Only Tuples and can add to the index/dead tuple bloating in
the postgres db. The pulls, keep alives and watch timeout can still use
the HOT approach because they don't touch the next_run_at column.
Note in the results that, the adv lock wait time reduced drastically
after using the index.
Another thing to note that there is a difference from the file_list
table where the dead tuples caused bad search performance when there
were huge number of records in the table (1TB+ table size). In that
case, the auto vacuum didn't help much because, the amount of dead
tuples required to trigger auto vacuum, itself was very big - resulting
in bad index bloating. In the scheduled_jobs, in most cases, almost all
the rows are updated frequently, resulting in relatively higher
percentage of dead tuples and hence trigger auto vacuum more frequently.
The above is visible from the below test results as well - for baseline
(current main branch approach), the HOT% is very good (91%) because the
updates didn't touch the indexes at all and hence, low no. of dead
tuples. For index_ff100 (index on next_run_at + default 100%
fillfactor), it already triggers 1 autovacuum. Same for 80% fill factor.
Results (100K rows, 8 clients, 60s)
### Tier 1 — pull query plan
| variant | plan | exec time | shared buffers |
|---------------|-----------------------------------|-----------:|---------------:|
| `baseline` | Parallel Seq Scan + top-N Sort | **13.5 ms**| 4424 |
| `index_ff100` | Index Scan `scheduled_jobs_pull_idx` | **0.039 ms** |
7 |
| `optimized` | Index Scan `scheduled_jobs_pull_idx` | **0.030 ms** | 7
|
→ **~350× faster** at 100K, and the gap **grows with table size** (seq
scan is O(rows), index is
O(log n + limit)). Fillfactor has no effect on the read plan, as
expected.
### Tier 2 — throughput, lock-wait, churn health
| metric (per variant) | `baseline` | `index_ff100` | `optimized` |
|----------------------------------|-----------:|--------------:|------------:|
| **total tps** | 75.7 | **930.6** | 875.3 |
| avg txn latency | 105.6 ms | 8.6 ms | 9.1 ms |
| **advisory-lock wait** (pull) | **177 ms** | **2.4 ms** | 1.7 ms |
| pull claim UPDATE | 26.4 ms | 0.93 ms | 0.80 ms |
| updates in 60s (`n_tup_upd`) | 22.5 K | 278.8 K | 261.9 K |
| **HOT %** (`n_tup_hot_upd`) | 91.3 % | 47.0 % | 50.2 % |
| index size | 18 MB | 28 MB | **25 MB** |
| true dead-tuple % (pgstattuple) | 0.10 % | 0.84 % | **0.11 %** |
| free space % (pgstattuple) | 3.3 % | 9.2 % | **19.1 %** |
| autovacuum runs | 0 | 1 | 1 |
# feat: Dashboard Table Chart — per-column formatting
## 1. Entry point — config panel
Column formatting lives in the panel **Config** panel, under a **Column
Formatting**
section. A **"Configure column formatting"** button opens the **Column
Formatting**
dialog.
<img width="1920" height="907" alt="image"
src="https://github.com/user-attachments/assets/21d05294-3d3e-4f84-893f-cf3b235adf0d"
/>
## 2. Column Formatting dialog — empty state
A 3-pane dialog. When nothing is configured yet it shows an empty state
(`OEmptyState`)
with a single **"+ Add field"** action. **Add field** is a dropdown
listing the panel's
columns (each with a **NUM / TEXT** type badge); the menu matches the
trigger width.
<img width="1920" height="907" alt="image"
src="https://github.com/user-attachments/assets/866fb119-76f3-48b2-98b6-50df24660a8c"
/>
## 3. Column Formatting dialog — editing a field
- **Left** — "+ Add field" + the list of added fields (type badge, name,
hover-to-remove ✕).
- **Middle** — the shared controls for the selected field:
- **Field Type** — `Auto` / `Num` / `Text` (Auto detects from the data;
Num/Text force it).
- **Value Formatting** — unit dropdown (numbers, bytes, durations,
percentage, currency, custom) — numeric columns only.
- **Alignment** — `Auto` / `Left` / `Center` / `Right` (Auto =
type-based default).
- **Styling** — Text color, Background color (curated swatches +
custom), and a Unique-value-color toggle.
- **Conditional Styling** — threshold rules (numeric columns only).
- **Right** — a **live preview** of the column rendered with the current
settings.
<img width="1920" height="907" alt="image"
src="https://github.com/user-attachments/assets/276289d8-f92d-46f2-a413-b58e840e2f6b"
/>
## 4. Conditional styling
Per-column rules read as a sentence: **If value** `[operator]` `[value]`
**then text
color** `[…]` **and background** `[…]`. Operators: `<`, `>`, `<=`, `>=`,
`=`, `!=`.
Add multiple rules — **the last matching rule wins**, so later rules
override earlier
ones (e.g. `> 1000` takes precedence over `> 400` for `2301`).
<img width="1920" height="907" alt="image"
src="https://github.com/user-attachments/assets/b47bde1c-11e4-4ec9-8d6b-03f702ae8902"
/>
**Example** — flag `count` when it's low:
> Conditional Styling → **Add rule** → operator `<`, value `50` → Text
`#b91c1c`, Background `#fef2f2`.
Persists in `override_config` as:
```json
{
"field": { "matchBy": "name", "value": "count" },
"config": [
{
"type": "conditional_styles",
"rules": [
{ "operator": "<", "threshold": 50, "textColor": "#b91c1c", "bgColor": "#fef2f2" }
]
}
]
}
```
## 5. Rendered result in the live table
Cells matching a rule render with the configured colors; non-matching
cells stay
normal. Thresholds may be decimals (e.g. `0.5`).
<img width="1920" height="907" alt="image"
src="https://github.com/user-attachments/assets/4c4146f9-b018-465e-81ae-fc04149cdc83"
/>
---
## Data model
Overrides persist as an array at `config.override_config`; each entry
targets one column
and carries typed config items. The discriminants (`tableConfigUtils.ts`
`OVERRIDE_CONFIG_TYPES`) are kept in sync with the Rust v8 schema
(`src/config/src/meta/dashboards/v8/mod.rs`):
```
field_type · unit · alignment · text_color · background_color · unique_value_color · conditional_styles
```
Unset values use `null` (inherit the panel default); `field_type`
defaults to `auto`.
Backward compatible: dashboards saved before this change load and render
unchanged (the
v8 `Config` enum has no `deny_unknown_fields` and no newly-required
field).
## Architecture
| Piece | File |
| --- | --- |
| Dialog (3-pane) |
`web/src/components/dashboards/OverrideConfigPopup.vue` |
| Shared sections |
`web/src/components/dashboards/ColumnFormatControls.vue` |
| Color picker | `web/src/components/dashboards/ColorSwatchPicker.vue` |
| Model + (de)serialize |
`web/src/composables/dashboard/useColumnFormatting.ts` |
| Parse / apply / format | `web/src/utils/dashboard/tableConfigUtils.ts`
|
| Render (cell styling) |
`web/src/components/dashboards/panels/TableRenderer.vue` |
| Config-panel entry |
`web/src/components/dashboards/addPanel/OverrideConfig.vue` |
| Rust v8 schema | `src/config/src/meta/dashboards/v8/mod.rs` |
## Scope changes (vs the original proposal)
- **Cell types removed** — the `Text / Bar / Spark` toggle, in-cell
**progress bars**,
and **sparklines** were prototyped and then dropped (frontend + the v8
`cell_type`
variant). Formatting is type/unit/alignment/color/conditional only.
- **Inline header popover removed** — formatting is reached via the
config-panel dialog
(no per-header tune-icon popover / "Edit all").
- **Conditional precedence** — last matching rule wins (was
first-match).
## Notable correctness fixes
- **Value-mapping ranges** — numeric values arriving as strings (e.g.
`"5"`) again match
range mappings (restored origin behavior; empty/null cells don't
false-match a
`0`-bounded range).
- **Conditional threshold** — decimal thresholds (e.g. `0.5`)
deserialize correctly on the
Rust side (flexible `OrdF64` deserializer).
- **Reset-on-type-change** — forcing a numeric column to `Text` clears
its now-hidden
unit/conditions (keyed by field, so removing a column never wipes a
different one).
---
# Column Filtering
> Detailed design: [`table-filtering.md`](./table-filtering.md)
In-table, multi-select, spreadsheet-style filtering for table panels.
Each column header
gets a filter button; its dropdown lists that column's **distinct
values** as a checkbox
list — tick/untick to narrow the rows. Filtering is **purely
client-side** (no re-query).
## 1. Enable it
A single **Column Filtering** toggle in the panel **Config** panel
(`config.table_filtering`, an `Option<bool>` in the v8 schema). Off by
default; when on,
the table renderers receive `:enable-filtering` and the header filter
buttons appear.
(Filtering is suppressed in print/export mode.)
<img width="1680" height="907" alt="image"
src="https://github.com/user-attachments/assets/d834a916-c7eb-4dce-9984-95276e996e87"
/>
## 2. Filter a column
Click a header's filter button to open the dropdown:
- **Distinct values** of the column as a checkbox list — pick any
subset.
- **Searchable** — a search box narrows the value list.
- **Clear** a column's filter in one click.
- Values display formatted (units/mappings) but match on the raw value.
- Multiple column filters combine with **AND**; sorting preserves active
filters.
<img width="1680" height="907" alt="image"
src="https://github.com/user-attachments/assets/0671b75e-e049-442f-bc56-53e861b9ff41"
/>
## Architecture
| Piece | File |
| --- | --- |
| Config toggle |
`web/src/components/dashboards/addPanel/ConfigPanel.vue`
(`table_filtering`) |
| Renderer wiring |
`web/src/components/dashboards/PanelSchemaRenderer.vue`
(`:enable-filtering`) |
| Filter UI + logic | `web/src/components/TenstackTable.vue` (gated by
`enableColumnFilter`) |
| v8 schema flag | `src/config/src/meta/dashboards/v8/mod.rs`
(`table_filtering: Option<bool>`) |
Implemented on the same branch as the formatting work;
`enableColumnFilter` is a no-op
when false, so non-table charts and filtering-off panels are unaffected.
</content>
Design At: dashboards/table-enhancement
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
## Problem
The enterprise build is broken on `main` (`error[E0061]`):
```
error[E0061]: this function takes 3 arguments but 2 arguments were supplied
--> src/super_cluster_queue/short_urls.rs:33:13
33 | infra::table::short_urls::add(&short_id, &original_url).await?;
| argument #3 of type `&str` is missing
```
#12829 ("fix: org id as mandatory check") made `org_id` a mandatory
third argument of `infra::table::short_urls::add`, but the
**enterprise-gated** super-cluster queue handler
`super_cluster_queue::short_urls::process` was not updated. This module
is `#[cfg(feature = "enterprise")]`, so OSS-only CI compiled fine while
the enterprise build (and every downstream Playwright shard) failed.
## Fix
The super-cluster queue message (`emit_put_event` → `short_url_put`)
only carries the `original_url` over the wire — `org_id` is never
published to the queue. So the receiver passes an empty `org_id`.
This matches the legacy semantics introduced in #12829, where
`short_urls::get` accepts empty-org rows from any org:
```rust
infra::table::short_urls::add(&short_id, &original_url, "").await?;
```
## Note / possible follow-up
With an empty `org_id`, short URLs replicated **across a super cluster**
are stored org-agnostic (accepted from any org on lookup), the same as
pre-existing legacy rows. If org-scoped short URLs need to survive
super-cluster replication, a follow-up should thread `org_id` through
the queue message value (producer `emit_put_event` and the
`short_url_put` payload) — a larger change touching the wire format.
This PR only unblocks the build with behavior consistent with #12829's
design.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This attempts to refactor the process_node fn to make it more readable
and change-able.
The commits should be reviewed individually, first extracts individual
match arms into their own fns, second does cleanup of fn params by
extracting them into structs.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Currently in pipeline execution, if any null record is encountered, it
results in errors. Instead it should just ignore the null records and
continue. It is very common in pipeline vrls to use -`. = null` when
certain condition matches.
Subhra264
Ashish Kolhe
udev
tsushanthandHengfei Yang
Rafael MendonçaandHengfei Yang
Abhay Padamaniandgithub-actions[bot]