## Problem
Fixes#13946. The sum of histogram bucket counts exceeded `count(*)`
over the same time range (e.g. 2744 vs 2704).
#12313 made the leader snap the follower scan `start_time` **down** to
the histogram bucket boundary in `RemoteScanNodes::get_remote_node()`,
to keep the first chart bucket fully populated. But that widens the
actual scan window: records in `[bucket_start, requested_start)` — data
the user explicitly excluded — get counted into the first bucket, while
`count(*)` (a separate query without `histogram_interval`) scans the
exact requested range. The reporter confirmed the diagnosis: aligning
`count(*)`'s start to the bucket grid makes the two numbers match.
#13896 was the same root cause surfacing on sub-10s ranges (worked
around in #13916 by dropping to 1s buckets there, without touching the
snap).
## Fix
Revert the snap: the scan window sent to followers now equals the
requested range exactly.
- `distributed_plan/node.rs`: `start_time` passes through unchanged; a
comment pins the invariant and a regression test
(`test_get_remote_node_keeps_raw_start_time_for_histogram`) asserts the
window is not widened when `histogram_interval > 0`.
- `sql/histogram.rs`: delete the now-unused `histogram_bucket_start()`
(zero remaining references, including in the enterprise repo).
- `sql/visitor/histogram_interval.rs`: reword two comments that were
written in terms of the snap. The 1s interval floor for sub-10s ranges
from #13916 is kept — a bucket wider than the whole range is bad
charting regardless of the snap.
Histogram semantics are now the same as Elasticsearch/Kibana
`date_histogram`: buckets only count in-range records, so `sum(buckets)
== count(*)` always holds, and the first bucket may be partial.
Safety notes:
- The tantivy `SimpleHistogram` fast path is unaffected: it aligns
bucket *labels* itself (`start_time - start_time % rounding_by`) and
only counts docs matched by the query's own time filter, consistent with
the `date_bin` path.
- The results cache is unaffected: `write_results` already aligns the
cache-accepted window *inward* and drops edge buckets, so partial (and
previously over-counted) first buckets never entered the cache.
## Trade-off / follow-up for the UI
The left-edge display artifact that motivated #12313 (first bar looks
lower because its bucket is cut by the range start) returns by design.
Suggested frontend handling, either:
- align the picker's `start_time` down to the returned
`histogram_interval` and issue **all** page queries (histogram + logs +
count) over that aligned range — Grafana's approach; totals stay
consistent and no partial bucket exists, or
- detect `buckets[0].key < start_time` and dim/annotate the first bar as
partial.
## Test plan
- `search` crate: 88 histogram tests + 9 node tests pass (includes the
new regression test); workspace clippy with CI flags passes.
- Single-node E2E: ingested 240 events at 500ms spacing over 120s;
queried with mid-bucket start/end (+17s/+107s), interval 30s. `count(*)`
= 180; buckets 26+60+60+34 = 180 (match). First bucket correctly holds
26 in-range records instead of the full 60 (old behavior summed to 214).
Paired with o2-enterprise `hf/issue-2491-admission-slot-leak` (same
branch name).
**Stacked on #13953** — base is that branch, so the diff here is just
this change.
Retargets to `main` automatically once #13953 merges.
## Problem
#13953 fixed the wal file locks. `do_get` touches two more trace-id
keyed
resources, and both had the identical defect: release depended on
control flow
reaching a call, inside the window where gRPC cancellation drops the
future.
**The node's admission slot.** Released from three hand-written sites —
the two
error arms in `do_get` and `FlightEncoderStream::drop`. All three are
skipped
when the request is cancelled during plan generation, because the
encoder
stream does not exist yet.
**The follower task status.** In the super-cluster path the entry was
inserted
before plan generation and removed after it:
```rust
if is_super_cluster && !SEARCH_SERVER.contain_key(&trace_id).await {
SEARCH_SERVER.insert(trace_id.clone(), TaskStatus::new_follower(vec![], false)).await;
}
let result = get_ctx_and_physical_plan(&trace_id, &req).await; // <-- cancellable
if is_super_cluster && !SEARCH_SERVER.is_leader(&trace_id).await {
SEARCH_SERVER.remove(&trace_id, false).await;
}
```
Cancel in the middle and the entry stays in the map for the life of the
process. Nothing else removes it: an explicit `cancel_query` would, but
a
timeout or a client disconnect would not.
## Approach
Both move onto the same ownership chain #13953 established.
`SessionGuard` — already armed on the first line of `do_get` and already
carried into `FlightEncoderStream` — now releases the admission slot
too, so
the three call sites collapse into its drop.
The follower task status is registered through an `AsyncDefer` that
lives in
the same scope as plan generation, so it is removed on every exit
including a
cancelled one. It is now removed only when this node registered it,
rather than
whenever the entry is not a leader's: a request can no longer clear an
entry it
did not create. In practice these coincide, since the composite
`{trace_id}-{job_id}`
is unique per request.
### Drive-by
The `physical plan generate error` and `physical plan execution error`
logs sat
*inside* `if get_o2_config().work_group.max_nodes_per_query > 0`, so a
plan
failure was silent unless work groups happened to be configured — and
silent in
every OSS build. They are unconditional now.
## Enterprise side
The paired PR gives `Running` reservations a TTL backstop. They had no
expiry
edge at all, so a slot whose release was lost — this cancellation path,
or a
leader that dies before sending `Release` — was held until the process
restarted. This PR closes the local path; the backstop covers the remote
one,
which no local guard can.
## Verification
- `cargo fmt --all` and the CI clippy command: clean
- Enterprise verified via the `Cargo.toml.openobserve` swap against the
paired
branch: `cargo check --all-targets` clean, and `cargo clippy --no-deps
-p openobserve-api-grpc --all-targets` clean. The workspace-wide
enterprise
clippy still reports the two pre-existing `cognitive_complexity` errors
in
`core/src/alerts/scheduler/handlers.rs` and `core/src/ofga/mod.rs`,
untouched
by this change
- `api-grpc` flight tests 21/21
## Also here
Three zh-CN strings whose English is *events* said 活动 (activity).
Unrelated to
the rest of the PR, carried along at review request.
| key | en-US | was | now |
|---|---|---|---|
| `home.docsCountLbl` | Events | 活动 | 事件 |
| `rum.searchEvents` | Search Events | 搜索活动 | 搜索事件 |
| `logs.noEvents.title` | No events found | 未找到任何活动 | 未找到任何事件 |
zh-TW already had 事件 for all three, and every other locale translates
them as
events. `logs.noEvents` also contradicted itself — `descNoFilters`
directly
below the title already said 未找到任何事件.
## Not in this PR
The `QueryManager` map has a `created_at` on every entry but no reaper.
With
this change its entries are removed on cancellation, so it is no longer
known
to grow — but it has no backstop the way the ledger now does.
Fixes o2-enterprise#2491. OSS-only — see "Enterprise side" below.
## Problem
Local wal parquet files that had already been merged and uploaded to
object
storage stayed on disk forever, because their `SEARCHING_FILES` refcount
never
returned to zero. The post-merge cleanup in
`jobs/src/job/files/parquet.rs`
checks `wal::lock_files_exists(file_key)`, defers the deletion when a
search
holds the file, and re-checks every `ZO_FILE_PUSH_INTERVAL` — forever,
if the
lock is never released. Queries stayed correct (`pending_delete::filter`
excludes those files from the search set); the symptom was disk.
The proximate cause is a missing decrement. The root cause is that
**release
depended on control flow reaching a line, while the only thing that can
cancel
the request guarantees no further line runs**.
### The unowned window
`SEARCHING_FILES` and `SEARCHING_REQUESTS` are global maps. Nothing in
the type
system points at an entry, so nothing releases one automatically:
```
lock_files() search_service/src/grpc/wal.rs, in get_file_list
| match_source, parquet footer reads, calculate_files_size, table build
lock_request() search_service/src/grpc/wal.rs
| pack segments, union table, plan rewrite, execute_stream_partitioned
FlightEncoderStreamBuilder::build() api/grpc .../flight/mod.rs <-- first owner with a Drop
```
`lock_request()` is not a handoff to an owner — `SEARCHING_REQUESTS` is
removed
from only by `clear_session_data`, itself reached by control flow. So
the whole
of plan generation was unprotected, not just the gap between the two
calls.
### Why that window is exactly the window cancellation lives in
Both gRPC cancellation paths can only fire *before* `do_get` returns a
response:
- `hyper-1.9.0/src/proto/h2/server.rs` polls `poll_reset` only on the
`Poll::Pending` branch — "check if the client has sent a RST_STREAM
frame
which would cancel the current request" — and drops the service future.
- `tonic-0.14.6/src/transport/service/grpc_timeout.rs` reads the
client's
`grpc-timeout` header and returns `Poll::Ready(Err(TimeoutExpired))`,
dropping
the inner future. Every flight request sets that header
(`make_flight_client` → `req.set_timeout(grpc_timeout)`).
Once the response stream exists, `FlightEncoderStream::drop` handles
teardown
and neither path applies. So the set of cancellable moments and the set
of
unowned moments coincided.
Routine triggers, all of which drop in-flight `do_get` calls to
ingesters:
`query_task.abort()` on leader timeout or user cancel
(`search_service/src/cluster/flight.rs`), `SearchStreamGuard::drop`
cancelling
on client disconnect, and the ingester `grpc-timeout` itself. None of
them are
visible: `remote_scan_exec.rs` treats `Cancelled` / `DeadlineExceeded`
from a
node as an empty partial result, so the query still succeeds.
## Approach
Give the locks an owner for every instant they exist.
- **`wal::SearchingFiles`** owns the raw locks from acquisition until
`into_request` hands them to `SEARCHING_REQUESTS`; `Drop` releases
whatever is
left. `get_file_list` returns it alongside the file list, so the type
forces
every early return to account for it.
- **`flight::SessionGuard`** is armed on the first line of `do_get` —
before
anything can be locked — owns `clear_session_data` for that trace id,
and
moves into `FlightEncoderStream`, whose `Drop` already performed it. It
is a
required argument of `FlightEncoderStreamBuilder::new`, not an optional
setter, so a future call site cannot forget it.
`lock_files`, `release_files` and `lock_request` are module-private now:
the
guard is the only way to take a wal lock. The eight hand-written
`release_files` call sites are gone, and `create_tables_from_files`
loses the
`on_error: F` generic that existed solely to run one of them —
`storage.rs` was
already passing `|| {}`.
Cancellation at any `.await` in the chain now releases:
| cancelled during | released by |
|---|---|
| `get_file_list`, metadata reads, table build | `SearchingFiles::drop`
|
| after `into_request`, before the stream exists | `SessionGuard::drop`
|
| streaming | `FlightEncoderStream::drop` → `SessionGuard::drop` |
### Others on the same chain
- `release_request` no longer early-returns on non-ingesters — the last
step of
an ownership chain should not be conditional — and logs only a release
that
actually happened, so `lock_request` / `release_request` now pair
one-to-one
per trace id and can be reconciled from logs.
- `lock_request` merges instead of overwriting; a second handoff for one
trace
id used to silently drop the first file list.
- `FlightEncoderStream::drop` releases session data unconditionally.
Letting
"am I a super-cluster follower leader?" decide whether wal locks are
released
was the same bug in miniature; the deferred lock only covers the
work-group
slot.
### Observability
Two gauges, both O(1) reads, published by the existing 60s ingester
collector
`update_parquet_metrics` (no stream label):
- `ingest_wal_searching_files` — size of the lock table
- `ingest_wal_pending_delete_files` — uploaded files still awaiting
deletion
A pending-delete backlog that rises and never drains is the leak,
plotted
directly. Previously the only signal was an *absent* `debug!` line.
## Enterprise side
**No paired PR — the fix is entirely in this repo.** The issue suspected
the
super-cluster follower-leader path, but `super_cluster/follower.rs` only
builds
a `RemoteScanExec` and never touches wal locks; `wal::lock_files` has
exactly
one call site, in `grpc/wal.rs`; and o2-enterprise contains no reference
to
`lock_files`, `release_files`, `lock_request` or `release_request`.
Verified by
a full enterprise build after making those three private.
## Verification
- `cargo fmt --all`, and the CI clippy command
(`--workspace --all-targets -W too_many_lines -W cognitive_complexity
-W excessive_nesting -D warnings`): clean
- Enterprise verified via the `Cargo.toml.openobserve` swap,
`cargo check --all-targets`: clean. Clippy on that build reports two
pre-existing `cognitive_complexity` errors in
`core/src/alerts/scheduler/handlers.rs`
and `core/src/ofga/mod.rs`, untouched by this change
- Unit tests: `common` wal 10/10 (4 new, covering drop-release,
duplicate
collapse, `release_one`, and the `into_request` handoff), `api-grpc`
flight
21/21, `search` file_list 4/4
## Not in this PR
- **A timestamped reaper for stale locks.** Redundant once the chain is
unbroken, but it is the lowest-risk first step when backporting to
`branch-v0.40.0` and `branch-v0.80.0`, which carry the same defect.
- **`admission::ledger` and the `SEARCH_SERVER` task status.** Both are
trace-id keyed and both release only by reaching a call, so a
cancellation in
the same window leaks a work-group slot the same way. Same bug class,
different resource, enterprise-gated — worth its own issue.
- **`clean_lock_files()`** (`/node/flush`, `/node/enable`) wipes the
whole
refcount table, including locks held by running queries. It was
survivable
while locks leaked; now it turns a leak into deleting a file a query is
reading. Retiring it is a product call.
## Problem
`api_fuzz_tests` (schemathesis) intermittently reported a 500 on
`POST /api/{org_id}/streams/{stream_name}` with the body
`failed to parse header value`.
That string is not ours — it is the `http` crate's `InvalidHeaderValue`
display. The chain:
- ~20 sites return `(StatusCode, [(ERROR_HEADER, format!(...))],
Json(...))`,
where the message interpolates user-supplied names (org, stream, **field
type**).
- The failing payload carried `"type": "v^0K…"`. `HeaderValue` accepts
only `b >= 32 && b != 127 || b == b'\t'`, so the control byte makes
header
construction fail.
- axum's `TryIntoHeaderError` then **discards the whole response** and
substitutes `(500, inner.to_string())`.
So the handler's real 400 and its JSON body never reached the client.
The
status code was not chosen by the handler at all — axum overwrote it.
The org name was never the trigger; it only had to exist so the request
got
past the auth check. The field type was the poison.
A second, quieter bug sits in the same place: non-ascii bytes *are*
accepted
into a header, but `HeaderValue::to_str()` only reads back `32..127`,
and the
audit middleware reads this header with exactly `to_str().ok()`. Any
error
mentioning a non-ascii stream name was being silently dropped from the
audit
log.
## Approach
Route every `X-Error-Message` write through one chokepoint in
`common::meta::http`:
- `error_header_value()` escapes anything outside visible ascii via
`char::escape_default`, which fixes both failure modes at once — the
header
always builds, and it always reads back.
- `HttpResponse::error_with_header()` sets the header and the JSON body
from a
single message, replacing the hand-built tuples.
Most sites were formatting the identical message twice (once for the
header,
once for the body), so this removes that duplication as well.
`enrichment_table` additionally had four
`HeaderValue::from_str(..).unwrap()`
calls — outright panics, not 500s, on the same class of input. Those are
gone.
Other `HeaderValue` construction sites (router `user_email`,
`x-api-node`)
already guard with `unwrap_or_else` and are untouched.
## Organization identifier validation
Investigating how such an org existed at all: `create_org` validates the
display name and assigns a uuid identifier, but
`core::organization::check_and_create_org` takes the identifier
**verbatim from
the URL path** with no validation. It is reachable from any
root-authenticated
ingestion write (`ZO_CREATE_ORG_THROUGH_INGESTION`, default true), which
is how
the fuzzer provisioned it.
`validate_auto_created_org_id` (ascii alphanumeric plus `-` and `_`) now
guards
both auto-create entry points. It runs **after** the existing-org
lookup, so it
only gates new creation: no deployment loses an org it already has, and
SSO
auto-provisioning from IdP group names keeps working for normal
identifiers.
## Verification
Against a local server with the CI payload:
- stream create → `400 {"code":400,"message":"invalid data type:
v^0K"}`
instead of the 500
- ingestion to the exotic org → 404; `autoorg-1` still auto-creates and
ingests
- valid create, duplicate create, settings update, settings on a missing
stream,
delete, enrichment upload → correct codes, no panics
Two edits sit under `#[cfg(feature = "cloud")]` and cannot be compiled
here:
swapping in `Cargo.toml.openobserve` fails inside o2-enterprise itself,
whose
`main` has not yet picked up the `get_orm_client_rw` split from #13927.
They
were verified by temporarily un-gating them and stubbing the cloud-only
predicate.
Design At: designs/dashboards/interactive-table-cell-drilldown
What changed
New component: DashboardLogDrawer.vue
Clicking a cell in a dashboard table panel now opens a side drawer
scoped to that field/value filter. The drawer shows filtered log events
with a SQL editor (Ctrl+Enter to run, Reset to restore default),
server-side pagination with configurable page size (50/100/200/500), and
an event detail sub-drawer on row click.
Event detail has three tabs:
Insights — Field Anomaly Profile (rarity scoring vs current page), Event
Timeline (SVG histogram of event frequency across the selected time
range), Log Patterns (grouped by structure via /patterns/extract, each
with "Open in Logs" that prefills the log explorer), and Surrounding
Events (events from the same stream in a configurable ±1–10 min window,
paginated 20/page with per-row expand)
Details — key/value field list with copy buttons
JSON — raw event JSON
Modified: PanelSchemaRenderer.vue
Fixed type mismatch: route.query.cell_panel returns string; panel schema
IDs may be number — comparison now uses String() on both sides
Added cell_event_ts to the URL param cleanup list so it's cleared when
the outer log drawer closes
Modified: en-US.json — added complete panel.logExplorer key tree
including the new insights subsection
Modified: design-debt-baseline.json — updated baseline after audit (new
file's rawVarInComponent uses are all in sanctioned color-mix()
positions)
URL persistence
All drawer state is encoded in URL params (cell_panel, cell_field,
cell_value, cell_stream, cell_stype, cell_t0, cell_t1, cell_event_ts).
Sharing the URL reopens both drawers and restores the exact event — on
mount it does a WHERE _timestamp = {ts} targeted fetch (size 1) rather
than searching through paginated results.
Zero backend changes
Uses only existing endpoints: search (SQL), search (histogram via
histogram() function), and patterns/extract.
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
Co-authored-by: Abhay Padamani <141122205+ktx-abhay@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Paired with o2-enterprise `refactor/orm-rw-ro-client` (same branch
name).
## Problem
Every ORM caller went through a single client backed by the sqlite
**write** pool, so writes were kept apart only by `table::get_lock()` —
a convention 365 call sites had to remember. Forgetting it surfaced as
`(code: 517) database is locked`: a deferred transaction that reads and
then upgrades to a write finds the snapshot moved under it, and sqlite
reports that **without ever consulting `busy_timeout`**, which is why
the failures were immediate rather than retried.
## Approach
Serialization now comes from the pool, not from a lock. The sqlite write
pool holds **one connection** — all sqlite allows anyway — so writers
queue on acquire and there is nothing to forget.
That is only affordable because reads move to the read-only pool.
Measured, same load, fresh data dir each run:
| | reads on the write connection | reads on the RO pool |
|---|---|---|
| 200 concurrent reads | p50 **135 ms** | p50 **1 ms** |
| writes during mixed load | p50 **189 ms** | p50 **14 ms** |
Both were 100% success with zero lock errors, so this is purely
throughput — but without the split, reads queue behind the single writer
and drag writes down with them.
## API
`infra::db` now exposes three functions and nothing else:
```rust
pub async fn get_orm_client_ro() -> &'static DatabaseConnection
pub async fn get_orm_client_rw() -> &'static DatabaseConnection
pub async fn get_orm_client_ddl() -> &'static DatabaseConnection
```
The statics behind them, every `connect_to_orm_*`, and the raw
sqlite/postgres pools are now private or `pub(crate)`. `ORM_CLIENT`,
`connect_to_orm` and `table::get_lock()` are gone, as is the raw
`CLIENT_RW` mutex (56 lock acquisitions, plus the 52 guard derefs that
existed only to reach through it).
Final split: **RO 247 / RW 360 / DDL 9**; `get_lock` 0, `ORM_CLIENT` 0,
`CLIENT_RW.lock()` 0.
## How read/write was decided
Per call site, then verified by resolving each call through **its own
file's imports** (bare-name matching is useless here — `get_by_id`,
`list` and `update` collide across modules) and propagating write-ness
along the edges that hand a connection to a callee.
A wrong write client only costs throughput; a wrong read client fails in
production. So the audit is zero-miss in the read direction, and a site
is only downgraded when *every* connection-passing callee resolves.
## Behaviour changes on sqlite
All three make sqlite match what postgres already does today — postgres
has never had this lock:
- check-then-write pairs like `folders::put` are no longer atomic; they
rely on their unique index (`folders_org_folder_id_idx`)
- `db::put` runs two sequential transactions, so concurrent puts to one
key are last-write-wins
- `db::delete` runs several statements outside a transaction
`write_pool_is_single_connection_on_sqlite` is now the **only** guard
against reintroducing 517 — please don't drop it.
## Review guide
The mutex removal is separable if you want to look at it on its own:
`src/infra/src/{db,file_list,scheduler,pipeline,schema/history}/sqlite.rs`.
Everything else is the client split.
## Verification
- `cargo check --workspace --all-targets` and the CI clippy command:
clean
- Tests match a HEAD baseline built with the same `.env`, exactly: infra
1410/2, core 1629/1, db 431/0, api-management 179/0. The three failures
are pre-existing and environmental (two need a postgres that isn't
running locally)
- Static scans, all zero: queries on the pool inside an open
transaction; transactions spanning a second connection acquisition;
ro/rw misclassification
- Runtime smoke on a fresh data dir: 200 concurrent reads, then 100
reads + 100 writes simultaneously — 400/400 HTTP 200, rows persisted,
**0** lock errors, 0 ERROR/WARN lines
- Enterprise verified via the `Cargo.toml.openobserve` swap (`cargo
check --all-targets`): clean
Also fixes `users::tests::set_up`, which truncated shared tables but
released its lock before the test body ran, letting a concurrent test
clear rows mid-test.
https://claude.ai/code/session_016ouf1RxiMpiRw5W7GvCUDh
Amey PawarandHengfei Yang
udevandHengfei Yang
Huaijin
Harsh Mahajan
Prabhat Sharma
Vaibhav Mittal
BhargavandHengfei Yang
Ashish Kolhe
0xfandomandHengfei Yang
Mohd Quamar TyagiandHengfei Yang
