Paired with o2-enterprise `hf/issue-2491-admission-slot-leak` (same
branch name).
**Stacked on #13953** — base is that branch, so the diff here is just
this change.
Retargets to `main` automatically once #13953 merges.
## Problem
#13953 fixed the wal file locks. `do_get` touches two more trace-id
keyed
resources, and both had the identical defect: release depended on
control flow
reaching a call, inside the window where gRPC cancellation drops the
future.
**The node's admission slot.** Released from three hand-written sites —
the two
error arms in `do_get` and `FlightEncoderStream::drop`. All three are
skipped
when the request is cancelled during plan generation, because the
encoder
stream does not exist yet.
**The follower task status.** In the super-cluster path the entry was
inserted
before plan generation and removed after it:
```rust
if is_super_cluster && !SEARCH_SERVER.contain_key(&trace_id).await {
SEARCH_SERVER.insert(trace_id.clone(), TaskStatus::new_follower(vec![], false)).await;
}
let result = get_ctx_and_physical_plan(&trace_id, &req).await; // <-- cancellable
if is_super_cluster && !SEARCH_SERVER.is_leader(&trace_id).await {
SEARCH_SERVER.remove(&trace_id, false).await;
}
```
Cancel in the middle and the entry stays in the map for the life of the
process. Nothing else removes it: an explicit `cancel_query` would, but
a
timeout or a client disconnect would not.
## Approach
Both move onto the same ownership chain #13953 established.
`SessionGuard` — already armed on the first line of `do_get` and already
carried into `FlightEncoderStream` — now releases the admission slot
too, so
the three call sites collapse into its drop.
The follower task status is registered through an `AsyncDefer` that
lives in
the same scope as plan generation, so it is removed on every exit
including a
cancelled one. It is now removed only when this node registered it,
rather than
whenever the entry is not a leader's: a request can no longer clear an
entry it
did not create. In practice these coincide, since the composite
`{trace_id}-{job_id}`
is unique per request.
### Drive-by
The `physical plan generate error` and `physical plan execution error`
logs sat
*inside* `if get_o2_config().work_group.max_nodes_per_query > 0`, so a
plan
failure was silent unless work groups happened to be configured — and
silent in
every OSS build. They are unconditional now.
## Enterprise side
The paired PR gives `Running` reservations a TTL backstop. They had no
expiry
edge at all, so a slot whose release was lost — this cancellation path,
or a
leader that dies before sending `Release` — was held until the process
restarted. This PR closes the local path; the backstop covers the remote
one,
which no local guard can.
## Verification
- `cargo fmt --all` and the CI clippy command: clean
- Enterprise verified via the `Cargo.toml.openobserve` swap against the
paired
branch: `cargo check --all-targets` clean, and `cargo clippy --no-deps
-p openobserve-api-grpc --all-targets` clean. The workspace-wide
enterprise
clippy still reports the two pre-existing `cognitive_complexity` errors
in
`core/src/alerts/scheduler/handlers.rs` and `core/src/ofga/mod.rs`,
untouched
by this change
- `api-grpc` flight tests 21/21
## Also here
Three zh-CN strings whose English is *events* said 活动 (activity).
Unrelated to
the rest of the PR, carried along at review request.
| key | en-US | was | now |
|---|---|---|---|
| `home.docsCountLbl` | Events | 活动 | 事件 |
| `rum.searchEvents` | Search Events | 搜索活动 | 搜索事件 |
| `logs.noEvents.title` | No events found | 未找到任何活动 | 未找到任何事件 |
zh-TW already had 事件 for all three, and every other locale translates
them as
events. `logs.noEvents` also contradicted itself — `descNoFilters`
directly
below the title already said 未找到任何事件.
## Not in this PR
The `QueryManager` map has a `created_at` on every entry but no reaper.
With
this change its entries are removed on cancellation, so it is no longer
known
to grow — but it has no backstop the way the ledger now does.
Fixes o2-enterprise#2491. OSS-only — see "Enterprise side" below.
## Problem
Local wal parquet files that had already been merged and uploaded to
object
storage stayed on disk forever, because their `SEARCHING_FILES` refcount
never
returned to zero. The post-merge cleanup in
`jobs/src/job/files/parquet.rs`
checks `wal::lock_files_exists(file_key)`, defers the deletion when a
search
holds the file, and re-checks every `ZO_FILE_PUSH_INTERVAL` — forever,
if the
lock is never released. Queries stayed correct (`pending_delete::filter`
excludes those files from the search set); the symptom was disk.
The proximate cause is a missing decrement. The root cause is that
**release
depended on control flow reaching a line, while the only thing that can
cancel
the request guarantees no further line runs**.
### The unowned window
`SEARCHING_FILES` and `SEARCHING_REQUESTS` are global maps. Nothing in
the type
system points at an entry, so nothing releases one automatically:
```
lock_files() search_service/src/grpc/wal.rs, in get_file_list
| match_source, parquet footer reads, calculate_files_size, table build
lock_request() search_service/src/grpc/wal.rs
| pack segments, union table, plan rewrite, execute_stream_partitioned
FlightEncoderStreamBuilder::build() api/grpc .../flight/mod.rs <-- first owner with a Drop
```
`lock_request()` is not a handoff to an owner — `SEARCHING_REQUESTS` is
removed
from only by `clear_session_data`, itself reached by control flow. So
the whole
of plan generation was unprotected, not just the gap between the two
calls.
### Why that window is exactly the window cancellation lives in
Both gRPC cancellation paths can only fire *before* `do_get` returns a
response:
- `hyper-1.9.0/src/proto/h2/server.rs` polls `poll_reset` only on the
`Poll::Pending` branch — "check if the client has sent a RST_STREAM
frame
which would cancel the current request" — and drops the service future.
- `tonic-0.14.6/src/transport/service/grpc_timeout.rs` reads the
client's
`grpc-timeout` header and returns `Poll::Ready(Err(TimeoutExpired))`,
dropping
the inner future. Every flight request sets that header
(`make_flight_client` → `req.set_timeout(grpc_timeout)`).
Once the response stream exists, `FlightEncoderStream::drop` handles
teardown
and neither path applies. So the set of cancellable moments and the set
of
unowned moments coincided.
Routine triggers, all of which drop in-flight `do_get` calls to
ingesters:
`query_task.abort()` on leader timeout or user cancel
(`search_service/src/cluster/flight.rs`), `SearchStreamGuard::drop`
cancelling
on client disconnect, and the ingester `grpc-timeout` itself. None of
them are
visible: `remote_scan_exec.rs` treats `Cancelled` / `DeadlineExceeded`
from a
node as an empty partial result, so the query still succeeds.
## Approach
Give the locks an owner for every instant they exist.
- **`wal::SearchingFiles`** owns the raw locks from acquisition until
`into_request` hands them to `SEARCHING_REQUESTS`; `Drop` releases
whatever is
left. `get_file_list` returns it alongside the file list, so the type
forces
every early return to account for it.
- **`flight::SessionGuard`** is armed on the first line of `do_get` —
before
anything can be locked — owns `clear_session_data` for that trace id,
and
moves into `FlightEncoderStream`, whose `Drop` already performed it. It
is a
required argument of `FlightEncoderStreamBuilder::new`, not an optional
setter, so a future call site cannot forget it.
`lock_files`, `release_files` and `lock_request` are module-private now:
the
guard is the only way to take a wal lock. The eight hand-written
`release_files` call sites are gone, and `create_tables_from_files`
loses the
`on_error: F` generic that existed solely to run one of them —
`storage.rs` was
already passing `|| {}`.
Cancellation at any `.await` in the chain now releases:
| cancelled during | released by |
|---|---|
| `get_file_list`, metadata reads, table build | `SearchingFiles::drop`
|
| after `into_request`, before the stream exists | `SessionGuard::drop`
|
| streaming | `FlightEncoderStream::drop` → `SessionGuard::drop` |
### Others on the same chain
- `release_request` no longer early-returns on non-ingesters — the last
step of
an ownership chain should not be conditional — and logs only a release
that
actually happened, so `lock_request` / `release_request` now pair
one-to-one
per trace id and can be reconciled from logs.
- `lock_request` merges instead of overwriting; a second handoff for one
trace
id used to silently drop the first file list.
- `FlightEncoderStream::drop` releases session data unconditionally.
Letting
"am I a super-cluster follower leader?" decide whether wal locks are
released
was the same bug in miniature; the deferred lock only covers the
work-group
slot.
### Observability
Two gauges, both O(1) reads, published by the existing 60s ingester
collector
`update_parquet_metrics` (no stream label):
- `ingest_wal_searching_files` — size of the lock table
- `ingest_wal_pending_delete_files` — uploaded files still awaiting
deletion
A pending-delete backlog that rises and never drains is the leak,
plotted
directly. Previously the only signal was an *absent* `debug!` line.
## Enterprise side
**No paired PR — the fix is entirely in this repo.** The issue suspected
the
super-cluster follower-leader path, but `super_cluster/follower.rs` only
builds
a `RemoteScanExec` and never touches wal locks; `wal::lock_files` has
exactly
one call site, in `grpc/wal.rs`; and o2-enterprise contains no reference
to
`lock_files`, `release_files`, `lock_request` or `release_request`.
Verified by
a full enterprise build after making those three private.
## Verification
- `cargo fmt --all`, and the CI clippy command
(`--workspace --all-targets -W too_many_lines -W cognitive_complexity
-W excessive_nesting -D warnings`): clean
- Enterprise verified via the `Cargo.toml.openobserve` swap,
`cargo check --all-targets`: clean. Clippy on that build reports two
pre-existing `cognitive_complexity` errors in
`core/src/alerts/scheduler/handlers.rs`
and `core/src/ofga/mod.rs`, untouched by this change
- Unit tests: `common` wal 10/10 (4 new, covering drop-release,
duplicate
collapse, `release_one`, and the `into_request` handoff), `api-grpc`
flight
21/21, `search` file_list 4/4
## Not in this PR
- **A timestamped reaper for stale locks.** Redundant once the chain is
unbroken, but it is the lowest-risk first step when backporting to
`branch-v0.40.0` and `branch-v0.80.0`, which carry the same defect.
- **`admission::ledger` and the `SEARCH_SERVER` task status.** Both are
trace-id keyed and both release only by reaching a call, so a
cancellation in
the same window leaks a work-group slot the same way. Same bug class,
different resource, enterprise-gated — worth its own issue.
- **`clean_lock_files()`** (`/node/flush`, `/node/enable`) wipes the
whole
refcount table, including locks held by running queries. It was
survivable
while locks leaked; now it turns a leak into deleting a file a query is
reading. Retiring it is a product call.
Design At: designs/dashboards/interactive-table-cell-drilldown
What changed
New component: DashboardLogDrawer.vue
Clicking a cell in a dashboard table panel now opens a side drawer
scoped to that field/value filter. The drawer shows filtered log events
with a SQL editor (Ctrl+Enter to run, Reset to restore default),
server-side pagination with configurable page size (50/100/200/500), and
an event detail sub-drawer on row click.
Event detail has three tabs:
Insights — Field Anomaly Profile (rarity scoring vs current page), Event
Timeline (SVG histogram of event frequency across the selected time
range), Log Patterns (grouped by structure via /patterns/extract, each
with "Open in Logs" that prefills the log explorer), and Surrounding
Events (events from the same stream in a configurable ±1–10 min window,
paginated 20/page with per-row expand)
Details — key/value field list with copy buttons
JSON — raw event JSON
Modified: PanelSchemaRenderer.vue
Fixed type mismatch: route.query.cell_panel returns string; panel schema
IDs may be number — comparison now uses String() on both sides
Added cell_event_ts to the URL param cleanup list so it's cleared when
the outer log drawer closes
Modified: en-US.json — added complete panel.logExplorer key tree
including the new insights subsection
Modified: design-debt-baseline.json — updated baseline after audit (new
file's rawVarInComponent uses are all in sanctioned color-mix()
positions)
URL persistence
All drawer state is encoded in URL params (cell_panel, cell_field,
cell_value, cell_stream, cell_stype, cell_t0, cell_t1, cell_event_ts).
Sharing the URL reopens both drawers and restores the exact event — on
mount it does a WHERE _timestamp = {ts} targeted fetch (size 1) rather
than searching through paginated results.
Zero backend changes
Uses only existing endpoints: search (SQL), search (histogram via
histogram() function), and patterns/extract.
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
Co-authored-by: Abhay Padamani <141122205+ktx-abhay@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Paired with o2-enterprise `refactor/orm-rw-ro-client` (same branch
name).
## Problem
Every ORM caller went through a single client backed by the sqlite
**write** pool, so writes were kept apart only by `table::get_lock()` —
a convention 365 call sites had to remember. Forgetting it surfaced as
`(code: 517) database is locked`: a deferred transaction that reads and
then upgrades to a write finds the snapshot moved under it, and sqlite
reports that **without ever consulting `busy_timeout`**, which is why
the failures were immediate rather than retried.
## Approach
Serialization now comes from the pool, not from a lock. The sqlite write
pool holds **one connection** — all sqlite allows anyway — so writers
queue on acquire and there is nothing to forget.
That is only affordable because reads move to the read-only pool.
Measured, same load, fresh data dir each run:
| | reads on the write connection | reads on the RO pool |
|---|---|---|
| 200 concurrent reads | p50 **135 ms** | p50 **1 ms** |
| writes during mixed load | p50 **189 ms** | p50 **14 ms** |
Both were 100% success with zero lock errors, so this is purely
throughput — but without the split, reads queue behind the single writer
and drag writes down with them.
## API
`infra::db` now exposes three functions and nothing else:
```rust
pub async fn get_orm_client_ro() -> &'static DatabaseConnection
pub async fn get_orm_client_rw() -> &'static DatabaseConnection
pub async fn get_orm_client_ddl() -> &'static DatabaseConnection
```
The statics behind them, every `connect_to_orm_*`, and the raw
sqlite/postgres pools are now private or `pub(crate)`. `ORM_CLIENT`,
`connect_to_orm` and `table::get_lock()` are gone, as is the raw
`CLIENT_RW` mutex (56 lock acquisitions, plus the 52 guard derefs that
existed only to reach through it).
Final split: **RO 247 / RW 360 / DDL 9**; `get_lock` 0, `ORM_CLIENT` 0,
`CLIENT_RW.lock()` 0.
## How read/write was decided
Per call site, then verified by resolving each call through **its own
file's imports** (bare-name matching is useless here — `get_by_id`,
`list` and `update` collide across modules) and propagating write-ness
along the edges that hand a connection to a callee.
A wrong write client only costs throughput; a wrong read client fails in
production. So the audit is zero-miss in the read direction, and a site
is only downgraded when *every* connection-passing callee resolves.
## Behaviour changes on sqlite
All three make sqlite match what postgres already does today — postgres
has never had this lock:
- check-then-write pairs like `folders::put` are no longer atomic; they
rely on their unique index (`folders_org_folder_id_idx`)
- `db::put` runs two sequential transactions, so concurrent puts to one
key are last-write-wins
- `db::delete` runs several statements outside a transaction
`write_pool_is_single_connection_on_sqlite` is now the **only** guard
against reintroducing 517 — please don't drop it.
## Review guide
The mutex removal is separable if you want to look at it on its own:
`src/infra/src/{db,file_list,scheduler,pipeline,schema/history}/sqlite.rs`.
Everything else is the client split.
## Verification
- `cargo check --workspace --all-targets` and the CI clippy command:
clean
- Tests match a HEAD baseline built with the same `.env`, exactly: infra
1410/2, core 1629/1, db 431/0, api-management 179/0. The three failures
are pre-existing and environmental (two need a postgres that isn't
running locally)
- Static scans, all zero: queries on the pool inside an open
transaction; transactions spanning a second connection acquisition;
ro/rw misclassification
- Runtime smoke on a fresh data dir: 200 concurrent reads, then 100
reads + 100 writes simultaneously — 400/400 HTTP 200, rows persisted,
**0** lock errors, 0 ERROR/WARN lines
- Enterprise verified via the `Cargo.toml.openobserve` swap (`cargo
check --all-targets`): clean
Also fixes `users::tests::set_up`, which truncated shared tables but
released its lock before the test body ran, letting a concurrent test
clear rows mid-test.
https://claude.ai/code/session_016ouf1RxiMpiRw5W7GvCUDh
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## What this is
Database Monitoring: an aggregation and presentation layer over
telemetry OpenObserve already ingests. It reads three vantages — client
spans, server-vantage logs, and collector metrics — and turns them into
per-statement, per-instance and per-database views.
No new storage engine and no new ingest path. Statements are normalized
and rolled up into a stats stream, and the read API queries that stream
through the existing search path.
## Shape of the change
| | |
|---|---|
| Read endpoints | 17 |
| UI pages | 8 (plus a shell) |
| Commits | 141 |
**Backend** — `src/core/src/db_monitoring/` holds the non-HTTP half
(normalization, rollup, server-vantage parsing). The read API lives in
`src/api/management/src/request/db_monitoring/` as a handler / models /
service trio; the service layer is split one file per feature.
**Frontend** — `web/src/views/DatabaseMonitoring/`, with shared logic in
`web/src/composables/dbm/`.
## Feature availability
Databases, queries, query detail and history, endpoints, samples,
activity, insights, plans, server metrics, server queries, server
samples, instances, instance metrics, badges — is available in OSS.
## Reviewing this
It is a large diff, and most of it is additive: 327 files, and the two
largest non-test files are the shared service module and the deadlocks
service. Suggested order:
1. `src/api/management/src/request/db_monitoring/handler.rs` — the 17
entry points and their gating
2. `src/api/management/src/request/db_monitoring/service/common.rs` —
auth, SQL building, the shared read path
3. `src/core/src/db_monitoring/rollup.rs` — how the stats stream is
produced
4. `web/src/views/DatabaseMonitoring/` — the pages
Time semantics are the subtlest part. Rollup rows are stamped at the
window **end**, so reads are exclusive at both edges; `stats_read_range`
carries the reasoning and a test pins it.
## Known follow-ups
- **The server-vantage stream named as `_o2_dbm_server`.**
## Testing
- Rust: unit tests across the DBM core and read service, including
SQL-building, auth-gating and normalizer cases
- Web: unit tests across the DBM pages and composables
- Playwright: DBM suite covering the pages and cross-page navigation
Verified before opening: OSS and enterprise builds both compile
warning-free, `cargo fmt` clean, `vue-tsc` clean, and the full web unit
suite passes (44,704 tests).
## Problem
`GET /config` is unauthenticated by design
## Fix
Split the endpoint:
- **`GET /config` (unauthenticated)** now returns only the login-page
bootstrap: `sso_enabled`, `native_login_enabled`, the four custom-logo
branding fields, the `rum` init block, `telemetry_enabled`,
`build_type`, and `commit_hash` (kept so the UI's stale-build detection
keeps working; version, license, billing orgs, storage region, etc. no
longer leak). An **exact-key-set test** pins this surface so a new field
cannot land on the unauthenticated endpoint unreviewed.
- **`GET /api/{org_id}/config` (authenticated)** serves the full
configuration, minus `instance`, which is dropped entirely. It goes
through `auth_middleware` like every service route and is classified as
a querier route, matching where `/config` was served before.
- The web app fetches the bootstrap at boot and the full config from
MainLayout once an org is known (with a watcher for fresh sessions), and
the logo-settings / Usage-tab refresh paths use the authenticated
endpoint.
## Paired PR
`o2-enterprise` branch `fix/config-api-exposure` adds the
`ROUTE_PERMISSIONS` entry (`bypass: true`, like the marketplace routes)
so any authenticated org member can fetch `GET /{org}/config` past
OpenFGA.
## Verification
- Router tests: bootstrap exact-key-set, `/config/reload` still 401,
`/{org_id}/config` dispatch + no-`instance` (3/3)
- `cargo test` for config / api-http / api-management crates: green
- Targeted web unit tests (config service, buildVersionChecker, General
settings): green
- Enterprise build verified via the `Cargo.toml.openobserve` swap:
`cargo check --all-targets` clean
- `o2_openfga` route-resolution tests for the new entry: green
- `cargo fmt --check` clean; CI clippy (`--workspace --all-targets -- -D
warnings`) clean
The query half of the TSID-major layout: #13882 wrote `tsid-major-v3-*`
files ordered by `(__hash__, _timestamp)`; this PR adds the `.midx`
series index next to them and makes PromQL prune through it. Supersedes
the remaining scope of #13839.
## Write side
- Every `tsid-major-v3-*` file written by the hour-end merge gets a
`.midx` series index (Arrow IPC + zstd), stored like the Tantivy index
under a derived root distinct from the data tree:
`files/{org}/metrics/{stream}/{date}/{hour}/tsid-major-v3-{id}.parquet`
→ `files/{org}/midx/{stream}/{date}/{hour}/tsid-major-v3-{id}.midx`
(Tantivy: `files/{org}/index/{stream}_{type}/…/{id}.ttv`): **one row per
contiguous TSID run** — its `row_count: u32` plus the label columns of
the run's first row (source types preserved). Runs tile the data file,
so each run's starting row is the prefix sum of the preceding counts and
is not stored; `__hash__` is excluded (matchers never reference it). The
whole index is one ZSTD-compressed IPC batch. Runs crossing an input
RecordBatch boundary appear as two adjacent entries; query-time range
coalescing makes that representation equivalent without buffering
samples. File and row-group boundaries stay independent of TSID
boundaries.
- `TsidSeriesIndexWriter` lives inside
`search/datafusion/merge/tsid_major.rs` and collects run entries
alongside the Parquet writer (one buffered row per run), encoding and
compressing once at finish. `MergedFile` carries `series_index:
Option<Vec<u8>>`; the compactor uploads it to the `midx/` tree
(compliance mode included).
- `.midx` is intentionally **not** tracked in `file_list`; its path is
derived from the data file's key
(`MetricsFileLayout::series_index_path`) and it is deleted together with
the parent in the delayed-deletion pass (`deleted.rs`); like the
`index/` tree, the `midx/` day directories are covered by retention's
local-disk deletion and the gc tool.
## Query side (PromQL)
- `promql_service::…::tsid_series_index::search` runs where Tantivy used
to, before the metrics table is registered: for each TSID-major file it
loads the `.midx` (concurrent, `2×target_partitions ≤ 64`), projects the
row-range columns plus the matched labels **by name against each series
index's own schema** (label sets / column orders differ between files),
builds the matcher filter **per series index** with
`physical_expr::create_physical_expr` — the exact same predicates as
`apply_matchers` (`matcher_predicates`) — evaluates it over the run
rows, coalesces adjacent/overlapping ranges and attaches them as
`FileSelection::RowRanges` (converted into a Parquet access plan by the
generic helper from #13861, no per-row bitmap).
- **Applied per file**: files of any other layout in the same query
(still-open hours, legacy data, late arrivals) are left in place and
scanned in full. A matched label missing from a series index
over-selects that file; a query whose matchers cannot be evaluated at
all selects the whole file; any series index failure logs and falls back
to a full scan. The final PromQL matchers are always applied, so results
are exact in every case.
- Evaluated selections are cached (LRU, 256 MiB) keyed by account +
series index path + the full matcher set.
- The Tantivy path in the PromQL storage scan is removed
(`convert_matchers_to_index_condition`, index-field plumbing,
`is_add_filter_back` bookkeeping). SQL search is untouched.
## Benchmark (from the #13839 POC, same layout + series index)
478,144,896 rows, 7 CPUs, one-hour PromQL range, step 15s, result cache
off, release + mimalloc:
| Layout | Filter steady median | No-filter steady median | Data files |
Total storage |
|---|---:|---:|---:|---:|
| main, 2 GiB | 695.709 ms | 4.638 s | 88 | 4,925.81 MiB |
| TSID-major + `.midx` | 269.496 ms | 2.793 s | 80 | 1,294.61 MiB |
## Validation
- `cargo fmt --check`; `cargo check` / `clippy` on config, search,
compaction, promql-service, jobs, search_service (`--tests`) clean
- `config tsid_layout` 3 passed (incl. `series_index_path`); `search
tsid_major mode:: merge::` 10 passed (rotation test now also asserts the
series index, exact-run and coalescing series index tests); `compaction
merge deleted` 33 passed; `promql-service` 20 passed + 1 pre-existing
wall-clock flake (`cache::test_promql_cache_set_and_get`, fails
intermittently on main too)
- End-to-end on the #13839 branch (same code paths): hour-end merge
writes N `tsid-major-v3-*` + `.midx`; PromQL takes the series index path
(`selected 39 ranges across 4/4 files`), counts/sums match the synthetic
data, cold-cache identical after restart; a mixed hour (3 major + 1 late
`tsid-sorted`) reports `1 files without a series index left to
tantivy/full scan` with unchanged results; delayed deletion removes
`.parquet` **and** `.midx`
> ### ⚠️ This PR carries four changes — review alongside
openobserve/o2-enterprise#2420
>
> 1. **Super-cluster support (consumer half)** — `8ceb041c43 …
5885c57b3a`.
> 2. **Synthetics OSS migration** — `3a380087a3 … 2596300dbf`, nine
commits — one per build-order step, plus a ninth for fixes found by
running the binary. Moves synthetics out of `o2_enterprise` into
`src/synthetics` (`openobserve-synthetics`), leaving only the
private-VPC-agent path enterprise: `job_api::lease`, `agent/*`, private
locations and the agent-setup install flow. Paired with
openobserve/o2-enterprise#2420.
>
> 3. **Last-check columns read from the results stream** — `6af411e808`,
`e7a3f2c759`. Outside the region whose scheduler runs a check, the list
showed a stale status and a blank Last check / response time:
`last_check_status` replicates only on change, `last_triggered_at` is a
runtime column that does not replicate, and `last_response_ms` was never
populated anywhere. The `synthetics_results` stream already carries all
three and is federated, so the list API reads it instead of replicating
the columns — no queue traffic, no schema change. Best-effort: any miss
keeps the stored DB values, so search being unavailable cannot take the
list down. OSS-only; no enterprise counterpart.
>
> 4. **Check secrets are readable in every region** — `d47a6b2345` (+
o2-enterprise `da71d7ec`). The per-org DEK is minted by each region for
itself, so a check created outside the scheduler region replicated fine
and then silently never ran — `resolve` could not decrypt its
credentials (o2-enterprise#2451). Secrets now cross the queue **in
clear** and the applier re-encrypts them under its own DEK on arrival,
fixing the class at the replication boundary with no key to distribute
or keep in sync.
>
> ⚠️ **This puts customer secrets in plaintext on the inter-region bus**
— a deliberate, recorded trade-off (`o2oi_` ingest tokens and probe
tokens already travel it in clear). Securing the bus itself (TLS on the
gateway links, short retention on secret-carrying subjects) is the real
fix and is **not** in this PR.
>
> ⚠️ **Deploy consumers before producers.** The payload's bytes did not
change, only the meaning of the secret strings, so an un-upgraded
consumer handed the new payload would write customer passwords to disk
in clear. The plaintext-carrying messages therefore use new variant
names (old ones retired as `CreateEncrypted`/`UpdateEncrypted`), so an
old region fails serde and redelivers until upgraded instead of silently
persisting plaintext.
>
> An OSS build can now create, schedule, dispatch to Lambda, take
`resolve`/`ack` and report a check. 7 393 lines moved into the new
crate; 679 lines stay enterprise.
>
> **🛑 Deployment prerequisite (2 of 2).** `ZO_SYNTHETICS_ENCRYPTION_KEY`
must be set to the **same** base64 64-byte value in every region of a
super cluster. Without it synthetics falls back to the per-org DEK and
checks carrying secrets silently fail to run outside the scheduler
region. It is read once at startup and pinned — changing it later
requires a restart, and changing it at all orphans every secret already
encrypted under the old value.
>
> **🛑 Deployment prerequisite (1 of 2).** 14 `O2_SYNTHETICS_*` env vars
are hard-renamed to `ZO_SYNTHETICS_*` with **no fallback read**, and
`ZO_SYNTHETICS_ENABLED` defaults to `false`. A deployment still setting
the old names does not lose tuning — **synthetics stops running
entirely, and the first symptom is checks silently not firing.** Update
env vars *before* rolling the binary.
>
> #### Things a reviewer should look at first
>
> - **`super_cluster_queue` does not depend on
`openobserve-synthetics`** (`cargo tree` → 0), which is what keeps the
applier's loop-prevention guarantee a property of the crate graph rather
than a convention. `a7d5fdd969` adds the assertion that fires if that
ever changes.
> - **`job_upload` was skipping authorization in OSS builds**
(`4c3a0b8f0d`). Unreachable before, because no synthetics route was
registered in OSS; registering it would have made it an unauthenticated
artifact upload. `authorize_probe` is ungated and the bypass is gone.
> - **The dependency edge had to reverse in a single commit**
(`3247d4093f`). Cargo rejects an `o2_enterprise` ↔ OSS package cycle at
the manifest level — `optional` and feature gating do not help — so the
two edges could never coexist, and everything needed for the reversal
lands together. This is why the spec's steps 4 and 5 are one commit and
the config moved earlier than planned.
>
> #### Accepted OSS limitations, all documented in the commits
>
> - No notifications: `notify_check_result` ends in
`alerts::alert::dispatch_notification`, which is enterprise. An OSS
build runs the check and records the run; it has nowhere to send an
alert, exactly as for alerts themselves.
> - No per-resource RBAC on synthetics, and no OpenFGA ownership tuples
— both are enterprise capabilities across the platform.
> - Private locations are refused with a message rather than created.
> - Agent-served public `net-*` pools are unavailable, so an OSS user
needs a Lambda for every public pool (spec §1).
---
Consumer half of synthetics super-cluster support. **Depends on
openobserve/o2-enterprise#2420** (producer, message types, queue struct)
— and **this side must deploy first**: `MessageType` bytes 83–85 hit a
panicking `unreachable!()` in `From<u8>` on a consumer that does not
know them, which takes down the subscriber task rather than failing one
message.
Based on `fix/lease-claimed-by-2270` (#13785), not `main`, so the diff
is only this work.
## What this fixes
Synthetics writes are not querier routes, so a create lands on ingesters
in whichever region received the request and commits to that region's
Postgres. There is no synthetics topic on the super-cluster queue, so a
check created in a region that runs no scheduler is never seen by the
region that does: the UI shows it enabled, the scheduler has never heard
of it, and nothing reports the gap.
## Commits
| commit | what |
| --- | --- |
| `8ceb041c43` | orphan-detection query + metrics |
| `1c3e10da48` | apply replicated check config |
| `79e6abc052` | apply replicated locations and probe tokens |
| `16d4fa0fde` | render unknown live status instead of a false pending |
| `5885c57b3a` | share the job-cluster predicate |
## Design notes worth reviewing
**Config columns only, and the list is not restated.**
`synthetics_checks::update` rebuilds its `ActiveModel` from the row
already in this region, and `update_mutable_fields` writes only
user-editable columns — so `next_run_at` and the alerting counters keep
local values for free. Hand-listing them in the consumer would duplicate
a list that already exists and let the two drift. A test asserts every
runtime column comes back `Unchanged`.
**`put()` is deliberately unused.** Its insert branch builds from
`build_active_model`, which ends in `..Default::default()` and never
assigns `owner`, `tz_offset` or `next_run_at` — a replicated create
through it would drop the owner and fire a scheduled-later check
immediately.
**`create()` gains `use_given_id`**, matching `table::alerts::create`.
It generated its own uuid, which would have made every region invent a
different primary key for the same check.
**Consumers apply through `infra::table` only.** The publish lives one
layer up in the enterprise service, and `infra` cannot reach the
enterprise crate — that layering is what stops a region re-broadcasting
what it just applied. `source_cluster` does not cover this: it only
stops a region re-consuming its *own* message. Each consumer greps its
own source to keep this true.
**Runtime tables are never replicated** — `synthetics_jobs`,
`synthetics_runs`, `synthetics_agents`, and the locations
`down_notified_at` claim. Asserted at both producer and consumer.
## Known gaps, named not hidden
- If two regions mint a token (or location) with the same name for one
org before either message crosses, the UNIQUE constraint rejects the
arriving row. Local row wins, logged loudly. Erroring would redeliver to
dead-letter, which is worse.
- Pre-existing and untouched:
`write_triggers_stream`/`write_results_stream` in the enterprise reaper
post to `_meta` with the *org's* ingest token, which 401s, and neither
checks response status. Orphan detection fixes this in its own path
only.
- Pre-existing: `create` never sets `tz_offset`, so a new check gets DB
default 0 until its first edit. Replicates identically everywhere, so it
causes no divergence.
## Spec corrections
The spec (`docs/synthetics-lcl/2026-08-11-p1-3-super-cluster-spec.md`,
local notes) is wrong in five places, all found by reading the tree:
§1's invariant evidence (see the enterprise PR), §10's never-run guard,
§4's column list (and it lists six runtime columns — there are seven,
`alerting` is missing), §3's "private rows read-only when remote", and
§7's claim that non-scheduler regions render OFFLINE (they render
`pending`).
## Verification
infra 42, super_cluster_queue 32, jobs 21, web 111/111 vitest +
type-check + design lint, `cargo fmt --check` and clippy clean, full
`--features enterprise` build.
Pre-existing failures on this branch, unrelated and untouched:
`super_cluster_queue::schemas::tests::test_parse_key_{valid,metrics}`,
`infra::table::ratelimit::tests::test_add_batch_missing_rule_id`, and 8
web vitest failures in `buildPayload.spec.ts`/`payloadContract.spec.ts`.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
## Summary
RCA (2026-08-19, 08:20-08:22 UTC): `GET
/api/{org}/annotation_queues/{queue_id}/items/{item_id}` (and its
`/reviews` sibling) intermittently returned 500. Originally attributed
to search work-group admission capacity exhaustion; tracing the actual
code found a different root cause.
**Real mechanism**: `eval_jobs::search::execute()`'s registered
executors (in `src/jobs/src/job/mod.rs`) called
`search_service::search("", ...)` with a hardcoded empty `trace_id`.
With no propagated OTel span in that detached task,
`search_service::search()` falls back to OpenTelemetry's *invalid* trace
ID for every such call - the same `000...000` seen in every failing log
line. Every concurrent internal search therefore shared one
`QueryManager` map key: one query's cleanup (`SEARCH_SERVER.remove`)
deleted another's still in-flight registration, so the next
`insert_sender` failed with "trace_id not found" -> `SearchCancelQuery`
-> 500. `work_group_checking` never reached the real concurrency-wait
branch in this failure mode - "work-group capacity" was never the actual
mechanism.
## Changes (3 commits)
1. **`fix(annotation-queues): merge item detail and reviews into one
call`** - the Queue Workbench always fetches item detail + reviews
together (confirmed: no caller ever needs one without the other,
including after review submission); fold them into one backend call,
cutting the per-item load from 2 HTTP round trips to 1. See companion PR
openobserve/o2-enterprise#2473 for the response-shape change this
depends on.
2. **`fix(annotation-queues): map search admission cancellation to
429`** - `annotation_queue_error_response` unconditionally 500'd on
`AnnotationQueueError::Search`/`Hydration`, even when the underlying
cause already has a defined 429 `http_status()` (`SearchCancelQuery` /
`RatelimitExceeded`). Downcast and map correctly instead.
3. **`fix(search): stop colliding on trace_id in internal eval
searches`** - the actual root-cause fix: generate a real per-call
`trace_id` via `config::ider::generate_trace_id()` at all three internal
`search_service::search("", ...)` call sites instead of passing `""`.
## Dependency
This PR's enterprise build depends on openobserve/o2-enterprise#2473
(same branch name `ud/fix-annotation-queue-hydration-500s`) via the path
dependency in `Cargo.toml.openobserve` - that PR adds the `reviews`
field on `AnnotationQueueItemDetail` this PR's merge (commit 1)
consumes.
## Test plan
- [x] `cargo build --features enterprise`
- [x] `cargo test -p openobserve-api-management --lib --features
enterprise annotation_queue` - 11 passed (incl. 2 new tests for the 429
mapping), no regressions
- [x] `cargo test -p o2_enterprise --lib annotation_queues` (companion
repo) - 16 passed, no regressions
- [x] `cargo clippy --features enterprise -p openobserve-jobs -p
search_service -p openobserve-api-management --no-deps` - clean (no
warnings in touched files)
- [x] `cargo fmt --check` - clean
- [x] Frontend: `npx vitest run` on `llm-queues.service.spec.ts` +
`QueueWorkbenchPage.backTarget.spec.ts` - 9 passed
- [x] `vue-tsc --noEmit` (`type-check:app`) - clean
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Yashodhan JoshiandYashodhan Joshi
Subhra264
Loakesh Indiran
Hengfei Yang
Ashish Kolhe
IluckySiandHengfei Yang
Amey PawarandHengfei Yang
Vaibhav Mittal
0xfandomandHengfei Yang
Huaijin