mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-05 08:09:22 +00:00
fd2306ffaa3ec2d760f05df3fcf3791efb1ff991
415
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Huaijin
|
d5fa24711c |
perf(metrics): choose label timestamp filter automatically (#13756)
## Summary - replace the `ZO_METRICS_INLIST_FILTER_ENABLED` switch with automatic timestamp filter selection - use an exact `IN` filter for up to 1024 timestamps and a bounded `BETWEEN` predicate above that limit - sort exact timestamp literals for stable query plans - add boundary coverage for empty, threshold, and over-threshold timestamp sets ## Rationale Large timestamp `IN` lists add planning and evaluation overhead, while always using `BETWEEN` can admit many unrelated label rows for sparse timestamp sets. A bounded exact-list strategy keeps the selective path for manageable sets without allowing query plans to grow indefinitely. Release benchmarks used 4.2M rows, five label columns, four hash partitions, and repeated median measurements: - with a representative sparse range returning about 262K rows through `BETWEEN`, 1024 timestamps favored `IN` (9.1 ms vs 10.1 ms), while 2048 favored `BETWEEN` (11.7 ms vs 10.2 ms) - when unique maximum timestamps equaled the series count and both predicates returned exactly one row per series, `BETWEEN` consistently won and the gap grew with list size, confirming the need for an upper bound ## Testing - `cargo test -p promql series_labels::tests` - `cargo clippy -p promql --tests --no-deps -- -D warnings` - `git diff origin/main...HEAD --check` ## Operational note `ZO_METRICS_INLIST_FILTER_ENABLED` is removed and no longer needs to be configured. |
||
 Hengfei Yang
|
238b0d5262 |
fix: support zstd-compressed HTTP requests (#13728)
## Summary - enable tower-http zstd request decompression - add a router middleware regression test using a real zstd-compressed JSON payload - update decompression documentation to list zstd support ## Root cause The Axum router uses `RequestDecompressionLayer`, but the workspace did not enable tower-http's `decompression-zstd` feature. Requests from clients such as Vector configured with `compression: "zstd"` were therefore rejected with `415 Unsupported Media Type` before reaching the ingestion handler. ## User impact Vector HTTP sinks can continue using zstd compression when sending batches to OpenObserve ingestion endpoints. ## Validation - `cargo fmt --all -- --check` - `cargo test -p openobserve-api-http handler::http::router::decompression::tests` - `git diff --check` Fixes #13726 |
||
 Pinguladora
|
575e8ea4d5 |
feat: add zstd compression on OTLP gRPC ingest (#13703)
Design at: #13702 / inline in PR description Add the `zstd` feature to tonic and enable `accept_compressed(Zstd)` on the OTLP logs, metrics and trace gRPC services in both `run_common` and run_router`. Closes #13702 ## Out of scope - Response compression on `send_compressed(Zstd)` - OTLP/HTTP path (tower-http zstd features) - snappy would be the better choice under CPU pressure, but tonic has no snappy support. |
||
|
Huaijin
|
aaed33543c |
refactor(search): group DataFusion and Tantivy feature configs (#13664)
## Summary - add ZO_FEATURE_METRICS_PUSHDOWN_FILTER_ENABLED as an opt-in filter pushdown switch for metrics queries - group DataFusion and Tantivy feature configuration under the search config struct - move inverted-index result cache sizing, footer caches, skip threshold, TopN group limit, and bloom footer cache settings from limit to search - preserve the moved cache settings' environment variable names and defaults - apply the metrics-specific setting to object storage, WAL, and Flight query contexts ## Testing - cargo fmt --all -- --check - cargo check for config, infra, search, tantivy_utils, search_service, and promql-service - cargo test -p config every_env_config_default_parses --lib - cargo test -p config inverted_index_config --lib - cargo test -p search test_create_session_config --lib - cargo clippy for all affected crates with -D warnings |
||
 Ashish Kolhe
|
58c7dafa55 |
fix: notification templates v2 — rich content templates + alert chart images (#13640)
## Summary
Two coupled alerting features:
**1. Notification Templates v2 ("content" templates)**
- New `kind: content` template type: a structured spec (title, body
markdown, fields, links, rows, per-channel title overrides,
severity-filtered visibility) rendered natively per channel — existing
`custom` (raw body) templates are untouched and byte-identical on the
send path.
- Per-channel renderers: Slack Block Kit, Teams (Adaptive Card +
MessageCard), Discord embeds, PagerDuty Events v2, Opsgenie, ServiceNow,
SNS, HTML/text email, and a stable machine `webhook` envelope.
- Escaping is renderer-owned (mrkdwn entities, HTML encoding, markdown
stripping) with a shared URL-scheme allowlist; golden-fixture tests per
channel.
- Template editor UI: guided content form (fields/links row tables,
compact rows config, channel title overrides), live preview panel with
per-channel preview + payload tab, test-send (rate-limited,
`[TEST]`-marked).
- Migration `m20260802_000001_add_template_kind`; prebuilt default
template + org backfill.
**2. Alert chart images (stateless, storage-free)**
- Alerts can attach a metric-history chart PNG rendered from the
triggers self-reporting stream (`actual_value` per evaluation), with
warn/crit threshold lines and the firing point appended.
- Zero storage: the chart URL *is* the data — deflate+base64url payload
signed with truncated HMAC-SHA256 (org-bound, expiry embedded), rendered
on fetch by the HTTP-terminating node (`GET
/api/v2/{org}/alerts/charts/render`). Signature verified before inflate
(decompression-bomb guard), uniform 404s, per-node render semaphore +
per-minute query budget + small LRU.
- Renderer: plotters + embedded DejaVu Sans (Bitstream Vera license
included), gap-broken line segments so sparse evaluation history doesn't
draw false plateaus.
- Channel delivery: Slack image block, email inline CID (no remote
fetch), Discord multipart attachment, Teams/PagerDuty image URLs;
per-destination opt-in via template chart flag.
- Slack layout live-A/B'd on a real channel: content + buttons/chart
split across two same-colored attachments so the severity stripe spans
everything while staying under Slack's "Show more" fold; clickable link
title; "View in OpenObserve" keeps a reserved primary-button slot.
- Slack image-delivery fallback: Slack validates image URLs server-side
and rejects the ENTIRE message with `400 invalid_attachments` when its
proxy can't fetch the chart (VPN-only/private `ZO_WEB_URL`) —
live-verified. On that rejection the send strips the image blocks and
resends once (alert always delivers), sets a process-wide flag so later
sends pre-strip (reachability is deployment-wide, not per-destination),
and re-probes hourly. Zero config; observable via `slack_image_rejected`
/ `slack_image_pre_stripped` metric events.
## Testing
- 147 notification unit tests (resolve/render/preview/test-send incl.
per-channel goldens and hostile-input/injection coverage), 8 chart
payload tests, renderer PNG tests.
- Live-verified end to end: Slack (Slackbot chart fetches ~100ms,
cache-immutable semantics), SES email inline image, multi-severity
firing scenario (warning→critical) with correct threshold lines.
- FE: AddTemplate/ContentTemplateForm/preview specs; UX audit pass with
fixes.
- `cargo clippy -D warnings` clean in both OSS and enterprise
arrangements.
## Enterprise pairing
Must merge together with openobserve/o2-enterprise#2353 (workspace
twin-manifest mirror for the new OSS deps).
|
||
|
Huaijin
|
e1e3091c06 |
refactor: move streaming aggregates to OSS (#13589)
## Summary - move streaming aggregation cache, aggregate execution, and optimizer code into the OSS search crate - move cache aggregation SQL analysis into the SQL visitor and enable streaming aggregation in search_service without enterprise gating - move MetadataCountExec into OSS and use it for metadata count optimization in all builds - remove migration-only dead code while keeping the cache module under search Companion cleanup: openobserve/o2-enterprise#2327. ## Testing - cargo fmt --all -- --check - cargo test -p search metadata_count --lib - cargo test -p search aggregate_optimize_rewrite --lib - cargo test -p search streaming_aggregate --lib - cargo test -p search cache::streaming_agg --lib - cargo test -p search datafusion::optimizer::stream_aggregate --lib - cargo test -p search_service partition --lib - cargo check -p search_service |
||
|
Huaijin
|
872ceb724a |
refactor: move search and compaction optimizations to OSS (#13583)
## Summary - move optimized Tantivy simple and multi histogram collectors into the open-source config crate - move general broadcast join eligibility, rewrite, execution, TmpExec codec, and shared async state into open-source search - enable enrichment-table broadcast join in OSS, including join reordering, remote-scan rewriting, EnrichmentExec, and its physical-plan codec - move AggregateTopkExec, its heap/sort streams, physical optimizer rule, and codec into open-source search - replace the enterprise-only Aggregate TopK settings with OSS-owned `ZO_DF_USE_AGG_TOPK_HEAP` and `ZO_DF_TOPK_HEAP_MAX_LIMIT` settings - move `approx_topk` and `approx_topk_distinct` UDAFs into open-source search and register them in both OSS and enterprise builds - omit the Aggregate TopK benchmark, benchmark documentation, and data-generator example from OSS - move Bloom term extraction, post-merge build, `.bf` writing, and orphan cleanup into the OSS compaction crate - run the Bloom build path after compaction in both OSS and enterprise builds, with non-fatal fallback to regular Tantivy search - move the direct `tantivy` and `object_store` dependency ownership required by Bloom building into OSS compaction - enable sorted timestamp, block, RANK, broadcast join, Aggregate TopK, approximate TopK, and Bloom paths in both OSS and enterprise builds - retain cross-super-cluster TmpExec fetching behind the enterprise feature - remove feature-gated enterprise imports and use the shared OSS implementations - add fastdivide for histogram bucket computation - resolve the duplicated SLO test attribute warning and remove an unused test-only alert-state migration helper ## Tests - `cargo fmt --all -- --check` (OSS and enterprise) - `cargo clippy --locked -p search --lib -- -D warnings` - `cargo clippy --locked -p compaction --lib -- -D warnings` - `cargo clippy --locked -p o2_enterprise --lib -- -D warnings` - `cargo test -p config tantivy::query::histogram_collector --lib` (21 passed) - `cargo test -p search tantivy::search::tests --lib` (9 passed) - `cargo test -p search enrichment --lib` (2 passed) - `cargo test -p search broadcast_join --lib` (7 passed) - `cargo test -p search join_reorder --lib` (8 passed) - `cargo test -p search datafusion::distributed_plan::codec::tmp_exec --lib` (1 passed) - `cargo test -p search aggregate_topk --lib` (7 passed) - `cargo test --locked -p search approx_topk --lib` (8 passed) - `cargo test --locked -p config meta::slo --lib` (371 passed) - `cargo test --locked -p infra bloom --lib` (30 passed) - `cargo check --locked -p config --tests` - `cargo check --locked -p infra --tests` - `cargo check --locked -p search --lib` - `cargo check --locked -p compaction --lib` - `cargo check --locked` (OSS root package) - `cargo check --locked --workspace` (enterprise workspace) - `cargo check --offline -p compaction --lib --features enterprise` with the paired repositories - `cargo check --offline -p openobserve --no-default-features --features enterprise` with the paired repositories Companion enterprise cleanup PR: https://github.com/openobserve/o2-enterprise/pull/2326 |
||
|
Huaijin
|
015a20a184 |
refactor: remove HTTP API compatibility re-exports (#13560)
## Summary - remove the HTTP request/model compatibility re-exports introduced by the domain crate split - migrate router, OpenAPI, enterprise action-server, and integration-test callers to the crates that own each API - remove the ingest Content-Type re-export and declare the direct root/test dependencies - delete the pure `handler::http::models` facade module ## Validation - `cargo fmt --all -- --check` - `cargo clippy -p openobserve-api-ingest -p openobserve-api-http -- -D warnings` - `cargo test --test integration_test --no-run` - `cargo check -p openobserve --no-default-features` - enterprise overlay: `cargo test --offline --workspace --no-default-features --features enterprise --no-run` - cloud overlay: `cargo test --offline --workspace --no-default-features --features cloud --no-run` - `git diff --check` |
||
|
Huaijin
|
6c029f7a2c |
chore(deps): bump vortex to 1cb48912 (#13540)
Bumps the `openobserve/vortex` pin from `57ffbf2b` to `1cb48912` in the workspace manifest. ## Upstream API changes handled - Arrow interop moved into a dedicated `vortex-arrow` crate, surfaced as `vortex::arrow`. `vortex::array::arrow::*` and `vortex::dtype::arrow::*` no longer exist, so `FromArrowArray`, `FromArrowType`, `ToArrowType` and `ArrowSessionExt` are now imported from `vortex::arrow`. - `ToArrowType` is no longer in scope implicitly; added the explicit import where `DType::to_arrow_schema()` is called. - The `vortex::compressor` facade narrowed to `BtrBlocksCompressor`, `BtrBlocksCompressorBuilder`, `Scheme`, `SchemeId`. `SchemeExt` and the scheme list are no longer re-exported there, so `search` takes a direct `vortex-btrblocks` dependency (same git rev) for `SchemeExt` / `schemes::integer::IntDictScheme`, which the UTF8 compressor uses to exclude the int-dict scheme. No behavior change intended — the compressor configuration and write strategies are identical. ## Verification - `cargo check --workspace --all-targets` clean - `cargo fmt --all -- --check` clean - `cargo test -p search --lib datafusion::vortex` — 6 passed - `cargo test -p tantivy_utils --lib vortex` — 3 passed - `cargo test -p config --lib utils::parquet` — 21 passed Paired enterprise PR (same branch name): https://github.com/openobserve/o2-enterprise/pull/2303 |
||
|
Huaijin
|
17ef03b8d6 |
refactor: cut release build time in the web and telemetry paths (#13528)
Two changes that cut build time, both measured on the same machine (M2
Max,
12 cores) with `cargo build --features mimalloc --profile release`.
## 1. The `web` crate no longer depends on `config`
`src/web` is a leaf crate so the RustEmbed expansion over `web/dist`
only
re-runs when the assets change — that expansion is 149.8s, of which
144.1s is
macro expansion (the crate already sets `#[br = false]` / `#[gzip =
false]`, so
this is not compression).
It also depended on `config`, for exactly one value:
`cfg.common.base_uri`. That
put a 50MB embed downstream of a 724-crate subgraph, so **every `config`
edit
paid the 148.5s embed again** — which is precisely what the crate's own
doc
comment says it exists to avoid.
`ui_routes()` now takes the `<base href>` value from the caller and
`base_href_middleware` carries it as middleware state.
`create_app_router()`
already computed that exact string (`"{base_uri}/web/"`) for its own
redirect,
so it is now resolved once and passed in rather than recomputed per
request
from a second `get_config()` call. `base_uri` is env-derived, normalised
once at
config init and never touched by `refresh_config`, so hoisting it out of
the
request path cannot drift.
The web crate's dependency tree goes **724 → 92 crates**, and `cargo
tree -i
config` no longer lists it.
## 2. Logging and tracing setup moved out of `main.rs`
`setup_logs`, `enable_tracing` and their span-processor / exporter types
compose
very large generic trees. Everything `main.rs` monomorphizes is emitted
into the
binary's own codegen units — the last unit on the critical path,
overlapping
with nothing.
`-Ztime-passes` on the binary unit (91.1s total):
```
LLVM_thinlto 39.8s
LLVM_passes 37.5s
codegen_to_LLVM_IR 5.3s
link_binary 5.3s (run_linker 3.9s)
type_check_crate 0.4s
```
The `openobserve` library does not depend on `openobserve-api-http`, so
its unit
compiles in parallel with the API crates — in the cold build it finished
23.9s
before the binary needed it. Moving the setup into `src/telemetry.rs`
(563
lines) puts that work in the slack instead of at the tail.
Two tests moved with the code and were dropped rather than relocated:
`test_setup_logs` and `test_enable_tracing_error_handling` both install
a global
tracing subscriber, and `setup_logs()` calls `.init()` unguarded, so
they can
only ever pass depending on ordering. They used to race just with each
other
inside the bin test binary; keeping them would have moved that race into
the lib
test binary next to the `cli` and `migration` tests. Neither asserted
anything —
the second wraps the call in `catch_unwind` and discards the result.
`main.rs` 1304 → 712 lines. **Binary unit 88.8s → 80.7s**; the library
unit grows
30.9s → 62.3s and is fully absorbed by the existing slack.
## 3. Unrelated: a dead-code warning in `openobserve-core`
Not a build-time change, just a warning fix noticed while checking the
above.
`COMPARE_ROLLUP_LOOKBACK_MICROS` was gated on `any(feature =
"enterprise",
test)`, but its only two readers (`fetch_arm_aggregate`,
`fetch_arm_failures`)
are `enterprise`-only and the unit tests below it exercise only the pure
aggregation helpers. So in an OSS build `cargo check -p openobserve-core
--all-targets` warned `constant COMPARE_ROLLUP_LOOKBACK_MICROS is never
used`.
Gate narrowed to `enterprise`, and the doc comment — which claimed the
opposite
— corrected. Happy to split this out if you'd rather it not ride along.
## 4. Review follow-ups
- `enable_tracing` became a public library API in §2 without a doc
comment,
unlike `setup_logs` beside it. Documented: which span processors attach
under
which env vars, why the returned provider must be held rather than
dropped,
and that it and `setup_logs` are mutually exclusive since both call
`.init()`.
- Added the `<base href>` test described under Verification below.
Not taken: the suggestion to re-add `test_setup_logs` /
`test_enable_tracing_error_handling` to `src/telemetry.rs`. Both install
a
global subscriber against `.init()`, so they cannot be made
order-independent,
and neither asserted anything — see §2.
`test_resource_key_value_creation` was
deleted for the same reason; it asserted that
`opentelemetry::KeyValue::new`
stores the key it was given.
## Measured and rejected
Recording these so nobody repeats them.
**Moving route registration into the crates that own the handlers.** 317
of the
371 routes were registered in one 622-line function in
`openobserve-api-http`.
Splitting them out to api-management (234), api-search (46),
api-pipelines (23)
and api-ingest (9) worked as designed — api-http went 145.0s → 63.0s —
but the
cold build only improved 654.4s → 637.7s. Splitting across a crate
boundary
**copies** work rather than relocating it: without LTO each crate
instantiates
the axum/tower machinery for itself, so the four receiving crates grew
222.1s
combined and total CPU rose 190.8s. It also just relocated the
bottleneck —
api-management became the new tail at 135.1s, with api-search finishing
only
6.9s behind it, so the tail is a plateau rather than a peak. Not carried
in this
branch.
| Tried | Result |
|---|---|
| sccache | 97.4% hit rate, **0** wall-clock gain; first fill +54% |
| Remove aws-lc-sys from the critical path | **0** — its `dur=72.0s` in
`--timings` is contention, it needs 15.6s alone |
| `codegen-units = 16` on the glue crates | **31.4s slower**, CPU
+297.8s — `lto` defaults to `false`, so thin-local LTO is on and its
cost scales with CGU count |
| Trim DataFusion features | `datafusion-core` depends on
`datafusion-functions` without `default-features = false`, so the
expression features cannot be turned off from the facade at all; what
remains is worth 13.8s of CPU (0.25%) |
A caveat on reading `cargo --timings`: a unit's `duration` is **not**
its
intrinsic cost — it is inflated by CPU contention when the machine is
saturated.
The aws-lc entry above is the clearest example.
## Verification
At this revision:
- `cargo check --all-targets` clean on OSS — 0 errors, 0 warnings — and
each
commit checks clean standalone, so `git bisect` works
- `cargo test -p web` passes
Carried over from the revision these changes were first written against
(the
edits since are doc comments, the `ui_routes` parameter rename, two
deleted
tests, a `Cargo.lock` revert, and the `cfg` narrowing in §3 — which is a
no-op
when `enterprise` is on):
- `cargo check` clean on the enterprise feature combination
- Cold release build succeeds
- Smoke-tested against a running binary: `/healthz` 200, UI and API
routes
resolve, unknown paths 404
The `<base href>` rewrite — the one runtime behaviour §1 touches — is
now
pinned by `base_href_is_rewritten_to_the_mount_point`, and checked end
to end
against the real embedded `index.html`: `ui_routes("/abc/web/")` serves
`<base href="/abc/web/" />`, matching what the old per-request
`format!("{base_uri}/web/")` produced for `ZO_BASE_URI=/abc`. The
committed test
drives a synthetic HTML response instead, so it still asserts something
in
builds where `web/dist` is absent. Verified by mutation: breaking the
rewrite
fails it.
|
||
|
Hengfei Yang
|
2bf19b2b44 |
feat(mcp): support MCP spec revision 2026-07-28 (stateless, dual-era server) (#13527)
Design at: #13526 ## Summary Implements MCP specification revision **2026-07-28** (stateless protocol) alongside the existing 2025-11-25 behavior on the same endpoint, following the spec's dual-era server model: requests carrying modern per-request `_meta` are served per 2026-07-28, while `initialize` selects legacy semantics. Legacy clients are unaffected. ## Changes - **Version negotiation** (`src/api/http/.../mcp/mod.rs`, `src/mcp/src/protocol.rs`): the `MCP-Protocol-Version` header is now validated against a supported-version set (`2026-07-28`, `2025-11-25`) instead of strict equality — the old check would have rejected every client that upgrades to the new revision. Unsupported versions (header or body `_meta`) get `400` + JSON-RPC `-32022` (`UnsupportedProtocolVersionError`) listing the supported versions so clients can retry. - **`server/discover`** (`src/mcp/src/protocol.rs`): new RPC required by 2026-07-28, returning `supportedVersions`, `capabilities`, server identity in `_meta`, `instructions`, and `ttlMs`/`cacheScope` cache fields. - **Modern result decoration** (`src/mcp/src/protocol.rs`): requests declaring `io.modelcontextprotocol/protocolVersion: 2026-07-28` in `_meta` get `resultType: "complete"` and `_meta` server identity on results, plus `ttlMs`/`cacheScope` on `tools/list`. Legacy responses carry none of these fields — byte-for-byte unchanged. - **Request metadata headers** (`src/api/http/.../mcp/mod.rs`): modern requests must mirror body fields into `Mcp-Method` / `Mcp-Name` headers (including the Base64 sentinel encoding for `Mcp-Name`); missing or mismatched headers are rejected with `400` + JSON-RPC `-32020` (`HeaderMismatch`), per the transport's server-validation rules. - **HTTP status mapping**: unknown methods on modern requests return `404` + `-32601` so dual-era clients can distinguish a modern server from a legacy one; version errors return `400`. Legacy requests keep errors in-band with `200` as before. - **Legacy paths untouched**: `initialize`, `ping`, the GET SSE keepalive channel, DELETE no-op, and notification → `202` handling are unchanged. Tool ordering was already deterministic (sorted at init), satisfying the new caching guidance. ## Testing - `cargo test -p openobserve-mcp` — 94 tests pass (new coverage: `server/discover` shape, unsupported-version error payload, modern decoration, legacy non-decoration, sentinel decoding, version set). - `cargo test -p openobserve-api-http --lib handler::http::request::mcp` — 13 tests pass (new coverage: modern happy path, missing/mismatched `Mcp-Method`/`Mcp-Name`, header↔body version mismatch, missing version header on modern requests, modern unknown method → 404, legacy header accepted). - `cargo clippy` clean on both crates. |
||
|
Hengfei Yang
|
94bddd7a7b |
feat: pack ingester wal persist into large files to avoid massive small files (#13497)
Design at: https://github.com/openobserve/openobserve/issues/13509 Closes #13509 ## Problem On clusters with 100k+ metrics streams, every memtable rotation persists **one parquet file per stream × hour**, producing thousands of tiny files per second on the ingester. The legacy file mover cannot keep up: it re-scans the whole `wal/files` tree every `file_push_interval` (capped at `file_push_limit` = 10000 files per cycle ≈ a hard ceiling of ~1000 files/s), repeatedly stats not-yet-expired files, and pays a full DataFusion plan + S3 PUT per stream×hour for a few KB of data. Generation outpaces consumption and wal files pile up unboundedly. ## Design Behind `ZO_FEATURE_WAL_PACK_ENABLED` (bool, default **off**), applying to **metrics streams only** — the stream-cardinality problem is metrics-specific; other stream types keep the legacy per-stream persist path. With the flag on, the "split by stream" step moves from persist time (disk) to upload time (memory): - **Persist**: all streams of a memtable are appended into a few large **pack** files (`wal/pack/{idx}/{memtable_id}.{seq}.pack`, rolled at `ZO_MAX_FILE_SIZE_ON_DISK`, default 512MB). Each segment inside a pack is a **complete, self-contained parquet file**; the pack footer (JSON + gxhash trailer, versioned) indexes segments with stream, partition key, offset/length and FileMeta. One rotation now writes O(1) sequential files instead of O(streams) tiny ones. - **Query**: an in-memory segment index (rebuilt from footers on startup) resolves stream → byte ranges; segments are materialized into record batches and reuse the memtable search table-building path. Dedup against in-memory data uses the existing memtable_ids mechanism. - **Mover**: pending streams are snapshotted from the segment index (no readdir/stat storm, no `file_push_limit` cap). A stream is flushed when its segments are big enough or the oldest exceeds `max_file_retention_time`. A **single segment is uploaded as-is (zero decode/re-encode)**; multiple segments are merged in memory via DataFusion MemTable + the existing `merge_parquet_files`. Object keys, file_list, incremental compaction trigger, tantivy index, cache_latest_files and enterprise service-streams all follow the legacy mover. **S3 layout, file_list schema, compactor and querier are completely unchanged.** - **Crash recovery**: packs follow the same `.lock` five-step flow as legacy files (`.pack.tmp` → `.lock` → delete wal → rename → unlock), keeping exactly-once semantics. Lock recovery stays synchronous on startup (it must run before the pack index rebuild), while the orphan `.par` sweep over the potentially millions-of-files `wal/files` tree moved to a background task so startup no longer blocks. - **Pack recycling**: a registry tracks per-pack `{total, consumed, readers}`; a pack file is deleted once all segments are consumed and no in-flight query read holds it (read guards replace the legacy `lock_files`/pending-delete mechanism for packs). Also included: shared-memtable hash changed from `{thread_id}_{org_id}` to `{org_id}_{stream_name}` so one stream maps to exactly one bucket, removing the per-request round-robin fan-out that multiplied file/segment counts. ## Testing - Unit tests: pack format roundtrip, size-based rollover, corruption/truncation rejection, real-parquet segment read through the index with time-range/memtable-id filtering, consume lifecycle (pack deleted when fully consumed), reader-guard deferred deletion, flush conditions. - End-to-end with the real binary (4 phases): 1. wal binary format is byte-identical to main (`git diff main... -- src/wal src/ingester/src/entry.rs` is empty); 2. old-format wal written with the flag off **replays into packs** with the flag on — no migration needed; 3. mixed-source search returns exact counts with legacy wal parquet + pack segments + memtable coexisting (300/400/500 across three batches, filters and row fetches correct); 4. pack mover merges multiple segments into one storage file, generates the tantivy index, deletes consumed packs; final storage-side queries and `match_all` are exact. ## Unrelated fixes included Two startup paths shared the same "treat any db error as absence" bug: - `init_db` used to treat ANY error from `get_db_schema_version` as version 0 ("assuming default of 0 and trying upgrade") and kicked off the full db upgrade — so a merely unreachable or overloaded database got migration DDL piled on top of its existing load. Now `get_db_schema_version` returns 0 only when the version is genuinely absent (missing key, or the meta table itself not created yet on a fresh install); transient errors are retried 5 times with backoff and then abort the startup so the orchestrator restarts the node once the database recovers. - bootstrap did the same with the **instance id**: `instance::get()` reported a missing key as an error, so "not stored yet" and "db unreachable" landed in one arm that generated a fresh instance id — and could overwrite the stored one once the db recovered. The instance id must stay stable for the lifetime of a deployment, so a new one is now generated only when it is genuinely absent; read errors are retried with backoff and then abort the startup, and a failed write of a newly generated id is no longer ignored. ## Notes for reviewers - Flag is default **off**; with the flag off, behavior is unchanged except the startup fixes and the shared-memtable hash change. The flag only gates the persist path — pack search always runs (no-op on an empty index), so packs already on disk stay searchable after the flag is turned off while the mover drains them. - Verified with the real binary that with the flag on, a metrics stream is packed while a logs stream keeps the legacy per-stream parquet, both queryable (SQL + PromQL). - Pairs with o2-enterprise branch `feat/ingester-wal-pack-drain` (drain completion check also scans `wal/pack`); that change must land before/with enabling the flag in production, otherwise a draining node could shut down with pack segments not yet uploaded. - Known follow-ups (not in this PR): pack-specific metrics, extending pack to other stream types, large-scale load test for segment-index memory sizing, `wal-reader`-style debugging tool for packs. |
||
|
Huaijin
|
ef3617f157 |
refactor: split HTTP APIs by domain (#13479)
## Summary
- organize API crates under
`src/api/{common,grpc,http,ingest,management,pipelines,search}`
- keep ingest, management, pipelines, and search independent from one
another; compose them only in the HTTP frontend
- move pattern APIs into search and simplify `main.rs` wiring
- update workspace manifests, lockfile, integration tests, and
cargo-deny entries
- keep `web/` unchanged
## Validation
- `cargo fmt --all -- --check`
- `cargo clippy --all-targets -- -D warnings`
- `cargo deny check licenses`
- enterprise overlay: `cargo build --no-default-features --features
enterprise`
- enterprise overlay: `cargo build --no-default-features --features
cloud`
- enterprise overlay: `cargo clippy --all-targets -- -D warnings`
The matching `o2-enterprise` overlay PR will be linked after creation.
|
||
|
Huaijin
|
d165299bcd |
refactor: extract TLS and node services from core (#13449)
## Summary - extract TLS certificate loading and client configuration from openobserve-core into openobserve-tls - extract node discovery, cluster information, ingester gRPC channel selection, and Tokio runtime metrics into openobserve-node - update Core, API, API Management, Jobs, and the root binary to depend on the new crates directly - preserve one-way dependencies: openobserve-core depends on openobserve-node, while openobserve-node does not depend on Core - register both crates in the enterprise workspace overlay ## Validation - cargo test -p openobserve-node --lib - CARGO_INCREMENTAL=0 cargo clippy -p openobserve-node -p openobserve-core -p openobserve-jobs -p openobserve-api-management -p openobserve-api -p openobserve -- -D warnings - RUSTFLAGS="--cfg tokio_unstable" cargo check -p openobserve-node - enterprise overlay: CARGO_INCREMENTAL=0 cargo clippy -p openobserve --no-default-features --features enterprise -- -D warnings - cargo deny check licenses --hide-inclusion-graph - cargo fmt --all -- --check - git diff --check Companion enterprise PR: openobserve/o2-enterprise#2267 |
||
|
Huaijin
|
3f9ce27542 |
refactor: split service dependencies out of core (#13446)
## Summary - decouple stream cleanup from enrichment-data and remove the transitive compaction/DataFusion dependency from stream - extract sourcemaps and organization storage services from core into dedicated crates - extract GitHub built-in regex access and model pricing into openobserve-builtins - rename pipeline and organization-storage persistence modules from store to db - remove now-unused direct dependencies from core - synchronize the enterprise workspace overlay in openobserve/o2-enterprise#2266 ## Validation OSS: - cargo check -p openobserve-builtins -p openobserve-core -p openobserve-jobs -p openobserve-api - cargo test -p openobserve-builtins --lib - cargo test -p openobserve-builtins --doc - cargo clippy -p openobserve-builtins -- -D warnings - cargo check -p openobserve-org-storage --features enterprise Enterprise overlay: - copied ../o2-enterprise/Cargo.toml.openobserve over Cargo.toml - cargo fmt --all -- --check - cargo check --all-targets --no-default-features --features enterprise - cargo clippy --all-targets --no-default-features --features enterprise -- -D warnings The default enterprise + vectorscan build was attempted and reached vectorscan-rs-sys, but the validation host does not have cmake installed. The enterprise-only build and clippy run completed successfully. |
||
|
Huaijin
|
6123240a64 |
refactor: clean up core service dependencies (#13436)
## Summary
- remove the remaining openobserve-core Tantivy facade and let Jobs
depend on tantivy_utils directly
- inject enrichment-table cleanup from Core so the stream crate no
longer depends on enrichment-data (and transitively compaction)
- define a StreamDataCleanup port for organization deletion and move the
compaction implementation to Jobs
- remove openobserve-core's direct compaction dependency and enterprise
feature forwarding
- allow OSS builds to preserve ZO_FILE_FORMAT=vortex now that Vortex
support is always compiled
- add regression coverage for Vortex configuration and injected stream
cleanup failures
## Dependency direction
jobs -> tantivy_utils
jobs -> compaction
jobs -> openobserve-core::StreamDataCleanup
core -> stream + enrichment-data
core -X-> compaction (direct)
stream -X-> enrichment-data -> compaction
## Verification
- cargo fmt --all -- --check
- cargo check -p stream -p openobserve-core -p openobserve-jobs
- cargo test -p stream --lib (24 passed)
- cargo test -p config --lib (2027 passed, 2 ignored)
- cargo test -p openobserve-core org_cleanup --lib (14 passed)
- cargo clippy -p config -p stream -p openobserve-core -p
openobserve-jobs -- -D warnings
Enterprise-only check is currently blocked by the existing
openobserve-cipher build error: o2_enterprise::enterprise is unavailable
in this checkout.
|
||
|
Huaijin
|
df9d35c3d3 |
feat(search): move vortex support from enterprise into open source (#13434)
## Summary
Moves Vortex file-format support out of the enterprise build and makes
it a core, always-compiled capability in open source.
- **Code move**: the enterprise `search/vortex.rs` module
(`Utf8Compressor`, `VORTEX_RUNTIME`, `write_vortex`,
`generate_vortex_access_plan`, plus its unit tests) now lives at
`src/search/src/datafusion/vortex.rs`; the three call sites
(`merge/downsampling.rs`, `merge/mod.rs`, `table_provider/helpers.rs`)
reference the local module.
- **Feature removal**: the `vortex` cargo feature is deleted from the
root and all 11 sub-crate manifests. All `#[cfg(feature = "vortex")]` /
`#[cfg(all(feature = "enterprise", feature = "vortex"))]` gates and
their "requires the vortex feature" error branches are gone —
`FileFormat::Vortex` match arms compile unconditionally in config,
search, and tantivy_utils.
- **Dependencies**: `vortex` / `vortex-datafusion` are now direct git
dependencies (`openobserve/vortex` rev `57ffbf2b`, the same rev the
enterprise workspace pinned); the empty dependency-resolution stub
crates under `src/enterprise/vortex{,-datafusion}` are deleted.
Companion PR in o2-enterprise removes the vortex module/feature there
and syncs `Cargo.toml.openobserve`; that PR should merge together with
(or right after) this one, since its template relies on this repo no
longer declaring the feature.
## Verification
- `cargo check --workspace` (OSS, default features) passes
- `cargo check -p search -p tantivy_utils -p config --all-targets`
passes (vortex tests compile)
- Enterprise combined workspace (`Cargo.toml.openobserve` swapped in)
`cargo check -p openobserve` passes
- `cargo fmt` applied to touched crates
|
||
|
Huaijin
|
1624a6ff24 |
fix(schema): restore persistence guards on auto-UDS settings write (#13431)
## Summary #13405 rewired `handle_diff_schema`'s auto-UDS persistence from `stream::save_stream_settings` to a direct `db::schema::update_setting` call (required by the new crate dependency direction: `schema` cannot depend on `stream`). The direct write dropped the persistence guards that path provided. This PR restores them inside the schema crate, without reintroducing a dependency on the stream service: - skip persisting when the stream is being deleted (`db::compact::retention::is_deleting_stream`), matching the old `save_stream_settings` early-return - enforce `cfg.limit.user_defined_schema_max_fields` on the persisted field list (previously, with `ZO_SCHEMA_MAX_FIELDS_TO_ENABLE_UDS` set above the cap, settings that were never persisted before #13405 would now be written); skipped writes are logged at warn level instead of silently dropped - sort `defined_schema_fields` before persisting so the stored order is deterministic (was `HashSet` iteration order, nondeterministic across nodes/restarts) Also restores the `calculate_files_size` unit tests that were dropped when `src/core/src/file_list.rs` was deleted in #13405 — they now live next to the function in `infra::file_list`. ## Validation - `cargo check -p schema -p infra` - `cargo test -p schema --lib` (7 passed) - `cargo test -p infra --lib file_list::tests::test_calculate_files_size` (5 passed) - `cargo clippy -p schema -p infra` - `cargo fmt -p schema -p infra -- --check` |
||
|
Huaijin
|
15275ae7cf |
refactor: split core foundation services (#13405)
## Summary This PR begins decomposing `openobserve-core` into focused service crates while preserving the existing application behavior and dependency direction. It extracts six independently owned areas: - `ingestion-common`: shared ingestion contracts and response types - `schema`: schema inference, validation, evolution, and persistence helpers - `compaction`: merge, retention, and compaction services - `enrichment-data`: enrichment-table lifecycle and runtime support - `promql-service`: PromQL service orchestration - `stream`: stream metadata, settings, retention, and lifecycle operations ## What changed - registered the extracted crates in the workspace and propagated enterprise/cloud/vectorscan/vortex features - moved implementations out of Core and updated API, Jobs, query, and queue consumers to use the new crates - removed Core facade aliases for extracted crates and converted remaining callers to direct crate paths - retained the small cross-service stream-delete coordinator in Core for now because it still coordinates Pipeline and Alert cleanup - restored required root/Core dependencies after validating the full enterprise build - added the missing `anyhow` dependency for enterprise-only `search_service::search_jobs` - updated the lockfile for the new crate dependency graph ## Dependency direction The extracted crates do not depend on `openobserve-core`. Core remains a temporary upper-level consumer while the remaining domain services are split incrementally. ```text ingestion-common -> common/config schema -> ingestion-common/db/infra stream -> schema/enrichment-data/db/infra compaction -> schema/search/search-service/db/infra openobserve-core -> ingestion-common/schema/stream/compaction/... api/jobs -> openobserve-core + extracted services ``` ## Enterprise compatibility The private OpenObserve manifest was updated in openobserve/o2-enterprise#2257 to register the extracted workspace members and dependencies. Related Enterprise PR: https://github.com/openobserve/o2-enterprise/pull/2257 ## Validation OSS manifest: - `cargo fmt --all -- --check` - `cargo clippy -- -D warnings` Using `o2-enterprise/Cargo.toml.openobserve` as the OpenObserve root manifest: - `cargo check --no-default-features --features enterprise` - `cargo check --no-default-features --features cloud` - `cargo clippy --workspace --no-default-features --features enterprise -- -D warnings` - `cargo clippy --workspace --no-default-features --features cloud -- -D warnings` Enterprise repository: - `cargo fmt --all -- --check` - `cargo clippy -- -A clippy::question-mark -D warnings` All checks above pass in the development profile. |
||
|
Huaijin
|
eec3207c24 |
refactor: decouple API compilation from web assets (#13403)
## Summary - move embedded UI routing from `openobserve-api` into the existing `web` crate - inject `web::ui_routes` into the API router while preserving the existing `ui_enabled` and missing-`dist` behavior - make `openobserve-api` and `web` sibling dependencies so they can compile independently ## Validation - `cargo fmt --all --check` - Enterprise manifest: `cargo clippy --workspace --no-default-features --features enterprise,mimalloc -- -D warnings` - `cargo clippy --workspace -- -D warnings` - `cargo clippy -p web --all-targets -- -D warnings` - `cargo test -p web` - `cargo deny check licenses` |
||
|
Huaijin
|
6adcae3a1f |
refactor: remove crate compatibility re-exports (#13381)
## Summary
- remove compatibility re-exports introduced during the recent crate
refactors
- migrate callers to the crates that own each API
- delete five pure facade modules and add the required direct crate
dependencies
- keep OSS and enterprise manifests aligned
## Validation
- cargo fmt --all -- --check
- cargo clippy --workspace -- -D warnings
- cargo test --locked --workspace --no-run
- enterprise overlay: cargo test --offline --workspace
--no-default-features --features enterprise --no-run
- enterprise overlay: cargo clippy --offline --workspace
--no-default-features --features enterprise -- -D warnings
- git diff --check
Companion enterprise PR uses the same branch name:
codex/remove-refactor-reexports.
## TODO / Follow-ups
- [ ] **Re-establish the pipeline write-guardrail lost with
`db::pipeline`.** That module was not a pure facade: it re-exported only
the six read-only accessors from `infra::pipeline` and its doc comment
recorded the invariant that *mutations must go through
`service::pipeline::store`* (which keeps the `ExecutablePipeline` caches
and the coordinator/super-cluster watch events in sync). Deleting it
points callers straight at `infra::pipeline`, which also exposes
`put`/`delete`/`drop_table` with no guardrail. Decide the layering and
enforce it:
- **Reads** may hit `infra` directly (raw persistence, no side effects).
- **Writes** must go through the wrapper that persists *and* emits the
coordinator/super-cluster event + refreshes the in-memory cache. That
wrapper is `db::<entity>::{set,delete}` for most entities (e.g.
`db::alerts::alert::set` →
`infra::coordinator::alerts::emit_put_event`), but for pipeline it is
`core::pipeline::store` (not `db`), because the
`STREAM_EXECUTABLE_PIPELINES` cache it maintains is a core-level
construct the `db` crate cannot build. The super-cluster receiver
(`super_cluster_queue::pipelines`) is the one sanctioned caller of
`infra::pipeline::{put,delete}` directly, and it manually pairs each
with `emit_*_event` to avoid a re-broadcast loop.
- Net invariant to protect: never call `infra::pipeline::{put,delete}`
without emitting the coordinator event — i.e. nowhere outside
`core::pipeline::store` and the super-cluster handler. A bare `infra`
write leaves in-memory caches stale until restart (see the warning
comment still live at `src/core/src/org_cleanup.rs:484`).
- [ ] Consider whether the pipeline write-wrapper living in
`core::pipeline::store` while every other entity's lives in
`db::<entity>` is worth unifying, or at least documenting as an
intentional exception.
|
||
|
Huaijin
|
f6560d7d02 |
refactor: decouple search from core (#13338)
## Summary - extract search orchestration from core into the `search_service` crate - keep the existing low-level `search` crate and directory names unchanged - extract usage reporting and audit services into dedicated crates - replace global async function pointers with narrow publisher traits - keep compatibility re-exports while updating workspace dependencies and feature propagation Companion enterprise PR: https://github.com/openobserve/o2-enterprise/pull/2229 ## Validation - `cargo check --workspace` - `cargo check -p search --tests` - `cargo test -p search_service --lib` (184 passed) - enterprise non-release build with the paired `o2-enterprise` manifest - `cargo fmt --all -- --check` - `git diff --check` |
||
|
Huaijin
|
e236c7e653 |
refactor: extract database layer into db crate (#13340)
## Summary - extract persistence, cache, watcher data access, and DB service modules from openobserve-core into a standalone workspace db crate - keep application orchestration that depends on search, organization provisioning, and other Core services behind the existing Core db facade - preserve existing call paths through re-exports while making the dependency direction openobserve-core -> db - move HTTP error mapping back to the Core/API boundary so db has no direct axum dependency - avoid provider/provisioner callback traits and function-pointer hooks ## Validation - cargo fmt --all - cargo check --all-targets - cargo check -p db --all-targets - Enterprise workspace: cargo check --all-targets --offline - Enterprise workspace: cargo check -p db --all-targets --features enterprise --offline Companion Enterprise manifest PR: openobserve/o2-enterprise#2230. |
||
|
Huaijin
|
b835ff2f86 |
refactor: consolidate vrl/js into transform crate and extract super_cluster_queue crate (#13324)
## Summary Two crate-split refactors continuing the workspace decomposition: **1. `transform` crate (renamed from `openobserve-vrl`, `src/vrl` → `src/transform`)** - Now the only crate depending on `vrl`, `vector-enrichment` and `rquickjs` directly; downstream crates use `transform::vrl::` / `transform::vector_enrichment::` re-export paths and drop their direct dependencies - The QuickJS runtime (`common/utils/js.rs`) moved into `transform::js` (js_security checks gated behind the crate's `enterprise` feature); `apply_vrl_fn`, `convert_to_vrl`/`convert_from_vrl` and `TRANSFORM_FAILED` moved from core - Exception: `src/config` keeps its direct `vrl` dependency (it sits below `transform` in the dependency graph) **2. `super_cluster_queue` extracted from the root crate into a workspace member** - Only depends on `openobserve-core`, `infra`, `config` and `o2_enterprise`, so it builds in parallel with `api` and edits no longer recompile the root crate (which re-embeds `web/dist` on every rebuild) - Gated behind its own `enterprise` feature — compiles as an empty crate in OSS builds so `cargo --workspace` keeps working against the `o2_enterprise` stub; `vectorscan` wired through for `re_pattern.rs` - Root `lib.rs` re-exports the crate, so `openobserve::super_cluster_queue::init()` in `main.rs` is unchanged Pairs with the `Cargo.toml.openobserve` template sync PR in o2-enterprise (must merge together). ## Test plan - `cargo check --workspace --all-targets` (OSS) passes - `cargo check -p openobserve --no-default-features --features enterprise` with the synced enterprise template passes - `cargo clippy -p super_cluster_queue --features enterprise,vectorscan` — zero warnings - `cargo clippy -- -D warnings` (CI invocation) passes; `cargo fmt --all` clean |
||
|
Hengfei Yang
|
33d1da9b6e |
feat: move MCP server to OSS (#13272)
## Summary - move the MCP protocol, tool discovery, schema simplification, execution, and response filtering implementation into a new OSS `openobserve-mcp` workspace crate - enable the real MCP POST, GET, and DELETE handlers in community builds - initialize the MCP tool catalog from the OpenAPI document for every edition - keep OAuth authorization-server discovery enterprise-only because it depends on Dex - add OSS handler and OpenAPI initialization coverage Closes #13266 Design at: #13266 ## Impact Community users can connect MCP clients directly to OpenObserve and use the OpenAPI-backed tool catalog. Enterprise builds continue to use the same MCP implementation from the OSS workspace crate. Companion PR openobserve/o2-enterprise#2206 removes the old private copy and updates the enterprise workspace overlay. |
||
|
Huaijin
|
3b44da3eee |
refactor: extract shared common crate (#13252)
## Summary - introduce a `common` workspace crate for shared metadata, infrastructure helpers, and utilities - keep service-backed code in `openobserve-core` while preserving the existing `common` compatibility namespace - flatten `src/core/src/service/` into the standard `src/core/src/` module layout without using `#[path]` - retain `service::*` as compatibility re-exports from `service.rs` - move cipher request/response types from core into the search crate while preserving `openobserve_core::cipher` - remove unused Cargo dependencies from workspace crates Enterprise companion: https://github.com/openobserve/o2-enterprise/pull/2202 ## Validation - `cargo check --workspace --all-targets` - `cargo clippy -p common -p openobserve-core -p search --lib -- -D warnings` - common crate unit tests: 271 passed, 4 ignored - enterprise overlay: `cargo check -p openobserve-core --features cloud` - migrated cipher tests: 8 passed - `cargo deny check licenses` - `cargo fmt --all -- --check` - `git diff --check` |
||
    
|
4031f76426 |
feat: synthetic monitoring (#12849)
Design at : https://github.com/openobserve/designs/tree/main/synthetics ref: https://github.com/openobserve/o2-enterprise/issues/2095 --------- Co-authored-by: omkark06 <omkar@zinclabs.io> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Claude <claude@users.noreply.github.com> |
||
|
Huaijin
|
66ca3d2439 |
refactor: extract PromQL evaluator into dedicated crate (#13239)
## Summary - add a dedicated promql workspace crate for parsing and evaluation - keep cluster routing, cache, WAL, file-list, and request orchestration in openobserve-core - preserve existing service::promql call sites through a compatibility facade; the facade re-exports an explicit item list (not a glob), and the promql search orchestration in core depends on the promql crate directly - inject enterprise query cancellation through TableProvider instead of coupling the evaluator to core search state - replace the core regex-UDF dependency with DataFusion regexp_like expressions ## Companion PR - openobserve/o2-enterprise#2198 updates the enterprise workspace override for the new promql crate ## Validation - cargo fmt --all -- --check - cargo check --workspace - cargo clippy -p promql --all-targets -- -D warnings - cargo clippy -p openobserve-core - cargo test -p promql: 363 passed - verified the promql dependency tree has no openobserve-core or search dependency - cargo check -p o2_enterprise --lib in the paired private workspace ## Notes The evaluator moves atomically in this PR. The existing service::promql::search subtree remains in core because it owns OpenObserve-specific orchestration rather than PromQL evaluation. The two web/ file changes only update source-path references in comments that pointed at the moved Rust files (src/service/promql/... -> src/promql/src/...). |
||
|
Huaijin
|
f36c9bd5f1 |
refactor: move search engine into search crate (#13238)
## Summary This is PR 2 of the atomic search-crate split, following #13234. - move the complete query-engine SCC into `src/search`: DataFusion, SQL, index, Tantivy, bloom pruning, inspector, and engine utilities - move 130 Rust files / 43,935 lines as a single atomic change - keep search orchestration, result-cache policy, cluster/gRPC/streaming, partitioning, usage, and audit responsibilities in `openobserve-core` - preserve existing `openobserve_core::service::search::*` paths through ordinary facade re-exports - move direct engine dependencies and feature ownership into the `search` manifest - remove core dependencies used only by the moved engine No query behavior change is intended. Source changes inside the moved engine are limited to crate-relative import paths and formatting. ## Dependency boundary ```text openobserve / API / jobs | v openobserve-core (orchestration) | v search (query engine) | v config / infra / proto / flight / openobserve-vrl ``` - `search` does not depend on `openobserve-core` - `openobserve-vrl` remains independent of both `search` and `openobserve-core` - the old core DataFusion, SQL, and Tantivy implementation directories no longer exist ## Validation - `cargo check --workspace` - `cargo check -p openobserve --features mimalloc` - `cargo clippy -- -D warnings` - `cargo clippy -p search -p openobserve-core --lib -- -D warnings` - `cargo fmt --all -- --check` - Cargo metadata dependency-boundary checks - `cargo test -p openobserve-vrl --lib` — 7 passed - `cargo test -p search --lib` — 876 passed; 12 existing histogram static-configuration tests fail in this local environment - `cargo test -p openobserve-core --lib` — 2407 passed, 6 existing global-state/filesystem/DB-isolation tests failed, 18 ignored - filtered reruns excluding those existing environment-sensitive tests passed ## Not run - private Enterprise / Vectorscan / Vortex overlay builds - local/distributed runtime smoke tests - cold/incremental build and query runtime performance benchmarks No performance improvement is claimed by this PR until those benchmarks are completed. |
||
|
Huaijin
|
059058519e |
refactor: make search engine sources crate-independent (#13234)
## Summary - add `src/vrl` as a leaf package named `openobserve-vrl` with Rust path `openobserve_vrl` - move `QUERY_FUNCTIONS`, enrichment-table ownership, GeoIP/MaxMind table registration, VRL compiler configuration, and shared runtime helpers into `openobserve-vrl` - let ingestion, jobs, core, and search use the shared VRL integration without depending on one another - keep DataFusion UDF registration direct through `Sql.org_id`; no `QueryExecutionContext`, `PreparedQueryTransform`, or `prepare_query_transforms` - move the cipher registry, query parameters, and file-cache helpers to `search` - move the generic file downloader implementation to `infra` - add feature forwarding for the `search` crate - keep compatibility re-exports for existing API, ingestion, gRPC, jobs, and Enterprise call sites Enterprise companion: https://github.com/openobserve/o2-enterprise/pull/2196 |
||
|
Huaijin
|
d6166aac49 |
perf: speed up embedded web asset compilation (#13233)
## Summary - upgrade rust-embed-for-web from 11.3 to 11.4.1 - disable unused compile-time gzip and Brotli generation for embedded UI assets - keep browser response compression through the global tower-http CompressionLayer Related enterprise PR: https://github.com/openobserve/o2-enterprise/pull/2192 ## Benchmark Clean release-profiling build with mimalloc: - before: 21m 13s - after: 15m 45s - improvement: 5m 28s, or 25.8% Focused web crate compilation improved from 546.66s to 98.55s. ## Testing - cargo metadata --locked --no-deps --format-version 1 - cargo clean - cargo build --features mimalloc --profile release-profiling |
||
|
Huaijin
|
5dc22a0ee8 |
refactor: establish search crate boundary (#13217)
## Summary - add the minimal `openobserve-search` workspace crate and move shared search/cache types into it - move the concrete enrichment table type and cache ownership to the search crate while retaining normal compatibility re-exports - move trace schema constants to `config` so the SQL engine no longer reaches into the traces service - expose concrete enrichment read functions from `infra` and make DataFusion call them directly - construct the stateless VRL runtime directly in the transform UDF - keep search execution and orchestration in `openobserve-core` for the later physical moves - do not introduce `SearchRuntime`, provider traits, callbacks, `#[path]`, or a search-to-core dependency This combines the original crate-boundary step with the dependency cleanup needed before SQL/DataFusion files can move safely. ## Validation - `cargo test -p openobserve-search --lib` (11 passed) - `cargo test -p openobserve-core service::enrichment::tests --lib` (19 passed) - `cargo check -p openobserve --features mimalloc` - enterprise + mimalloc check using the o2-enterprise overlay - cloud + mimalloc check using the o2-enterprise overlay - `cargo fmt --all -- --check` - strict all-target clippy for `openobserve-search` and `openobserve-core` - strict library clippy for `infra` - dependency check confirms `openobserve-search` does not depend on `openobserve-core` `infra --all-targets` currently reports three pre-existing inconsistent digit-grouping warnings in `src/infra/src/table/entity/alert_dedup_state.rs`; they are unrelated to this change and are not included here. |
||
|
Huaijin
|
3559d483ae |
refactor: move web UI asset embedding into a dedicated leaf crate (#13211)
## Problem Every rebuild of `openobserve-api` re-runs the `RustEmbed` macro expansion, which embeds `web/dist` (53MB, 1035 files) and compresses every file with gzip + brotli(q11) — single-threaded, non-incremental, and it peaks at 8GB RSS. `-Ztime-passes` attributes ~540s of the 769s api compile to `expand_crate`. Net effect: touching one file in `openobserve-core` costs a **16m45s** rebuild on an M-series laptop, and neither incremental compilation nor codegen-unit tuning can help, because macro expansion happens before both. ## Fix Move the `#[derive(RustEmbed)]` into a new `src/web` leaf crate with no workspace dependencies. The embed + compression now only recompiles when `web/dist` changes; edits anywhere in the backend no longer pay for it. **Measured (same machine, default profile settings, `--profile release-profiling`):** | | before | after | |---|---|---| | `openobserve-api` compile | 769s | 231s | | touch `src/core` → binary | 16m45s | **6m44s** | ## Also in this PR - **`src/web/build.rs`**: the derive has no file tracking, so a rebuilt `web/dist` silently shipped stale embedded assets until something else dirtied the api crate. `rerun-if-changed` on the embed folder fixes that. The folder is parsed from the `#[folder]` attribute rather than hardcoded, so it stays correct when the e2e workflows rewrite it. A missing `index.html` now prints a build warning instead of silently embedding nothing (the #13197 failure mode). - **Playwright workflows**: the embed-folder `sed` now targets `src/web/src/lib.rs` and is guarded with `grep -q`, so a future path drift fails at the sed step instead of silently shipping an empty UI and failing at the login page (see openobserve/o2-enterprise#2175 for that incident). - **`deny.toml`**: allow the new crate's AGPL-3.0 license (CI runs `cargo deny check licenses`, which requires a per-crate exception). ## Validation - `cargo deny check licenses` passes - Smoke-tested the binary: `/healthz` ok, `/web/` serves `index.html`, JS assets return 200 from the embedded bundle - Enterprise overlay resolves: `cargo metadata` passes with the companion `Cargo.toml.openobserve` change ## Companion openobserve/o2-enterprise#2184 carries the overlay + e2e workflow sync on the **same branch name** (`perf/web-embed-crate`); the two should merge together. |
||
|
Huaijin
|
bcea47f644 |
refactor: extract management API crate (#13202)
## Summary - add the `openobserve-api-management` crate under `src/api_management` - physically move management handlers, models, and authentication helpers out of `openobserve-api` - preserve the existing API module paths through re-exports - move shared stream permission checking into `openobserve-core` to avoid a management-to-query dependency - allow the new crate's `AGPL-3.0` license in `deny.toml` (fixes the cargo-deny `check` job) - keep router, OpenAPI generation, and rate-limit path mapping in `openobserve-api` ## Dependency boundary `openobserve-api-management` depends on `openobserve-api-common` and `openobserve-core`; it does not depend on `openobserve-api-query`. ## Note on `split-debuginfo` Removing `split-debuginfo = "packed"` from `[profile.release-profiling]` is intentional (not a formatting artifact): profiling builds now keep DWARF debuginfo embedded in the binary, so profiling artifacts can be symbolized at runtime without shipping a separate debuginfo file. The o2-enterprise overlay `Cargo.toml.openobserve` is kept in sync. ## Companion PR - openobserve/o2-enterprise#2175 ## Validation - OSS check with `mimalloc` - OSS strict clippy with all targets - enterprise check and strict clippy - cloud check and strict clippy - `cargo test -p openobserve-api-management --lib`: 300 passed - `cargo deny check licenses` - `cargo fmt --all` - `git diff --check` Build-time benchmarking is intentionally deferred for this stage. |
||
|
Huaijin
|
cd8610cafa |
refactor: extract shared and query API crates (#13199)
## Summary
- add `openobserve-api-common` for HTTP extractors and request DTOs
shared across API domain crates
- add `openobserve-api-query` and physically move the search, PromQL,
and traces handlers (about 16.4K lines)
- preserve the existing `handler::http::request::{search,promql,traces}`
paths through re-exports so routing, Swagger generation, and rate-limit
metadata remain compatible
- forward enterprise, cloud, vectorscan, and vortex features through the
new crate boundary
- avoid `#[path]` source inclusion
## Enterprise overlay
Requires openobserve/o2-enterprise#2174, which registers the two new
workspace crates in `Cargo.toml.openobserve`.
## Validation
- `cargo check -p openobserve --features mimalloc`
- enterprise check with `--no-default-features --features
enterprise,mimalloc`
- cloud check with `--no-default-features --features cloud,mimalloc`
- OSS and cloud/enterprise all-targets Clippy with `-D warnings`
- `openobserve-api-common`: 9 tests passed
- `openobserve-api-query`: 136 independent tests passed single-threaded;
the unchanged DB-dependent saved-view test was excluded because
standalone crate tests do not initialize the infra database
- `cargo fmt --all -- --check`
Build-time benchmarking is intentionally deferred until the crate-split
series is ready for a controlled `release-profiling` comparison.
|
||
|
Hengfei Yang
|
17dbff89b3 |
feat: skip file download cache for data older than configurable max age (#13201)
Design at: #13200 ## What changed Adds a max-age admission policy for the file download cache: - New configs `ZO_DISK_CACHE_MAX_AGE_DAYS` and `ZO_MEMORY_CACHE_MAX_AGE_DAYS` (default `0` = disabled, current behavior unchanged). - `queue_download()` now skips files whose `max_ts` is older than the configured max age — they never enter the download queue; queries read them directly from object storage (the existing cache-miss fallback). - The `cache_latest_files` gRPC event handler applies the same filter when collecting files, covering the batch `download_from_node` path that writes to the cache via `file_data::set` without going through the queue (e.g. merged files produced by compaction of old partitions). - The check is exposed as `file_downloader::exceeds_cache_max_age(ts, cache_type)`; `ts <= 0` (unknown timestamp) is never skipped. ## Why `time_lru` has an eviction policy but no admission policy. With a cache sized for the recent hot window (e.g. 3 days), a query on old data (e.g. 30 days ago): 1. Triggers gc **before** insert: the first old-file insert evicts `max(release_size, data_size * 100)` — by default ~10% of the cache — from the oldest hours of the *hot* window. 2. Once inserted, the old files themselves become the oldest `time_lru` bucket and are evicted first, often within the same query — S3 download → disk write → immediate delete, pure wasted work with zero retention for repeat queries. With a max age matching the cache window, deep-history queries no longer disturb the hot cache or waste S3 bandwidth and disk IO. ## Coverage All query-side download paths funnel through `queue_download()` with the file's real `max_ts` and are covered: search (parquet), tantivy index (`.ttv`), promql/metrics, and file_list dump. Deliberately **not** gated: compactor working downloads (`cache_remote_files` and merge-output writes — required reads/writes, not speculative caching) and the results/aggregations/metrics_results caches (separate concern). ## Tests Unit tests for the age check (disabled, unknown ts, recent file, old file); `cargo test -p openobserve-core --lib file_downloader` passes. |
||
|
Huaijin
|
5f65481821 |
refactor: extract background jobs crate (#13189)
## Summary - extract background scheduling code into the src/jobs workspace crate - physically move job sources without path-based module indirection - move file downloading, MMDB initialization signaling, and grouped-alert delivery behind core service APIs to keep the dependency direction one-way - preserve the existing root job API through re-export and forward enterprise, cloud, vectorscan, and vortex features Companion o2-enterprise PR: https://github.com/openobserve/o2-enterprise/pull/2173 ## Validation - cargo check -p openobserve --features mimalloc - cargo clippy -p openobserve -p openobserve-api -p openobserve-core -p openobserve-jobs --all-targets --features mimalloc -- -D warnings - enterprise cargo check with the o2-enterprise overlay - cloud cargo check and strict Clippy with the o2-enterprise overlay - cargo fmt --all -- --check Build-time benchmarking is intentionally deferred until the crate-splitting series is complete. |
||
|
|
b92d5765ed |
refactor: extract API frontend crate (#13181)
## Summary Second step of the crate split (after #13168 which extracted `openobserve-core`): move `src/handler` and `src/router` into a new `openobserve-api` crate at `src/api`, so the HTTP/gRPC transport frontend (routing, auth, extractors, OpenAPI generation) builds as its own compilation unit. All 155 moved source files are byte-identical moves (`git mv` only, zero content changes). The real changes are limited to manifests and scaffolding: - **`src/api/src/lib.rs`** re-exports `openobserve_core::{common, job, service}` (plus `cipher` under `enterprise`) at the crate root, so the existing `crate::common::...` / `crate::service::...` paths inside the moved files keep resolving without touching them. - **`src/lib.rs`** re-exports `openobserve_api::{handler, router}`, so external paths like `openobserve::handler::...` used by `main.rs` and `tests/integration_test.rs` are unchanged. - **Feature forwarding**: `enterprise` / `cloud` / `vectorscan` / `vortex` / `profiling` on the root crate now forward to `openobserve-api` as well. - **`src/enterprise-stubs` → `src/enterprise`**: directory rename, plus a new `o2_ratelimit` dependency-resolution stub (the api crate depends on it under the `enterprise` feature; there was no stub for it before). - `#![feature(variant_count)]` moves with its only use site (`models/dashboards.rs`) into the api crate. - **CI**: added `deny.toml` license exceptions for the two new AGPL-3.0 workspace crates (`openobserve-api`, `o2_ratelimit`); updated the `ui.rs` path patched by the playwright workflows. `src/api/Cargo.toml` is shared verbatim between OSS and enterprise builds — only the root manifest is replaced by the enterprise overlay; the `o2_*` deps resolve via `workspace = true` to either the stubs (OSS) or the real crates (enterprise). Companion PR: openobserve/o2-enterprise#2168 ## Testing - `cargo check --workspace` passes (OSS). - Verified all moved files are 0-diff renames (`git diff -M90% --stat`). - Verified every `crate::` reference in the moved code resolves through the re-exports (`service`/`common`/`handler`/`job`/`router`/`cipher`), with the single `crate::cipher` use correctly gated behind `enterprise`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
|
HuaijinandClaude Fable 5
|
81cc0f2255 |
refactor: extract application core crate (#13168)
## Summary Extract the application core (`src/common`, `src/service`, `src/job`, `src/cipher`) out of the root binary crate into a new workspace crate `src/core` (package `openobserve-core`). The root crate keeps the binary entrypoints and edges: `cli`, `handler`, `router`, `migration`, and `super_cluster_queue`. This builds on #13162 (remove service → handler dependencies), which made the extraction possible. > Naming note: the directory is `src/core` but the package stays `openobserve-core` — a Cargo dependency literally named `core` shadows Rust's built-in `core` crate and breaks the `::core::` paths emitted by derive macros (thiserror, serde, ...). ## What moved - `src/common`, `src/service`, `src/job`, `src/cipher` → `src/core/src/...` (467 files moved; 385 of them byte-identical, the rest only needed `pub(crate)` → `pub` widening and clippy fixes required by the new crate boundary) - `ua_regex/regexes.yaml` → `src/core/ua_regex/` (same content, consumer moved with it) - `src/handler/grpc/request/search/mod.rs` → `src/core/src/service/search/grpc_server.rs` (byte-identical; old path kept via `pub use ... as search`) - `impl From<MetricsQueryRequest> for cluster_rpc::MetricsQueryRequest` moved from `handler/grpc/mod.rs` into `service/promql` (orphan rule; field-for-field identical) - The `impl From<XError> for Response` blocks scattered across HTTP handler files (alerts, dashboards, reports, folders, pipeline, eval_jobs, providers, ratelimit) are consolidated into `src/core/src/service/http.rs` with unchanged status-code mappings ## Compatibility `src/lib.rs` re-exports the moved modules (`pub use openobserve_core::{common, job, service}`, and `cipher` under the `enterprise` feature), so all existing `crate::service::...` / `openobserve::common::...` paths keep working — handlers, tests, and downstream code need no changes. ## Enterprise dependency resolution `openobserve-core` optionally depends on `o2_enterprise` / `o2_dex` / `o2_openfga` / `vortex` behind features. So that the OSS repo resolves on its own, workspace deps point at empty stub packages under `src/enterprise-stubs/` (never compiled into default builds). Enterprise builds replace the workspace manifest with `Cargo.toml.openobserve`, which points these at the real crates. ⚠️ Must be merged together with openobserve/o2-enterprise#2162 (same branch name) — merging either PR alone breaks the enterprise build. ## Feature forwarding Root `enterprise` / `cloud` / `vectorscan` / `vortex` features forward to the corresponding `openobserve-core` features; the core crate forwards to `config` / `infra` / `o2_enterprise` exactly as the root crate did before. ## Unit-test fix The `service::sourcemaps` tests relied on the workspace root being the test cwd (shared zip fixture via a relative path, sqlite db dir/tables created by `init-db` from `coverage.sh`). Unit tests of a sub-crate run with the crate dir as cwd, so they now read the fixture via `CARGO_MANIFEST_DIR` and create the db dir + the tables they touch themselves; `src/core/data` is cleaned between the postgres and sqlite CI passes and gitignored, mirroring `src/infra/data`. ## Cleanup The extracted crate's manifest is pruned to what core actually uses (45 dependency entries copied from the root crate plus a dead `profiling` feature removed), and the stale `roaring` dependency (unused since the tantivy result cache rewrite in #7890) is dropped from the root manifest. All new files carry the standard AGPL header. ## Testing - `cargo check --workspace --all-targets`, `cargo clippy -D warnings`, and `cargo fmt --check` pass - `service::sourcemaps` unit tests pass under `--test-threads=1` (the CI setting; they share one db group and are not parallel-safe, same as before) - No logic changes anywhere: the diff is moves + visibility widening + clippy appeasement; error-to-HTTP-response mappings verified variant-by-variant - e2e label kept so Playwright runs --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Huaijin
|
9700fd6c43 |
perf: parallel zstd compress (#12960)
related to https://github.com/openobserve/o2-enterprise/issues/1860 - [x] limit parallel by the target partitions ## Summary This PR improves Flight `do_get` response throughput by executing and encoding output partitions in parallel instead of funneling everything through a single sequential stream/encoder path. ## Changes - Switches Flight `do_get` from `execute_stream` to `execute_stream_partitioned`. - Adds per-partition encoder tasks that coalesce RecordBatches, encode Arrow IPC, and apply ZSTD compression independently. - Splits Flight encoding into: - `ChunkCoalescer` for 8192-row batch coalescing. - `FlightDataEncoder` for IPC/ZSTD encoding of completed chunks. - Adds `PartitionEncoderStream` to handle per-partition batch coalescing and encoding. - Preserves custom Flight messages behavior: - early-emitted messages are sent before the first data message; - stats/metrics/peak-memory messages are sent after data completes. - Uses a fresh dictionary tracker per encoded chunk so dictionary-encoded columns remain decodable when chunks are encoded independently. - Cleans up stream lifecycle handling by aborting partition tasks before clearing session resources. ## Performance 100M records single node optimzier disable(to simulate the distribute query) ### Q1 ```sql SELECT trace_id, count(*) AS cnt, max(_timestamp) AS ts FROM "k8s_logs" GROUP BY trace_id ORDER BY cnt DESC LIMIT 100 ``` 12M trace_id unique value 12M records send from follower to leader main branch: 6s this branch: 3s ### Q2 ```sql SELECT count(distinct request_id) FROM "k8s_logs" ``` 100M request_id unique value 100M records send from follower to leader main branch: 30s this branch: 11s ### Q3 ```sql SELECT kubernetes_namespace_name, COUNT(DISTINCT trace_id) AS cnt FROM k8s_logs GROUP BY kubernetes_namespace_name ORDER BY cnt DESC ``` 10 kubernetes_namespace_name unique value, 12M trace_id unique value 63M records send from follower to leader main branch: 21s this branch: 8.3s ### Q4 ```sql SELECT host, COUNT(DISTINCT trace_id) AS cnt FROM k8s_logs GROUP BY host ORDER BY cnt DESC ``` 2160 host unique value, 12M trace_id unique value 99.57M records send from follower to leader main branch: 56s this branch: 20s |
||
|
Huaijin
|
ce726253f8 | fix: projection for hashjoin in broadcast join (#12875) | ||
|
Huaijin
|
d7b9cc495c | chore: use datafuson proto instead of copy (#12812) | ||
|
Huaijin
|
be9113c735 | fix: limit prune issue (#12758) | ||
|
Huaijin
|
5be5a302d3 |
perf: improve approx_distinct performance (#12714)
- close https://github.com/openobserve/o2-enterprise/issues/1862 ## todo - [x] fix access plan - [x] use forked datafusion |
||
|
Huaijin
|
0831badd02 |
perf: upgrade datafusion to 54 (#12665)
- [x] ~add sort when to many groups, we can do this, but for now do not do this~ - [x] remove the unsafe for memory pool - [x] add approx_distinct improve, already fork, will add in next pr - [x] fork vortex to add cast fix, and export `.exclude_schemes([IntDictScheme.id()])` - [x] ~enable_physical_uncorrelated_scalar_subquery test~ |
||
|
Hengfei Yang
|
890771e319 | perf: keep improve histogram on tantivy (#12693) | ||
|
Huaijin
|
f548a18de4 | refactor: access plan struct improve (#12646) | ||
|
Hengfei Yang
|
4c43f00036 | perf: keep improve histogram on tantivy (#12640) | ||
|
Hengfei Yang
|
d996349aa4 |
feat: rewrite tantivy histogram collectors, support histogram() timezone in optimize rules (#12627)
Design at: #12620 Closes #12620 Fixes #12564 ## What Rewrites both tantivy histogram optimize-rule collectors following the same approach as the recent `TopNCollector` / `SingleSegmentDocIdCollector` rewrites, and adds timezone support to both rules. - `MultiHistogramCollector` (replaces the aggregation framework for `SimpleMultiHistogram`) - `SimpleHistogramCollector` (replaces `tantivy::collector::HistogramCollector` for `SimpleHistogram`) ### Timezone support (#12564) `histogram(ts, interval, tz)` rewrites to `date_bin(interval, to_timestamp_micros(_timestamp + offset), origin)`; the optimizer's `get_column_name` hit the `BinaryExpr` and silently skipped both rules. Now: - the visitors extract the fixed offset from that shape (`get_timestamp_offset`), so the rules fire with a timezone - `IndexOptimizeMode::SimpleHistogram` / `SimpleMultiHistogram` carry a `ts_offset` field (no proto change — histogram modes never cross gRPC, they're re-derived per node) - bucket edges are computed in shifted wall-clock space and the collectors bucket on `_timestamp + ts_offset`, so emitted keys match `date_bin` exactly when merging with non-tantivy partials ## Testing - New collector unit tests: dense vs sparse equality, ts_offset shifting, bounds filtering, missing column / missing breakdown docs, per-bucket truncation with tie-breaks - Visitor tests now include real-DataFusion-plan timezone cases (`'+08:00'` simple, `'-05:30'` multi) asserting the extracted shifted bounds and offset - `cargo test -p config` (1936 passed), plus `index_optimizer` / `search::tantivy` / `tantivy_optimize_exec` suites in the main crate; clippy clean - Low cardinality: fast `20%` than main ``` histogram(_timestamp, '1h') AS ts, count(*) FROM default GROUP BY ts ORDER BY ts ASC ``` - Low cardinality: fast `30%` than main ``` histgoram(_timestamp, '1h') AS ts, breakdown, count(*) FROM default GROUP BY ts, breakdown ORDER BY ts ASC ``` - High cardinality: faster `5x` than main ``` histgoram(_timestamp, '1h') AS ts, breakdown, count(*) FROM default GROUP BY ts, breakdown ORDER BY ts ASC ``` |
||
|
Hengfei Yang
|
9c82beeeed |
fix: OTLP/JSON metrics silently dropping histograms and int values (#12615)
Close #12345 ## Problem OTLP histogram metrics sent over HTTP/JSON (e.g. otel-collector with `encoding: json`) created streams but stored 0 events, with no `_count`/`_sum`/`_bucket` streams. Int-valued counters/gauges were silently recorded as 0. The protobuf path was unaffected, which is why this only surfaced for collectors explicitly configured with JSON encoding. ## Root cause `otlp_json` deserializes the body directly into opentelemetry-proto's serde-derived types, which only partially implement the proto3 JSON mapping (64-bit integers encoded as strings, enums by name, zero/empty fields omitted). `Metric.data` is a flattened `Option` oneof, and serde converts any inner deserialization error of a flattened `Option` into `None` instead of failing — so the whole metric silently became `data: None`, while the stream itself had already been created by the schema setup. Same mechanism on `NumberDataPoint.value` turned string `asInt` values into 0. ## Changes - Upgrade opentelemetry stack 0.31 → 0.32, which fixes string-encoded u64 handling for plain histograms and summaries upstream. - Add `otlp_json_compat` normalizer for the gaps still present in 0.32 (each verified empirically, with unit tests): - string `asInt` values on data points and exemplars - `aggregationTemporality` as enum name - `Exemplar`'s value oneof is not flattened in 0.32 - missing `serde(default)` on `ExponentialHistogramDataPoint`/`Buckets`/`ValueAtQuantile`/`Exemplar`, which rejected omitted zero/empty fields - Count and log metrics whose data oneof failed to decode (`partial_success.rejected_data_points`) instead of silently skipping them, and stop overwriting specific partial-success error messages with the generic retention message. - Adapt to OTLP 0.32 API additions (`KeyValue.key_strindex`, `AnyValue::StringValueStrindex`). - Upgrade workspace reqwest 0.12 → 0.13, required because opentelemetry-http 0.32 only implements `HttpClient` for reqwest 0.13 (`rustls` feature replaces `rustls-tls-native-roots` via the platform verifier; `json`/`query`/`form` are now explicit features). The `segment` crate is pinned to reqwest 0.12, so `TELEMETRY_CLIENT` moved into `telemetry.rs` as a plain `reqwest::Client` that posts the Track message itself, replicating `segment::HttpClient::send`. - Remove the unused pyroscope profiling integration (optional feature, dependencies, agent setup, `ZO_PROF_PYROSCOPE_*` config). ## Verification - 10 unit tests in `otlp_json_compat` covering the full payload matrix (collector-style string integers, enum names, omitted zero fields, exemplars with and without trace context, already-normalized payloads, malformed input). - End-to-end: posted the issue's collector-style JSON payload to `/api/default/v1/metrics` — `_bucket` (3 events, correct cumulative counts with exemplars attached), `_count`, `_sum` populated for histogram and exponential histogram, summary quantiles stored, `asInt` counter stored as 42 instead of 0. - `cargo clippy -- -D warnings` clean; lib test failures are pre-existing local-environment issues that reproduce identically on main. |