mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-05 08:09:22 +00:00
ee71df7ce73cec00a9dba13dbe4b2e910e16ebfc
412
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Huaijin
|
aaed33543c |
refactor(search): group DataFusion and Tantivy feature configs (#13664)
## Summary - add ZO_FEATURE_METRICS_PUSHDOWN_FILTER_ENABLED as an opt-in filter pushdown switch for metrics queries - group DataFusion and Tantivy feature configuration under the search config struct - move inverted-index result cache sizing, footer caches, skip threshold, TopN group limit, and bloom footer cache settings from limit to search - preserve the moved cache settings' environment variable names and defaults - apply the metrics-specific setting to object storage, WAL, and Flight query contexts ## Testing - cargo fmt --all -- --check - cargo check for config, infra, search, tantivy_utils, search_service, and promql-service - cargo test -p config every_env_config_default_parses --lib - cargo test -p config inverted_index_config --lib - cargo test -p search test_create_session_config --lib - cargo clippy for all affected crates with -D warnings |
||
 Ashish Kolhe
|
58c7dafa55 |
fix: notification templates v2 — rich content templates + alert chart images (#13640)
## Summary
Two coupled alerting features:
**1. Notification Templates v2 ("content" templates)**
- New `kind: content` template type: a structured spec (title, body
markdown, fields, links, rows, per-channel title overrides,
severity-filtered visibility) rendered natively per channel — existing
`custom` (raw body) templates are untouched and byte-identical on the
send path.
- Per-channel renderers: Slack Block Kit, Teams (Adaptive Card +
MessageCard), Discord embeds, PagerDuty Events v2, Opsgenie, ServiceNow,
SNS, HTML/text email, and a stable machine `webhook` envelope.
- Escaping is renderer-owned (mrkdwn entities, HTML encoding, markdown
stripping) with a shared URL-scheme allowlist; golden-fixture tests per
channel.
- Template editor UI: guided content form (fields/links row tables,
compact rows config, channel title overrides), live preview panel with
per-channel preview + payload tab, test-send (rate-limited,
`[TEST]`-marked).
- Migration `m20260802_000001_add_template_kind`; prebuilt default
template + org backfill.
**2. Alert chart images (stateless, storage-free)**
- Alerts can attach a metric-history chart PNG rendered from the
triggers self-reporting stream (`actual_value` per evaluation), with
warn/crit threshold lines and the firing point appended.
- Zero storage: the chart URL *is* the data — deflate+base64url payload
signed with truncated HMAC-SHA256 (org-bound, expiry embedded), rendered
on fetch by the HTTP-terminating node (`GET
/api/v2/{org}/alerts/charts/render`). Signature verified before inflate
(decompression-bomb guard), uniform 404s, per-node render semaphore +
per-minute query budget + small LRU.
- Renderer: plotters + embedded DejaVu Sans (Bitstream Vera license
included), gap-broken line segments so sparse evaluation history doesn't
draw false plateaus.
- Channel delivery: Slack image block, email inline CID (no remote
fetch), Discord multipart attachment, Teams/PagerDuty image URLs;
per-destination opt-in via template chart flag.
- Slack layout live-A/B'd on a real channel: content + buttons/chart
split across two same-colored attachments so the severity stripe spans
everything while staying under Slack's "Show more" fold; clickable link
title; "View in OpenObserve" keeps a reserved primary-button slot.
- Slack image-delivery fallback: Slack validates image URLs server-side
and rejects the ENTIRE message with `400 invalid_attachments` when its
proxy can't fetch the chart (VPN-only/private `ZO_WEB_URL`) —
live-verified. On that rejection the send strips the image blocks and
resends once (alert always delivers), sets a process-wide flag so later
sends pre-strip (reachability is deployment-wide, not per-destination),
and re-probes hourly. Zero config; observable via `slack_image_rejected`
/ `slack_image_pre_stripped` metric events.
## Testing
- 147 notification unit tests (resolve/render/preview/test-send incl.
per-channel goldens and hostile-input/injection coverage), 8 chart
payload tests, renderer PNG tests.
- Live-verified end to end: Slack (Slackbot chart fetches ~100ms,
cache-immutable semantics), SES email inline image, multi-severity
firing scenario (warning→critical) with correct threshold lines.
- FE: AddTemplate/ContentTemplateForm/preview specs; UX audit pass with
fixes.
- `cargo clippy -D warnings` clean in both OSS and enterprise
arrangements.
## Enterprise pairing
Must merge together with openobserve/o2-enterprise#2353 (workspace
twin-manifest mirror for the new OSS deps).
|
||
|
Huaijin
|
e1e3091c06 |
refactor: move streaming aggregates to OSS (#13589)
## Summary - move streaming aggregation cache, aggregate execution, and optimizer code into the OSS search crate - move cache aggregation SQL analysis into the SQL visitor and enable streaming aggregation in search_service without enterprise gating - move MetadataCountExec into OSS and use it for metadata count optimization in all builds - remove migration-only dead code while keeping the cache module under search Companion cleanup: openobserve/o2-enterprise#2327. ## Testing - cargo fmt --all -- --check - cargo test -p search metadata_count --lib - cargo test -p search aggregate_optimize_rewrite --lib - cargo test -p search streaming_aggregate --lib - cargo test -p search cache::streaming_agg --lib - cargo test -p search datafusion::optimizer::stream_aggregate --lib - cargo test -p search_service partition --lib - cargo check -p search_service |
||
|
Huaijin
|
872ceb724a |
refactor: move search and compaction optimizations to OSS (#13583)
## Summary - move optimized Tantivy simple and multi histogram collectors into the open-source config crate - move general broadcast join eligibility, rewrite, execution, TmpExec codec, and shared async state into open-source search - enable enrichment-table broadcast join in OSS, including join reordering, remote-scan rewriting, EnrichmentExec, and its physical-plan codec - move AggregateTopkExec, its heap/sort streams, physical optimizer rule, and codec into open-source search - replace the enterprise-only Aggregate TopK settings with OSS-owned `ZO_DF_USE_AGG_TOPK_HEAP` and `ZO_DF_TOPK_HEAP_MAX_LIMIT` settings - move `approx_topk` and `approx_topk_distinct` UDAFs into open-source search and register them in both OSS and enterprise builds - omit the Aggregate TopK benchmark, benchmark documentation, and data-generator example from OSS - move Bloom term extraction, post-merge build, `.bf` writing, and orphan cleanup into the OSS compaction crate - run the Bloom build path after compaction in both OSS and enterprise builds, with non-fatal fallback to regular Tantivy search - move the direct `tantivy` and `object_store` dependency ownership required by Bloom building into OSS compaction - enable sorted timestamp, block, RANK, broadcast join, Aggregate TopK, approximate TopK, and Bloom paths in both OSS and enterprise builds - retain cross-super-cluster TmpExec fetching behind the enterprise feature - remove feature-gated enterprise imports and use the shared OSS implementations - add fastdivide for histogram bucket computation - resolve the duplicated SLO test attribute warning and remove an unused test-only alert-state migration helper ## Tests - `cargo fmt --all -- --check` (OSS and enterprise) - `cargo clippy --locked -p search --lib -- -D warnings` - `cargo clippy --locked -p compaction --lib -- -D warnings` - `cargo clippy --locked -p o2_enterprise --lib -- -D warnings` - `cargo test -p config tantivy::query::histogram_collector --lib` (21 passed) - `cargo test -p search tantivy::search::tests --lib` (9 passed) - `cargo test -p search enrichment --lib` (2 passed) - `cargo test -p search broadcast_join --lib` (7 passed) - `cargo test -p search join_reorder --lib` (8 passed) - `cargo test -p search datafusion::distributed_plan::codec::tmp_exec --lib` (1 passed) - `cargo test -p search aggregate_topk --lib` (7 passed) - `cargo test --locked -p search approx_topk --lib` (8 passed) - `cargo test --locked -p config meta::slo --lib` (371 passed) - `cargo test --locked -p infra bloom --lib` (30 passed) - `cargo check --locked -p config --tests` - `cargo check --locked -p infra --tests` - `cargo check --locked -p search --lib` - `cargo check --locked -p compaction --lib` - `cargo check --locked` (OSS root package) - `cargo check --locked --workspace` (enterprise workspace) - `cargo check --offline -p compaction --lib --features enterprise` with the paired repositories - `cargo check --offline -p openobserve --no-default-features --features enterprise` with the paired repositories Companion enterprise cleanup PR: https://github.com/openobserve/o2-enterprise/pull/2326 |
||
|
Huaijin
|
015a20a184 |
refactor: remove HTTP API compatibility re-exports (#13560)
## Summary - remove the HTTP request/model compatibility re-exports introduced by the domain crate split - migrate router, OpenAPI, enterprise action-server, and integration-test callers to the crates that own each API - remove the ingest Content-Type re-export and declare the direct root/test dependencies - delete the pure `handler::http::models` facade module ## Validation - `cargo fmt --all -- --check` - `cargo clippy -p openobserve-api-ingest -p openobserve-api-http -- -D warnings` - `cargo test --test integration_test --no-run` - `cargo check -p openobserve --no-default-features` - enterprise overlay: `cargo test --offline --workspace --no-default-features --features enterprise --no-run` - cloud overlay: `cargo test --offline --workspace --no-default-features --features cloud --no-run` - `git diff --check` |
||
|
Huaijin
|
6c029f7a2c |
chore(deps): bump vortex to 1cb48912 (#13540)
Bumps the `openobserve/vortex` pin from `57ffbf2b` to `1cb48912` in the workspace manifest. ## Upstream API changes handled - Arrow interop moved into a dedicated `vortex-arrow` crate, surfaced as `vortex::arrow`. `vortex::array::arrow::*` and `vortex::dtype::arrow::*` no longer exist, so `FromArrowArray`, `FromArrowType`, `ToArrowType` and `ArrowSessionExt` are now imported from `vortex::arrow`. - `ToArrowType` is no longer in scope implicitly; added the explicit import where `DType::to_arrow_schema()` is called. - The `vortex::compressor` facade narrowed to `BtrBlocksCompressor`, `BtrBlocksCompressorBuilder`, `Scheme`, `SchemeId`. `SchemeExt` and the scheme list are no longer re-exported there, so `search` takes a direct `vortex-btrblocks` dependency (same git rev) for `SchemeExt` / `schemes::integer::IntDictScheme`, which the UTF8 compressor uses to exclude the int-dict scheme. No behavior change intended — the compressor configuration and write strategies are identical. ## Verification - `cargo check --workspace --all-targets` clean - `cargo fmt --all -- --check` clean - `cargo test -p search --lib datafusion::vortex` — 6 passed - `cargo test -p tantivy_utils --lib vortex` — 3 passed - `cargo test -p config --lib utils::parquet` — 21 passed Paired enterprise PR (same branch name): https://github.com/openobserve/o2-enterprise/pull/2303 |
||
|
Huaijin
|
17ef03b8d6 |
refactor: cut release build time in the web and telemetry paths (#13528)
Two changes that cut build time, both measured on the same machine (M2
Max,
12 cores) with `cargo build --features mimalloc --profile release`.
## 1. The `web` crate no longer depends on `config`
`src/web` is a leaf crate so the RustEmbed expansion over `web/dist`
only
re-runs when the assets change — that expansion is 149.8s, of which
144.1s is
macro expansion (the crate already sets `#[br = false]` / `#[gzip =
false]`, so
this is not compression).
It also depended on `config`, for exactly one value:
`cfg.common.base_uri`. That
put a 50MB embed downstream of a 724-crate subgraph, so **every `config`
edit
paid the 148.5s embed again** — which is precisely what the crate's own
doc
comment says it exists to avoid.
`ui_routes()` now takes the `<base href>` value from the caller and
`base_href_middleware` carries it as middleware state.
`create_app_router()`
already computed that exact string (`"{base_uri}/web/"`) for its own
redirect,
so it is now resolved once and passed in rather than recomputed per
request
from a second `get_config()` call. `base_uri` is env-derived, normalised
once at
config init and never touched by `refresh_config`, so hoisting it out of
the
request path cannot drift.
The web crate's dependency tree goes **724 → 92 crates**, and `cargo
tree -i
config` no longer lists it.
## 2. Logging and tracing setup moved out of `main.rs`
`setup_logs`, `enable_tracing` and their span-processor / exporter types
compose
very large generic trees. Everything `main.rs` monomorphizes is emitted
into the
binary's own codegen units — the last unit on the critical path,
overlapping
with nothing.
`-Ztime-passes` on the binary unit (91.1s total):
```
LLVM_thinlto 39.8s
LLVM_passes 37.5s
codegen_to_LLVM_IR 5.3s
link_binary 5.3s (run_linker 3.9s)
type_check_crate 0.4s
```
The `openobserve` library does not depend on `openobserve-api-http`, so
its unit
compiles in parallel with the API crates — in the cold build it finished
23.9s
before the binary needed it. Moving the setup into `src/telemetry.rs`
(563
lines) puts that work in the slack instead of at the tail.
Two tests moved with the code and were dropped rather than relocated:
`test_setup_logs` and `test_enable_tracing_error_handling` both install
a global
tracing subscriber, and `setup_logs()` calls `.init()` unguarded, so
they can
only ever pass depending on ordering. They used to race just with each
other
inside the bin test binary; keeping them would have moved that race into
the lib
test binary next to the `cli` and `migration` tests. Neither asserted
anything —
the second wraps the call in `catch_unwind` and discards the result.
`main.rs` 1304 → 712 lines. **Binary unit 88.8s → 80.7s**; the library
unit grows
30.9s → 62.3s and is fully absorbed by the existing slack.
## 3. Unrelated: a dead-code warning in `openobserve-core`
Not a build-time change, just a warning fix noticed while checking the
above.
`COMPARE_ROLLUP_LOOKBACK_MICROS` was gated on `any(feature =
"enterprise",
test)`, but its only two readers (`fetch_arm_aggregate`,
`fetch_arm_failures`)
are `enterprise`-only and the unit tests below it exercise only the pure
aggregation helpers. So in an OSS build `cargo check -p openobserve-core
--all-targets` warned `constant COMPARE_ROLLUP_LOOKBACK_MICROS is never
used`.
Gate narrowed to `enterprise`, and the doc comment — which claimed the
opposite
— corrected. Happy to split this out if you'd rather it not ride along.
## 4. Review follow-ups
- `enable_tracing` became a public library API in §2 without a doc
comment,
unlike `setup_logs` beside it. Documented: which span processors attach
under
which env vars, why the returned provider must be held rather than
dropped,
and that it and `setup_logs` are mutually exclusive since both call
`.init()`.
- Added the `<base href>` test described under Verification below.
Not taken: the suggestion to re-add `test_setup_logs` /
`test_enable_tracing_error_handling` to `src/telemetry.rs`. Both install
a
global subscriber against `.init()`, so they cannot be made
order-independent,
and neither asserted anything — see §2.
`test_resource_key_value_creation` was
deleted for the same reason; it asserted that
`opentelemetry::KeyValue::new`
stores the key it was given.
## Measured and rejected
Recording these so nobody repeats them.
**Moving route registration into the crates that own the handlers.** 317
of the
371 routes were registered in one 622-line function in
`openobserve-api-http`.
Splitting them out to api-management (234), api-search (46),
api-pipelines (23)
and api-ingest (9) worked as designed — api-http went 145.0s → 63.0s —
but the
cold build only improved 654.4s → 637.7s. Splitting across a crate
boundary
**copies** work rather than relocating it: without LTO each crate
instantiates
the axum/tower machinery for itself, so the four receiving crates grew
222.1s
combined and total CPU rose 190.8s. It also just relocated the
bottleneck —
api-management became the new tail at 135.1s, with api-search finishing
only
6.9s behind it, so the tail is a plateau rather than a peak. Not carried
in this
branch.
| Tried | Result |
|---|---|
| sccache | 97.4% hit rate, **0** wall-clock gain; first fill +54% |
| Remove aws-lc-sys from the critical path | **0** — its `dur=72.0s` in
`--timings` is contention, it needs 15.6s alone |
| `codegen-units = 16` on the glue crates | **31.4s slower**, CPU
+297.8s — `lto` defaults to `false`, so thin-local LTO is on and its
cost scales with CGU count |
| Trim DataFusion features | `datafusion-core` depends on
`datafusion-functions` without `default-features = false`, so the
expression features cannot be turned off from the facade at all; what
remains is worth 13.8s of CPU (0.25%) |
A caveat on reading `cargo --timings`: a unit's `duration` is **not**
its
intrinsic cost — it is inflated by CPU contention when the machine is
saturated.
The aws-lc entry above is the clearest example.
## Verification
At this revision:
- `cargo check --all-targets` clean on OSS — 0 errors, 0 warnings — and
each
commit checks clean standalone, so `git bisect` works
- `cargo test -p web` passes
Carried over from the revision these changes were first written against
(the
edits since are doc comments, the `ui_routes` parameter rename, two
deleted
tests, a `Cargo.lock` revert, and the `cfg` narrowing in §3 — which is a
no-op
when `enterprise` is on):
- `cargo check` clean on the enterprise feature combination
- Cold release build succeeds
- Smoke-tested against a running binary: `/healthz` 200, UI and API
routes
resolve, unknown paths 404
The `<base href>` rewrite — the one runtime behaviour §1 touches — is
now
pinned by `base_href_is_rewritten_to_the_mount_point`, and checked end
to end
against the real embedded `index.html`: `ui_routes("/abc/web/")` serves
`<base href="/abc/web/" />`, matching what the old per-request
`format!("{base_uri}/web/")` produced for `ZO_BASE_URI=/abc`. The
committed test
drives a synthetic HTML response instead, so it still asserts something
in
builds where `web/dist` is absent. Verified by mutation: breaking the
rewrite
fails it.
|
||
 Hengfei Yang
|
2bf19b2b44 |
feat(mcp): support MCP spec revision 2026-07-28 (stateless, dual-era server) (#13527)
Design at: #13526 ## Summary Implements MCP specification revision **2026-07-28** (stateless protocol) alongside the existing 2025-11-25 behavior on the same endpoint, following the spec's dual-era server model: requests carrying modern per-request `_meta` are served per 2026-07-28, while `initialize` selects legacy semantics. Legacy clients are unaffected. ## Changes - **Version negotiation** (`src/api/http/.../mcp/mod.rs`, `src/mcp/src/protocol.rs`): the `MCP-Protocol-Version` header is now validated against a supported-version set (`2026-07-28`, `2025-11-25`) instead of strict equality — the old check would have rejected every client that upgrades to the new revision. Unsupported versions (header or body `_meta`) get `400` + JSON-RPC `-32022` (`UnsupportedProtocolVersionError`) listing the supported versions so clients can retry. - **`server/discover`** (`src/mcp/src/protocol.rs`): new RPC required by 2026-07-28, returning `supportedVersions`, `capabilities`, server identity in `_meta`, `instructions`, and `ttlMs`/`cacheScope` cache fields. - **Modern result decoration** (`src/mcp/src/protocol.rs`): requests declaring `io.modelcontextprotocol/protocolVersion: 2026-07-28` in `_meta` get `resultType: "complete"` and `_meta` server identity on results, plus `ttlMs`/`cacheScope` on `tools/list`. Legacy responses carry none of these fields — byte-for-byte unchanged. - **Request metadata headers** (`src/api/http/.../mcp/mod.rs`): modern requests must mirror body fields into `Mcp-Method` / `Mcp-Name` headers (including the Base64 sentinel encoding for `Mcp-Name`); missing or mismatched headers are rejected with `400` + JSON-RPC `-32020` (`HeaderMismatch`), per the transport's server-validation rules. - **HTTP status mapping**: unknown methods on modern requests return `404` + `-32601` so dual-era clients can distinguish a modern server from a legacy one; version errors return `400`. Legacy requests keep errors in-band with `200` as before. - **Legacy paths untouched**: `initialize`, `ping`, the GET SSE keepalive channel, DELETE no-op, and notification → `202` handling are unchanged. Tool ordering was already deterministic (sorted at init), satisfying the new caching guidance. ## Testing - `cargo test -p openobserve-mcp` — 94 tests pass (new coverage: `server/discover` shape, unsupported-version error payload, modern decoration, legacy non-decoration, sentinel decoding, version set). - `cargo test -p openobserve-api-http --lib handler::http::request::mcp` — 13 tests pass (new coverage: modern happy path, missing/mismatched `Mcp-Method`/`Mcp-Name`, header↔body version mismatch, missing version header on modern requests, modern unknown method → 404, legacy header accepted). - `cargo clippy` clean on both crates. |
||
|
Hengfei Yang
|
94bddd7a7b |
feat: pack ingester wal persist into large files to avoid massive small files (#13497)
Design at: https://github.com/openobserve/openobserve/issues/13509 Closes #13509 ## Problem On clusters with 100k+ metrics streams, every memtable rotation persists **one parquet file per stream × hour**, producing thousands of tiny files per second on the ingester. The legacy file mover cannot keep up: it re-scans the whole `wal/files` tree every `file_push_interval` (capped at `file_push_limit` = 10000 files per cycle ≈ a hard ceiling of ~1000 files/s), repeatedly stats not-yet-expired files, and pays a full DataFusion plan + S3 PUT per stream×hour for a few KB of data. Generation outpaces consumption and wal files pile up unboundedly. ## Design Behind `ZO_FEATURE_WAL_PACK_ENABLED` (bool, default **off**), applying to **metrics streams only** — the stream-cardinality problem is metrics-specific; other stream types keep the legacy per-stream persist path. With the flag on, the "split by stream" step moves from persist time (disk) to upload time (memory): - **Persist**: all streams of a memtable are appended into a few large **pack** files (`wal/pack/{idx}/{memtable_id}.{seq}.pack`, rolled at `ZO_MAX_FILE_SIZE_ON_DISK`, default 512MB). Each segment inside a pack is a **complete, self-contained parquet file**; the pack footer (JSON + gxhash trailer, versioned) indexes segments with stream, partition key, offset/length and FileMeta. One rotation now writes O(1) sequential files instead of O(streams) tiny ones. - **Query**: an in-memory segment index (rebuilt from footers on startup) resolves stream → byte ranges; segments are materialized into record batches and reuse the memtable search table-building path. Dedup against in-memory data uses the existing memtable_ids mechanism. - **Mover**: pending streams are snapshotted from the segment index (no readdir/stat storm, no `file_push_limit` cap). A stream is flushed when its segments are big enough or the oldest exceeds `max_file_retention_time`. A **single segment is uploaded as-is (zero decode/re-encode)**; multiple segments are merged in memory via DataFusion MemTable + the existing `merge_parquet_files`. Object keys, file_list, incremental compaction trigger, tantivy index, cache_latest_files and enterprise service-streams all follow the legacy mover. **S3 layout, file_list schema, compactor and querier are completely unchanged.** - **Crash recovery**: packs follow the same `.lock` five-step flow as legacy files (`.pack.tmp` → `.lock` → delete wal → rename → unlock), keeping exactly-once semantics. Lock recovery stays synchronous on startup (it must run before the pack index rebuild), while the orphan `.par` sweep over the potentially millions-of-files `wal/files` tree moved to a background task so startup no longer blocks. - **Pack recycling**: a registry tracks per-pack `{total, consumed, readers}`; a pack file is deleted once all segments are consumed and no in-flight query read holds it (read guards replace the legacy `lock_files`/pending-delete mechanism for packs). Also included: shared-memtable hash changed from `{thread_id}_{org_id}` to `{org_id}_{stream_name}` so one stream maps to exactly one bucket, removing the per-request round-robin fan-out that multiplied file/segment counts. ## Testing - Unit tests: pack format roundtrip, size-based rollover, corruption/truncation rejection, real-parquet segment read through the index with time-range/memtable-id filtering, consume lifecycle (pack deleted when fully consumed), reader-guard deferred deletion, flush conditions. - End-to-end with the real binary (4 phases): 1. wal binary format is byte-identical to main (`git diff main... -- src/wal src/ingester/src/entry.rs` is empty); 2. old-format wal written with the flag off **replays into packs** with the flag on — no migration needed; 3. mixed-source search returns exact counts with legacy wal parquet + pack segments + memtable coexisting (300/400/500 across three batches, filters and row fetches correct); 4. pack mover merges multiple segments into one storage file, generates the tantivy index, deletes consumed packs; final storage-side queries and `match_all` are exact. ## Unrelated fixes included Two startup paths shared the same "treat any db error as absence" bug: - `init_db` used to treat ANY error from `get_db_schema_version` as version 0 ("assuming default of 0 and trying upgrade") and kicked off the full db upgrade — so a merely unreachable or overloaded database got migration DDL piled on top of its existing load. Now `get_db_schema_version` returns 0 only when the version is genuinely absent (missing key, or the meta table itself not created yet on a fresh install); transient errors are retried 5 times with backoff and then abort the startup so the orchestrator restarts the node once the database recovers. - bootstrap did the same with the **instance id**: `instance::get()` reported a missing key as an error, so "not stored yet" and "db unreachable" landed in one arm that generated a fresh instance id — and could overwrite the stored one once the db recovered. The instance id must stay stable for the lifetime of a deployment, so a new one is now generated only when it is genuinely absent; read errors are retried with backoff and then abort the startup, and a failed write of a newly generated id is no longer ignored. ## Notes for reviewers - Flag is default **off**; with the flag off, behavior is unchanged except the startup fixes and the shared-memtable hash change. The flag only gates the persist path — pack search always runs (no-op on an empty index), so packs already on disk stay searchable after the flag is turned off while the mover drains them. - Verified with the real binary that with the flag on, a metrics stream is packed while a logs stream keeps the legacy per-stream parquet, both queryable (SQL + PromQL). - Pairs with o2-enterprise branch `feat/ingester-wal-pack-drain` (drain completion check also scans `wal/pack`); that change must land before/with enabling the flag in production, otherwise a draining node could shut down with pack segments not yet uploaded. - Known follow-ups (not in this PR): pack-specific metrics, extending pack to other stream types, large-scale load test for segment-index memory sizing, `wal-reader`-style debugging tool for packs. |
||
|
Huaijin
|
ef3617f157 |
refactor: split HTTP APIs by domain (#13479)
## Summary
- organize API crates under
`src/api/{common,grpc,http,ingest,management,pipelines,search}`
- keep ingest, management, pipelines, and search independent from one
another; compose them only in the HTTP frontend
- move pattern APIs into search and simplify `main.rs` wiring
- update workspace manifests, lockfile, integration tests, and
cargo-deny entries
- keep `web/` unchanged
## Validation
- `cargo fmt --all -- --check`
- `cargo clippy --all-targets -- -D warnings`
- `cargo deny check licenses`
- enterprise overlay: `cargo build --no-default-features --features
enterprise`
- enterprise overlay: `cargo build --no-default-features --features
cloud`
- enterprise overlay: `cargo clippy --all-targets -- -D warnings`
The matching `o2-enterprise` overlay PR will be linked after creation.
|
||
|
Huaijin
|
d165299bcd |
refactor: extract TLS and node services from core (#13449)
## Summary - extract TLS certificate loading and client configuration from openobserve-core into openobserve-tls - extract node discovery, cluster information, ingester gRPC channel selection, and Tokio runtime metrics into openobserve-node - update Core, API, API Management, Jobs, and the root binary to depend on the new crates directly - preserve one-way dependencies: openobserve-core depends on openobserve-node, while openobserve-node does not depend on Core - register both crates in the enterprise workspace overlay ## Validation - cargo test -p openobserve-node --lib - CARGO_INCREMENTAL=0 cargo clippy -p openobserve-node -p openobserve-core -p openobserve-jobs -p openobserve-api-management -p openobserve-api -p openobserve -- -D warnings - RUSTFLAGS="--cfg tokio_unstable" cargo check -p openobserve-node - enterprise overlay: CARGO_INCREMENTAL=0 cargo clippy -p openobserve --no-default-features --features enterprise -- -D warnings - cargo deny check licenses --hide-inclusion-graph - cargo fmt --all -- --check - git diff --check Companion enterprise PR: openobserve/o2-enterprise#2267 |
||
|
Huaijin
|
3f9ce27542 |
refactor: split service dependencies out of core (#13446)
## Summary - decouple stream cleanup from enrichment-data and remove the transitive compaction/DataFusion dependency from stream - extract sourcemaps and organization storage services from core into dedicated crates - extract GitHub built-in regex access and model pricing into openobserve-builtins - rename pipeline and organization-storage persistence modules from store to db - remove now-unused direct dependencies from core - synchronize the enterprise workspace overlay in openobserve/o2-enterprise#2266 ## Validation OSS: - cargo check -p openobserve-builtins -p openobserve-core -p openobserve-jobs -p openobserve-api - cargo test -p openobserve-builtins --lib - cargo test -p openobserve-builtins --doc - cargo clippy -p openobserve-builtins -- -D warnings - cargo check -p openobserve-org-storage --features enterprise Enterprise overlay: - copied ../o2-enterprise/Cargo.toml.openobserve over Cargo.toml - cargo fmt --all -- --check - cargo check --all-targets --no-default-features --features enterprise - cargo clippy --all-targets --no-default-features --features enterprise -- -D warnings The default enterprise + vectorscan build was attempted and reached vectorscan-rs-sys, but the validation host does not have cmake installed. The enterprise-only build and clippy run completed successfully. |
||
|
Huaijin
|
6123240a64 |
refactor: clean up core service dependencies (#13436)
## Summary
- remove the remaining openobserve-core Tantivy facade and let Jobs
depend on tantivy_utils directly
- inject enrichment-table cleanup from Core so the stream crate no
longer depends on enrichment-data (and transitively compaction)
- define a StreamDataCleanup port for organization deletion and move the
compaction implementation to Jobs
- remove openobserve-core's direct compaction dependency and enterprise
feature forwarding
- allow OSS builds to preserve ZO_FILE_FORMAT=vortex now that Vortex
support is always compiled
- add regression coverage for Vortex configuration and injected stream
cleanup failures
## Dependency direction
jobs -> tantivy_utils
jobs -> compaction
jobs -> openobserve-core::StreamDataCleanup
core -> stream + enrichment-data
core -X-> compaction (direct)
stream -X-> enrichment-data -> compaction
## Verification
- cargo fmt --all -- --check
- cargo check -p stream -p openobserve-core -p openobserve-jobs
- cargo test -p stream --lib (24 passed)
- cargo test -p config --lib (2027 passed, 2 ignored)
- cargo test -p openobserve-core org_cleanup --lib (14 passed)
- cargo clippy -p config -p stream -p openobserve-core -p
openobserve-jobs -- -D warnings
Enterprise-only check is currently blocked by the existing
openobserve-cipher build error: o2_enterprise::enterprise is unavailable
in this checkout.
|
||
|
Huaijin
|
df9d35c3d3 |
feat(search): move vortex support from enterprise into open source (#13434)
## Summary
Moves Vortex file-format support out of the enterprise build and makes
it a core, always-compiled capability in open source.
- **Code move**: the enterprise `search/vortex.rs` module
(`Utf8Compressor`, `VORTEX_RUNTIME`, `write_vortex`,
`generate_vortex_access_plan`, plus its unit tests) now lives at
`src/search/src/datafusion/vortex.rs`; the three call sites
(`merge/downsampling.rs`, `merge/mod.rs`, `table_provider/helpers.rs`)
reference the local module.
- **Feature removal**: the `vortex` cargo feature is deleted from the
root and all 11 sub-crate manifests. All `#[cfg(feature = "vortex")]` /
`#[cfg(all(feature = "enterprise", feature = "vortex"))]` gates and
their "requires the vortex feature" error branches are gone —
`FileFormat::Vortex` match arms compile unconditionally in config,
search, and tantivy_utils.
- **Dependencies**: `vortex` / `vortex-datafusion` are now direct git
dependencies (`openobserve/vortex` rev `57ffbf2b`, the same rev the
enterprise workspace pinned); the empty dependency-resolution stub
crates under `src/enterprise/vortex{,-datafusion}` are deleted.
Companion PR in o2-enterprise removes the vortex module/feature there
and syncs `Cargo.toml.openobserve`; that PR should merge together with
(or right after) this one, since its template relies on this repo no
longer declaring the feature.
## Verification
- `cargo check --workspace` (OSS, default features) passes
- `cargo check -p search -p tantivy_utils -p config --all-targets`
passes (vortex tests compile)
- Enterprise combined workspace (`Cargo.toml.openobserve` swapped in)
`cargo check -p openobserve` passes
- `cargo fmt` applied to touched crates
|
||
|
Huaijin
|
1624a6ff24 |
fix(schema): restore persistence guards on auto-UDS settings write (#13431)
## Summary #13405 rewired `handle_diff_schema`'s auto-UDS persistence from `stream::save_stream_settings` to a direct `db::schema::update_setting` call (required by the new crate dependency direction: `schema` cannot depend on `stream`). The direct write dropped the persistence guards that path provided. This PR restores them inside the schema crate, without reintroducing a dependency on the stream service: - skip persisting when the stream is being deleted (`db::compact::retention::is_deleting_stream`), matching the old `save_stream_settings` early-return - enforce `cfg.limit.user_defined_schema_max_fields` on the persisted field list (previously, with `ZO_SCHEMA_MAX_FIELDS_TO_ENABLE_UDS` set above the cap, settings that were never persisted before #13405 would now be written); skipped writes are logged at warn level instead of silently dropped - sort `defined_schema_fields` before persisting so the stored order is deterministic (was `HashSet` iteration order, nondeterministic across nodes/restarts) Also restores the `calculate_files_size` unit tests that were dropped when `src/core/src/file_list.rs` was deleted in #13405 — they now live next to the function in `infra::file_list`. ## Validation - `cargo check -p schema -p infra` - `cargo test -p schema --lib` (7 passed) - `cargo test -p infra --lib file_list::tests::test_calculate_files_size` (5 passed) - `cargo clippy -p schema -p infra` - `cargo fmt -p schema -p infra -- --check` |
||
|
Huaijin
|
15275ae7cf |
refactor: split core foundation services (#13405)
## Summary This PR begins decomposing `openobserve-core` into focused service crates while preserving the existing application behavior and dependency direction. It extracts six independently owned areas: - `ingestion-common`: shared ingestion contracts and response types - `schema`: schema inference, validation, evolution, and persistence helpers - `compaction`: merge, retention, and compaction services - `enrichment-data`: enrichment-table lifecycle and runtime support - `promql-service`: PromQL service orchestration - `stream`: stream metadata, settings, retention, and lifecycle operations ## What changed - registered the extracted crates in the workspace and propagated enterprise/cloud/vectorscan/vortex features - moved implementations out of Core and updated API, Jobs, query, and queue consumers to use the new crates - removed Core facade aliases for extracted crates and converted remaining callers to direct crate paths - retained the small cross-service stream-delete coordinator in Core for now because it still coordinates Pipeline and Alert cleanup - restored required root/Core dependencies after validating the full enterprise build - added the missing `anyhow` dependency for enterprise-only `search_service::search_jobs` - updated the lockfile for the new crate dependency graph ## Dependency direction The extracted crates do not depend on `openobserve-core`. Core remains a temporary upper-level consumer while the remaining domain services are split incrementally. ```text ingestion-common -> common/config schema -> ingestion-common/db/infra stream -> schema/enrichment-data/db/infra compaction -> schema/search/search-service/db/infra openobserve-core -> ingestion-common/schema/stream/compaction/... api/jobs -> openobserve-core + extracted services ``` ## Enterprise compatibility The private OpenObserve manifest was updated in openobserve/o2-enterprise#2257 to register the extracted workspace members and dependencies. Related Enterprise PR: https://github.com/openobserve/o2-enterprise/pull/2257 ## Validation OSS manifest: - `cargo fmt --all -- --check` - `cargo clippy -- -D warnings` Using `o2-enterprise/Cargo.toml.openobserve` as the OpenObserve root manifest: - `cargo check --no-default-features --features enterprise` - `cargo check --no-default-features --features cloud` - `cargo clippy --workspace --no-default-features --features enterprise -- -D warnings` - `cargo clippy --workspace --no-default-features --features cloud -- -D warnings` Enterprise repository: - `cargo fmt --all -- --check` - `cargo clippy -- -A clippy::question-mark -D warnings` All checks above pass in the development profile. |
||
|
Huaijin
|
eec3207c24 |
refactor: decouple API compilation from web assets (#13403)
## Summary - move embedded UI routing from `openobserve-api` into the existing `web` crate - inject `web::ui_routes` into the API router while preserving the existing `ui_enabled` and missing-`dist` behavior - make `openobserve-api` and `web` sibling dependencies so they can compile independently ## Validation - `cargo fmt --all --check` - Enterprise manifest: `cargo clippy --workspace --no-default-features --features enterprise,mimalloc -- -D warnings` - `cargo clippy --workspace -- -D warnings` - `cargo clippy -p web --all-targets -- -D warnings` - `cargo test -p web` - `cargo deny check licenses` |
||
|
Huaijin
|
6adcae3a1f |
refactor: remove crate compatibility re-exports (#13381)
## Summary
- remove compatibility re-exports introduced during the recent crate
refactors
- migrate callers to the crates that own each API
- delete five pure facade modules and add the required direct crate
dependencies
- keep OSS and enterprise manifests aligned
## Validation
- cargo fmt --all -- --check
- cargo clippy --workspace -- -D warnings
- cargo test --locked --workspace --no-run
- enterprise overlay: cargo test --offline --workspace
--no-default-features --features enterprise --no-run
- enterprise overlay: cargo clippy --offline --workspace
--no-default-features --features enterprise -- -D warnings
- git diff --check
Companion enterprise PR uses the same branch name:
codex/remove-refactor-reexports.
## TODO / Follow-ups
- [ ] **Re-establish the pipeline write-guardrail lost with
`db::pipeline`.** That module was not a pure facade: it re-exported only
the six read-only accessors from `infra::pipeline` and its doc comment
recorded the invariant that *mutations must go through
`service::pipeline::store`* (which keeps the `ExecutablePipeline` caches
and the coordinator/super-cluster watch events in sync). Deleting it
points callers straight at `infra::pipeline`, which also exposes
`put`/`delete`/`drop_table` with no guardrail. Decide the layering and
enforce it:
- **Reads** may hit `infra` directly (raw persistence, no side effects).
- **Writes** must go through the wrapper that persists *and* emits the
coordinator/super-cluster event + refreshes the in-memory cache. That
wrapper is `db::<entity>::{set,delete}` for most entities (e.g.
`db::alerts::alert::set` →
`infra::coordinator::alerts::emit_put_event`), but for pipeline it is
`core::pipeline::store` (not `db`), because the
`STREAM_EXECUTABLE_PIPELINES` cache it maintains is a core-level
construct the `db` crate cannot build. The super-cluster receiver
(`super_cluster_queue::pipelines`) is the one sanctioned caller of
`infra::pipeline::{put,delete}` directly, and it manually pairs each
with `emit_*_event` to avoid a re-broadcast loop.
- Net invariant to protect: never call `infra::pipeline::{put,delete}`
without emitting the coordinator event — i.e. nowhere outside
`core::pipeline::store` and the super-cluster handler. A bare `infra`
write leaves in-memory caches stale until restart (see the warning
comment still live at `src/core/src/org_cleanup.rs:484`).
- [ ] Consider whether the pipeline write-wrapper living in
`core::pipeline::store` while every other entity's lives in
`db::<entity>` is worth unifying, or at least documenting as an
intentional exception.
|
||
|
Huaijin
|
f6560d7d02 |
refactor: decouple search from core (#13338)
## Summary - extract search orchestration from core into the `search_service` crate - keep the existing low-level `search` crate and directory names unchanged - extract usage reporting and audit services into dedicated crates - replace global async function pointers with narrow publisher traits - keep compatibility re-exports while updating workspace dependencies and feature propagation Companion enterprise PR: https://github.com/openobserve/o2-enterprise/pull/2229 ## Validation - `cargo check --workspace` - `cargo check -p search --tests` - `cargo test -p search_service --lib` (184 passed) - enterprise non-release build with the paired `o2-enterprise` manifest - `cargo fmt --all -- --check` - `git diff --check` |
||
|
Huaijin
|
e236c7e653 |
refactor: extract database layer into db crate (#13340)
## Summary - extract persistence, cache, watcher data access, and DB service modules from openobserve-core into a standalone workspace db crate - keep application orchestration that depends on search, organization provisioning, and other Core services behind the existing Core db facade - preserve existing call paths through re-exports while making the dependency direction openobserve-core -> db - move HTTP error mapping back to the Core/API boundary so db has no direct axum dependency - avoid provider/provisioner callback traits and function-pointer hooks ## Validation - cargo fmt --all - cargo check --all-targets - cargo check -p db --all-targets - Enterprise workspace: cargo check --all-targets --offline - Enterprise workspace: cargo check -p db --all-targets --features enterprise --offline Companion Enterprise manifest PR: openobserve/o2-enterprise#2230. |
||
|
Huaijin
|
b835ff2f86 |
refactor: consolidate vrl/js into transform crate and extract super_cluster_queue crate (#13324)
## Summary Two crate-split refactors continuing the workspace decomposition: **1. `transform` crate (renamed from `openobserve-vrl`, `src/vrl` → `src/transform`)** - Now the only crate depending on `vrl`, `vector-enrichment` and `rquickjs` directly; downstream crates use `transform::vrl::` / `transform::vector_enrichment::` re-export paths and drop their direct dependencies - The QuickJS runtime (`common/utils/js.rs`) moved into `transform::js` (js_security checks gated behind the crate's `enterprise` feature); `apply_vrl_fn`, `convert_to_vrl`/`convert_from_vrl` and `TRANSFORM_FAILED` moved from core - Exception: `src/config` keeps its direct `vrl` dependency (it sits below `transform` in the dependency graph) **2. `super_cluster_queue` extracted from the root crate into a workspace member** - Only depends on `openobserve-core`, `infra`, `config` and `o2_enterprise`, so it builds in parallel with `api` and edits no longer recompile the root crate (which re-embeds `web/dist` on every rebuild) - Gated behind its own `enterprise` feature — compiles as an empty crate in OSS builds so `cargo --workspace` keeps working against the `o2_enterprise` stub; `vectorscan` wired through for `re_pattern.rs` - Root `lib.rs` re-exports the crate, so `openobserve::super_cluster_queue::init()` in `main.rs` is unchanged Pairs with the `Cargo.toml.openobserve` template sync PR in o2-enterprise (must merge together). ## Test plan - `cargo check --workspace --all-targets` (OSS) passes - `cargo check -p openobserve --no-default-features --features enterprise` with the synced enterprise template passes - `cargo clippy -p super_cluster_queue --features enterprise,vectorscan` — zero warnings - `cargo clippy -- -D warnings` (CI invocation) passes; `cargo fmt --all` clean |
||
|
Hengfei Yang
|
33d1da9b6e |
feat: move MCP server to OSS (#13272)
## Summary - move the MCP protocol, tool discovery, schema simplification, execution, and response filtering implementation into a new OSS `openobserve-mcp` workspace crate - enable the real MCP POST, GET, and DELETE handlers in community builds - initialize the MCP tool catalog from the OpenAPI document for every edition - keep OAuth authorization-server discovery enterprise-only because it depends on Dex - add OSS handler and OpenAPI initialization coverage Closes #13266 Design at: #13266 ## Impact Community users can connect MCP clients directly to OpenObserve and use the OpenAPI-backed tool catalog. Enterprise builds continue to use the same MCP implementation from the OSS workspace crate. Companion PR openobserve/o2-enterprise#2206 removes the old private copy and updates the enterprise workspace overlay. |
||
|
Huaijin
|
3b44da3eee |
refactor: extract shared common crate (#13252)
## Summary - introduce a `common` workspace crate for shared metadata, infrastructure helpers, and utilities - keep service-backed code in `openobserve-core` while preserving the existing `common` compatibility namespace - flatten `src/core/src/service/` into the standard `src/core/src/` module layout without using `#[path]` - retain `service::*` as compatibility re-exports from `service.rs` - move cipher request/response types from core into the search crate while preserving `openobserve_core::cipher` - remove unused Cargo dependencies from workspace crates Enterprise companion: https://github.com/openobserve/o2-enterprise/pull/2202 ## Validation - `cargo check --workspace --all-targets` - `cargo clippy -p common -p openobserve-core -p search --lib -- -D warnings` - common crate unit tests: 271 passed, 4 ignored - enterprise overlay: `cargo check -p openobserve-core --features cloud` - migrated cipher tests: 8 passed - `cargo deny check licenses` - `cargo fmt --all -- --check` - `git diff --check` |
||
    
|
4031f76426 |
feat: synthetic monitoring (#12849)
Design at : https://github.com/openobserve/designs/tree/main/synthetics ref: https://github.com/openobserve/o2-enterprise/issues/2095 --------- Co-authored-by: omkark06 <omkar@zinclabs.io> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: Claude <claude@users.noreply.github.com> |
||
|
Huaijin
|
66ca3d2439 |
refactor: extract PromQL evaluator into dedicated crate (#13239)
## Summary - add a dedicated promql workspace crate for parsing and evaluation - keep cluster routing, cache, WAL, file-list, and request orchestration in openobserve-core - preserve existing service::promql call sites through a compatibility facade; the facade re-exports an explicit item list (not a glob), and the promql search orchestration in core depends on the promql crate directly - inject enterprise query cancellation through TableProvider instead of coupling the evaluator to core search state - replace the core regex-UDF dependency with DataFusion regexp_like expressions ## Companion PR - openobserve/o2-enterprise#2198 updates the enterprise workspace override for the new promql crate ## Validation - cargo fmt --all -- --check - cargo check --workspace - cargo clippy -p promql --all-targets -- -D warnings - cargo clippy -p openobserve-core - cargo test -p promql: 363 passed - verified the promql dependency tree has no openobserve-core or search dependency - cargo check -p o2_enterprise --lib in the paired private workspace ## Notes The evaluator moves atomically in this PR. The existing service::promql::search subtree remains in core because it owns OpenObserve-specific orchestration rather than PromQL evaluation. The two web/ file changes only update source-path references in comments that pointed at the moved Rust files (src/service/promql/... -> src/promql/src/...). |
||
|
Huaijin
|
f36c9bd5f1 |
refactor: move search engine into search crate (#13238)
## Summary This is PR 2 of the atomic search-crate split, following #13234. - move the complete query-engine SCC into `src/search`: DataFusion, SQL, index, Tantivy, bloom pruning, inspector, and engine utilities - move 130 Rust files / 43,935 lines as a single atomic change - keep search orchestration, result-cache policy, cluster/gRPC/streaming, partitioning, usage, and audit responsibilities in `openobserve-core` - preserve existing `openobserve_core::service::search::*` paths through ordinary facade re-exports - move direct engine dependencies and feature ownership into the `search` manifest - remove core dependencies used only by the moved engine No query behavior change is intended. Source changes inside the moved engine are limited to crate-relative import paths and formatting. ## Dependency boundary ```text openobserve / API / jobs | v openobserve-core (orchestration) | v search (query engine) | v config / infra / proto / flight / openobserve-vrl ``` - `search` does not depend on `openobserve-core` - `openobserve-vrl` remains independent of both `search` and `openobserve-core` - the old core DataFusion, SQL, and Tantivy implementation directories no longer exist ## Validation - `cargo check --workspace` - `cargo check -p openobserve --features mimalloc` - `cargo clippy -- -D warnings` - `cargo clippy -p search -p openobserve-core --lib -- -D warnings` - `cargo fmt --all -- --check` - Cargo metadata dependency-boundary checks - `cargo test -p openobserve-vrl --lib` — 7 passed - `cargo test -p search --lib` — 876 passed; 12 existing histogram static-configuration tests fail in this local environment - `cargo test -p openobserve-core --lib` — 2407 passed, 6 existing global-state/filesystem/DB-isolation tests failed, 18 ignored - filtered reruns excluding those existing environment-sensitive tests passed ## Not run - private Enterprise / Vectorscan / Vortex overlay builds - local/distributed runtime smoke tests - cold/incremental build and query runtime performance benchmarks No performance improvement is claimed by this PR until those benchmarks are completed. |
||
|
Huaijin
|
059058519e |
refactor: make search engine sources crate-independent (#13234)
## Summary - add `src/vrl` as a leaf package named `openobserve-vrl` with Rust path `openobserve_vrl` - move `QUERY_FUNCTIONS`, enrichment-table ownership, GeoIP/MaxMind table registration, VRL compiler configuration, and shared runtime helpers into `openobserve-vrl` - let ingestion, jobs, core, and search use the shared VRL integration without depending on one another - keep DataFusion UDF registration direct through `Sql.org_id`; no `QueryExecutionContext`, `PreparedQueryTransform`, or `prepare_query_transforms` - move the cipher registry, query parameters, and file-cache helpers to `search` - move the generic file downloader implementation to `infra` - add feature forwarding for the `search` crate - keep compatibility re-exports for existing API, ingestion, gRPC, jobs, and Enterprise call sites Enterprise companion: https://github.com/openobserve/o2-enterprise/pull/2196 |
||
|
Huaijin
|
d6166aac49 |
perf: speed up embedded web asset compilation (#13233)
## Summary - upgrade rust-embed-for-web from 11.3 to 11.4.1 - disable unused compile-time gzip and Brotli generation for embedded UI assets - keep browser response compression through the global tower-http CompressionLayer Related enterprise PR: https://github.com/openobserve/o2-enterprise/pull/2192 ## Benchmark Clean release-profiling build with mimalloc: - before: 21m 13s - after: 15m 45s - improvement: 5m 28s, or 25.8% Focused web crate compilation improved from 546.66s to 98.55s. ## Testing - cargo metadata --locked --no-deps --format-version 1 - cargo clean - cargo build --features mimalloc --profile release-profiling |
||
|
Huaijin
|
5dc22a0ee8 |
refactor: establish search crate boundary (#13217)
## Summary - add the minimal `openobserve-search` workspace crate and move shared search/cache types into it - move the concrete enrichment table type and cache ownership to the search crate while retaining normal compatibility re-exports - move trace schema constants to `config` so the SQL engine no longer reaches into the traces service - expose concrete enrichment read functions from `infra` and make DataFusion call them directly - construct the stateless VRL runtime directly in the transform UDF - keep search execution and orchestration in `openobserve-core` for the later physical moves - do not introduce `SearchRuntime`, provider traits, callbacks, `#[path]`, or a search-to-core dependency This combines the original crate-boundary step with the dependency cleanup needed before SQL/DataFusion files can move safely. ## Validation - `cargo test -p openobserve-search --lib` (11 passed) - `cargo test -p openobserve-core service::enrichment::tests --lib` (19 passed) - `cargo check -p openobserve --features mimalloc` - enterprise + mimalloc check using the o2-enterprise overlay - cloud + mimalloc check using the o2-enterprise overlay - `cargo fmt --all -- --check` - strict all-target clippy for `openobserve-search` and `openobserve-core` - strict library clippy for `infra` - dependency check confirms `openobserve-search` does not depend on `openobserve-core` `infra --all-targets` currently reports three pre-existing inconsistent digit-grouping warnings in `src/infra/src/table/entity/alert_dedup_state.rs`; they are unrelated to this change and are not included here. |
||
|
Huaijin
|
3559d483ae |
refactor: move web UI asset embedding into a dedicated leaf crate (#13211)
## Problem Every rebuild of `openobserve-api` re-runs the `RustEmbed` macro expansion, which embeds `web/dist` (53MB, 1035 files) and compresses every file with gzip + brotli(q11) — single-threaded, non-incremental, and it peaks at 8GB RSS. `-Ztime-passes` attributes ~540s of the 769s api compile to `expand_crate`. Net effect: touching one file in `openobserve-core` costs a **16m45s** rebuild on an M-series laptop, and neither incremental compilation nor codegen-unit tuning can help, because macro expansion happens before both. ## Fix Move the `#[derive(RustEmbed)]` into a new `src/web` leaf crate with no workspace dependencies. The embed + compression now only recompiles when `web/dist` changes; edits anywhere in the backend no longer pay for it. **Measured (same machine, default profile settings, `--profile release-profiling`):** | | before | after | |---|---|---| | `openobserve-api` compile | 769s | 231s | | touch `src/core` → binary | 16m45s | **6m44s** | ## Also in this PR - **`src/web/build.rs`**: the derive has no file tracking, so a rebuilt `web/dist` silently shipped stale embedded assets until something else dirtied the api crate. `rerun-if-changed` on the embed folder fixes that. The folder is parsed from the `#[folder]` attribute rather than hardcoded, so it stays correct when the e2e workflows rewrite it. A missing `index.html` now prints a build warning instead of silently embedding nothing (the #13197 failure mode). - **Playwright workflows**: the embed-folder `sed` now targets `src/web/src/lib.rs` and is guarded with `grep -q`, so a future path drift fails at the sed step instead of silently shipping an empty UI and failing at the login page (see openobserve/o2-enterprise#2175 for that incident). - **`deny.toml`**: allow the new crate's AGPL-3.0 license (CI runs `cargo deny check licenses`, which requires a per-crate exception). ## Validation - `cargo deny check licenses` passes - Smoke-tested the binary: `/healthz` ok, `/web/` serves `index.html`, JS assets return 200 from the embedded bundle - Enterprise overlay resolves: `cargo metadata` passes with the companion `Cargo.toml.openobserve` change ## Companion openobserve/o2-enterprise#2184 carries the overlay + e2e workflow sync on the **same branch name** (`perf/web-embed-crate`); the two should merge together. |
||
|
Huaijin
|
bcea47f644 |
refactor: extract management API crate (#13202)
## Summary - add the `openobserve-api-management` crate under `src/api_management` - physically move management handlers, models, and authentication helpers out of `openobserve-api` - preserve the existing API module paths through re-exports - move shared stream permission checking into `openobserve-core` to avoid a management-to-query dependency - allow the new crate's `AGPL-3.0` license in `deny.toml` (fixes the cargo-deny `check` job) - keep router, OpenAPI generation, and rate-limit path mapping in `openobserve-api` ## Dependency boundary `openobserve-api-management` depends on `openobserve-api-common` and `openobserve-core`; it does not depend on `openobserve-api-query`. ## Note on `split-debuginfo` Removing `split-debuginfo = "packed"` from `[profile.release-profiling]` is intentional (not a formatting artifact): profiling builds now keep DWARF debuginfo embedded in the binary, so profiling artifacts can be symbolized at runtime without shipping a separate debuginfo file. The o2-enterprise overlay `Cargo.toml.openobserve` is kept in sync. ## Companion PR - openobserve/o2-enterprise#2175 ## Validation - OSS check with `mimalloc` - OSS strict clippy with all targets - enterprise check and strict clippy - cloud check and strict clippy - `cargo test -p openobserve-api-management --lib`: 300 passed - `cargo deny check licenses` - `cargo fmt --all` - `git diff --check` Build-time benchmarking is intentionally deferred for this stage. |
||
|
Huaijin
|
cd8610cafa |
refactor: extract shared and query API crates (#13199)
## Summary
- add `openobserve-api-common` for HTTP extractors and request DTOs
shared across API domain crates
- add `openobserve-api-query` and physically move the search, PromQL,
and traces handlers (about 16.4K lines)
- preserve the existing `handler::http::request::{search,promql,traces}`
paths through re-exports so routing, Swagger generation, and rate-limit
metadata remain compatible
- forward enterprise, cloud, vectorscan, and vortex features through the
new crate boundary
- avoid `#[path]` source inclusion
## Enterprise overlay
Requires openobserve/o2-enterprise#2174, which registers the two new
workspace crates in `Cargo.toml.openobserve`.
## Validation
- `cargo check -p openobserve --features mimalloc`
- enterprise check with `--no-default-features --features
enterprise,mimalloc`
- cloud check with `--no-default-features --features cloud,mimalloc`
- OSS and cloud/enterprise all-targets Clippy with `-D warnings`
- `openobserve-api-common`: 9 tests passed
- `openobserve-api-query`: 136 independent tests passed single-threaded;
the unchanged DB-dependent saved-view test was excluded because
standalone crate tests do not initialize the infra database
- `cargo fmt --all -- --check`
Build-time benchmarking is intentionally deferred until the crate-split
series is ready for a controlled `release-profiling` comparison.
|
||
|
Hengfei Yang
|
17dbff89b3 |
feat: skip file download cache for data older than configurable max age (#13201)
Design at: #13200 ## What changed Adds a max-age admission policy for the file download cache: - New configs `ZO_DISK_CACHE_MAX_AGE_DAYS` and `ZO_MEMORY_CACHE_MAX_AGE_DAYS` (default `0` = disabled, current behavior unchanged). - `queue_download()` now skips files whose `max_ts` is older than the configured max age — they never enter the download queue; queries read them directly from object storage (the existing cache-miss fallback). - The `cache_latest_files` gRPC event handler applies the same filter when collecting files, covering the batch `download_from_node` path that writes to the cache via `file_data::set` without going through the queue (e.g. merged files produced by compaction of old partitions). - The check is exposed as `file_downloader::exceeds_cache_max_age(ts, cache_type)`; `ts <= 0` (unknown timestamp) is never skipped. ## Why `time_lru` has an eviction policy but no admission policy. With a cache sized for the recent hot window (e.g. 3 days), a query on old data (e.g. 30 days ago): 1. Triggers gc **before** insert: the first old-file insert evicts `max(release_size, data_size * 100)` — by default ~10% of the cache — from the oldest hours of the *hot* window. 2. Once inserted, the old files themselves become the oldest `time_lru` bucket and are evicted first, often within the same query — S3 download → disk write → immediate delete, pure wasted work with zero retention for repeat queries. With a max age matching the cache window, deep-history queries no longer disturb the hot cache or waste S3 bandwidth and disk IO. ## Coverage All query-side download paths funnel through `queue_download()` with the file's real `max_ts` and are covered: search (parquet), tantivy index (`.ttv`), promql/metrics, and file_list dump. Deliberately **not** gated: compactor working downloads (`cache_remote_files` and merge-output writes — required reads/writes, not speculative caching) and the results/aggregations/metrics_results caches (separate concern). ## Tests Unit tests for the age check (disabled, unknown ts, recent file, old file); `cargo test -p openobserve-core --lib file_downloader` passes. |
||
|
Huaijin
|
5f65481821 |
refactor: extract background jobs crate (#13189)
## Summary - extract background scheduling code into the src/jobs workspace crate - physically move job sources without path-based module indirection - move file downloading, MMDB initialization signaling, and grouped-alert delivery behind core service APIs to keep the dependency direction one-way - preserve the existing root job API through re-export and forward enterprise, cloud, vectorscan, and vortex features Companion o2-enterprise PR: https://github.com/openobserve/o2-enterprise/pull/2173 ## Validation - cargo check -p openobserve --features mimalloc - cargo clippy -p openobserve -p openobserve-api -p openobserve-core -p openobserve-jobs --all-targets --features mimalloc -- -D warnings - enterprise cargo check with the o2-enterprise overlay - cloud cargo check and strict Clippy with the o2-enterprise overlay - cargo fmt --all -- --check Build-time benchmarking is intentionally deferred until the crate-splitting series is complete. |
||
|
|
b92d5765ed |
refactor: extract API frontend crate (#13181)
## Summary Second step of the crate split (after #13168 which extracted `openobserve-core`): move `src/handler` and `src/router` into a new `openobserve-api` crate at `src/api`, so the HTTP/gRPC transport frontend (routing, auth, extractors, OpenAPI generation) builds as its own compilation unit. All 155 moved source files are byte-identical moves (`git mv` only, zero content changes). The real changes are limited to manifests and scaffolding: - **`src/api/src/lib.rs`** re-exports `openobserve_core::{common, job, service}` (plus `cipher` under `enterprise`) at the crate root, so the existing `crate::common::...` / `crate::service::...` paths inside the moved files keep resolving without touching them. - **`src/lib.rs`** re-exports `openobserve_api::{handler, router}`, so external paths like `openobserve::handler::...` used by `main.rs` and `tests/integration_test.rs` are unchanged. - **Feature forwarding**: `enterprise` / `cloud` / `vectorscan` / `vortex` / `profiling` on the root crate now forward to `openobserve-api` as well. - **`src/enterprise-stubs` → `src/enterprise`**: directory rename, plus a new `o2_ratelimit` dependency-resolution stub (the api crate depends on it under the `enterprise` feature; there was no stub for it before). - `#![feature(variant_count)]` moves with its only use site (`models/dashboards.rs`) into the api crate. - **CI**: added `deny.toml` license exceptions for the two new AGPL-3.0 workspace crates (`openobserve-api`, `o2_ratelimit`); updated the `ui.rs` path patched by the playwright workflows. `src/api/Cargo.toml` is shared verbatim between OSS and enterprise builds — only the root manifest is replaced by the enterprise overlay; the `o2_*` deps resolve via `workspace = true` to either the stubs (OSS) or the real crates (enterprise). Companion PR: openobserve/o2-enterprise#2168 ## Testing - `cargo check --workspace` passes (OSS). - Verified all moved files are 0-diff renames (`git diff -M90% --stat`). - Verified every `crate::` reference in the moved code resolves through the re-exports (`service`/`common`/`handler`/`job`/`router`/`cipher`), with the single `crate::cipher` use correctly gated behind `enterprise`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
|
HuaijinandClaude Fable 5
|
81cc0f2255 |
refactor: extract application core crate (#13168)
## Summary Extract the application core (`src/common`, `src/service`, `src/job`, `src/cipher`) out of the root binary crate into a new workspace crate `src/core` (package `openobserve-core`). The root crate keeps the binary entrypoints and edges: `cli`, `handler`, `router`, `migration`, and `super_cluster_queue`. This builds on #13162 (remove service → handler dependencies), which made the extraction possible. > Naming note: the directory is `src/core` but the package stays `openobserve-core` — a Cargo dependency literally named `core` shadows Rust's built-in `core` crate and breaks the `::core::` paths emitted by derive macros (thiserror, serde, ...). ## What moved - `src/common`, `src/service`, `src/job`, `src/cipher` → `src/core/src/...` (467 files moved; 385 of them byte-identical, the rest only needed `pub(crate)` → `pub` widening and clippy fixes required by the new crate boundary) - `ua_regex/regexes.yaml` → `src/core/ua_regex/` (same content, consumer moved with it) - `src/handler/grpc/request/search/mod.rs` → `src/core/src/service/search/grpc_server.rs` (byte-identical; old path kept via `pub use ... as search`) - `impl From<MetricsQueryRequest> for cluster_rpc::MetricsQueryRequest` moved from `handler/grpc/mod.rs` into `service/promql` (orphan rule; field-for-field identical) - The `impl From<XError> for Response` blocks scattered across HTTP handler files (alerts, dashboards, reports, folders, pipeline, eval_jobs, providers, ratelimit) are consolidated into `src/core/src/service/http.rs` with unchanged status-code mappings ## Compatibility `src/lib.rs` re-exports the moved modules (`pub use openobserve_core::{common, job, service}`, and `cipher` under the `enterprise` feature), so all existing `crate::service::...` / `openobserve::common::...` paths keep working — handlers, tests, and downstream code need no changes. ## Enterprise dependency resolution `openobserve-core` optionally depends on `o2_enterprise` / `o2_dex` / `o2_openfga` / `vortex` behind features. So that the OSS repo resolves on its own, workspace deps point at empty stub packages under `src/enterprise-stubs/` (never compiled into default builds). Enterprise builds replace the workspace manifest with `Cargo.toml.openobserve`, which points these at the real crates. ⚠️ Must be merged together with openobserve/o2-enterprise#2162 (same branch name) — merging either PR alone breaks the enterprise build. ## Feature forwarding Root `enterprise` / `cloud` / `vectorscan` / `vortex` features forward to the corresponding `openobserve-core` features; the core crate forwards to `config` / `infra` / `o2_enterprise` exactly as the root crate did before. ## Unit-test fix The `service::sourcemaps` tests relied on the workspace root being the test cwd (shared zip fixture via a relative path, sqlite db dir/tables created by `init-db` from `coverage.sh`). Unit tests of a sub-crate run with the crate dir as cwd, so they now read the fixture via `CARGO_MANIFEST_DIR` and create the db dir + the tables they touch themselves; `src/core/data` is cleaned between the postgres and sqlite CI passes and gitignored, mirroring `src/infra/data`. ## Cleanup The extracted crate's manifest is pruned to what core actually uses (45 dependency entries copied from the root crate plus a dead `profiling` feature removed), and the stale `roaring` dependency (unused since the tantivy result cache rewrite in #7890) is dropped from the root manifest. All new files carry the standard AGPL header. ## Testing - `cargo check --workspace --all-targets`, `cargo clippy -D warnings`, and `cargo fmt --check` pass - `service::sourcemaps` unit tests pass under `--test-threads=1` (the CI setting; they share one db group and are not parallel-safe, same as before) - No logic changes anywhere: the diff is moves + visibility widening + clippy appeasement; error-to-HTTP-response mappings verified variant-by-variant - e2e label kept so Playwright runs --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Huaijin
|
9700fd6c43 |
perf: parallel zstd compress (#12960)
related to https://github.com/openobserve/o2-enterprise/issues/1860 - [x] limit parallel by the target partitions ## Summary This PR improves Flight `do_get` response throughput by executing and encoding output partitions in parallel instead of funneling everything through a single sequential stream/encoder path. ## Changes - Switches Flight `do_get` from `execute_stream` to `execute_stream_partitioned`. - Adds per-partition encoder tasks that coalesce RecordBatches, encode Arrow IPC, and apply ZSTD compression independently. - Splits Flight encoding into: - `ChunkCoalescer` for 8192-row batch coalescing. - `FlightDataEncoder` for IPC/ZSTD encoding of completed chunks. - Adds `PartitionEncoderStream` to handle per-partition batch coalescing and encoding. - Preserves custom Flight messages behavior: - early-emitted messages are sent before the first data message; - stats/metrics/peak-memory messages are sent after data completes. - Uses a fresh dictionary tracker per encoded chunk so dictionary-encoded columns remain decodable when chunks are encoded independently. - Cleans up stream lifecycle handling by aborting partition tasks before clearing session resources. ## Performance 100M records single node optimzier disable(to simulate the distribute query) ### Q1 ```sql SELECT trace_id, count(*) AS cnt, max(_timestamp) AS ts FROM "k8s_logs" GROUP BY trace_id ORDER BY cnt DESC LIMIT 100 ``` 12M trace_id unique value 12M records send from follower to leader main branch: 6s this branch: 3s ### Q2 ```sql SELECT count(distinct request_id) FROM "k8s_logs" ``` 100M request_id unique value 100M records send from follower to leader main branch: 30s this branch: 11s ### Q3 ```sql SELECT kubernetes_namespace_name, COUNT(DISTINCT trace_id) AS cnt FROM k8s_logs GROUP BY kubernetes_namespace_name ORDER BY cnt DESC ``` 10 kubernetes_namespace_name unique value, 12M trace_id unique value 63M records send from follower to leader main branch: 21s this branch: 8.3s ### Q4 ```sql SELECT host, COUNT(DISTINCT trace_id) AS cnt FROM k8s_logs GROUP BY host ORDER BY cnt DESC ``` 2160 host unique value, 12M trace_id unique value 99.57M records send from follower to leader main branch: 56s this branch: 20s |
||
|
Huaijin
|
ce726253f8 | fix: projection for hashjoin in broadcast join (#12875) | ||
|
Huaijin
|
d7b9cc495c | chore: use datafuson proto instead of copy (#12812) | ||
|
Huaijin
|
be9113c735 | fix: limit prune issue (#12758) | ||
|
Huaijin
|
5be5a302d3 |
perf: improve approx_distinct performance (#12714)
- close https://github.com/openobserve/o2-enterprise/issues/1862 ## todo - [x] fix access plan - [x] use forked datafusion |
||
|
Huaijin
|
0831badd02 |
perf: upgrade datafusion to 54 (#12665)
- [x] ~add sort when to many groups, we can do this, but for now do not do this~ - [x] remove the unsafe for memory pool - [x] add approx_distinct improve, already fork, will add in next pr - [x] fork vortex to add cast fix, and export `.exclude_schemes([IntDictScheme.id()])` - [x] ~enable_physical_uncorrelated_scalar_subquery test~ |
||
|
Hengfei Yang
|
890771e319 | perf: keep improve histogram on tantivy (#12693) | ||
|
Huaijin
|
f548a18de4 | refactor: access plan struct improve (#12646) | ||
|
Hengfei Yang
|
4c43f00036 | perf: keep improve histogram on tantivy (#12640) | ||
|
Hengfei Yang
|
d996349aa4 |
feat: rewrite tantivy histogram collectors, support histogram() timezone in optimize rules (#12627)
Design at: #12620 Closes #12620 Fixes #12564 ## What Rewrites both tantivy histogram optimize-rule collectors following the same approach as the recent `TopNCollector` / `SingleSegmentDocIdCollector` rewrites, and adds timezone support to both rules. - `MultiHistogramCollector` (replaces the aggregation framework for `SimpleMultiHistogram`) - `SimpleHistogramCollector` (replaces `tantivy::collector::HistogramCollector` for `SimpleHistogram`) ### Timezone support (#12564) `histogram(ts, interval, tz)` rewrites to `date_bin(interval, to_timestamp_micros(_timestamp + offset), origin)`; the optimizer's `get_column_name` hit the `BinaryExpr` and silently skipped both rules. Now: - the visitors extract the fixed offset from that shape (`get_timestamp_offset`), so the rules fire with a timezone - `IndexOptimizeMode::SimpleHistogram` / `SimpleMultiHistogram` carry a `ts_offset` field (no proto change — histogram modes never cross gRPC, they're re-derived per node) - bucket edges are computed in shifted wall-clock space and the collectors bucket on `_timestamp + ts_offset`, so emitted keys match `date_bin` exactly when merging with non-tantivy partials ## Testing - New collector unit tests: dense vs sparse equality, ts_offset shifting, bounds filtering, missing column / missing breakdown docs, per-bucket truncation with tie-breaks - Visitor tests now include real-DataFusion-plan timezone cases (`'+08:00'` simple, `'-05:30'` multi) asserting the extracted shifted bounds and offset - `cargo test -p config` (1936 passed), plus `index_optimizer` / `search::tantivy` / `tantivy_optimize_exec` suites in the main crate; clippy clean - Low cardinality: fast `20%` than main ``` histogram(_timestamp, '1h') AS ts, count(*) FROM default GROUP BY ts ORDER BY ts ASC ``` - Low cardinality: fast `30%` than main ``` histgoram(_timestamp, '1h') AS ts, breakdown, count(*) FROM default GROUP BY ts, breakdown ORDER BY ts ASC ``` - High cardinality: faster `5x` than main ``` histgoram(_timestamp, '1h') AS ts, breakdown, count(*) FROM default GROUP BY ts, breakdown ORDER BY ts ASC ``` |
||
|
Hengfei Yang
|
9c82beeeed |
fix: OTLP/JSON metrics silently dropping histograms and int values (#12615)
Close #12345 ## Problem OTLP histogram metrics sent over HTTP/JSON (e.g. otel-collector with `encoding: json`) created streams but stored 0 events, with no `_count`/`_sum`/`_bucket` streams. Int-valued counters/gauges were silently recorded as 0. The protobuf path was unaffected, which is why this only surfaced for collectors explicitly configured with JSON encoding. ## Root cause `otlp_json` deserializes the body directly into opentelemetry-proto's serde-derived types, which only partially implement the proto3 JSON mapping (64-bit integers encoded as strings, enums by name, zero/empty fields omitted). `Metric.data` is a flattened `Option` oneof, and serde converts any inner deserialization error of a flattened `Option` into `None` instead of failing — so the whole metric silently became `data: None`, while the stream itself had already been created by the schema setup. Same mechanism on `NumberDataPoint.value` turned string `asInt` values into 0. ## Changes - Upgrade opentelemetry stack 0.31 → 0.32, which fixes string-encoded u64 handling for plain histograms and summaries upstream. - Add `otlp_json_compat` normalizer for the gaps still present in 0.32 (each verified empirically, with unit tests): - string `asInt` values on data points and exemplars - `aggregationTemporality` as enum name - `Exemplar`'s value oneof is not flattened in 0.32 - missing `serde(default)` on `ExponentialHistogramDataPoint`/`Buckets`/`ValueAtQuantile`/`Exemplar`, which rejected omitted zero/empty fields - Count and log metrics whose data oneof failed to decode (`partial_success.rejected_data_points`) instead of silently skipping them, and stop overwriting specific partial-success error messages with the generic retention message. - Adapt to OTLP 0.32 API additions (`KeyValue.key_strindex`, `AnyValue::StringValueStrindex`). - Upgrade workspace reqwest 0.12 → 0.13, required because opentelemetry-http 0.32 only implements `HttpClient` for reqwest 0.13 (`rustls` feature replaces `rustls-tls-native-roots` via the platform verifier; `json`/`query`/`form` are now explicit features). The `segment` crate is pinned to reqwest 0.12, so `TELEMETRY_CLIENT` moved into `telemetry.rs` as a plain `reqwest::Client` that posts the Track message itself, replicating `segment::HttpClient::send`. - Remove the unused pyroscope profiling integration (optional feature, dependencies, agent setup, `ZO_PROF_PYROSCOPE_*` config). ## Verification - 10 unit tests in `otlp_json_compat` covering the full payload matrix (collector-style string integers, enum names, omitted zero fields, exemplars with and without trace context, already-normalized payloads, malformed input). - End-to-end: posted the issue's collector-style JSON payload to `/api/default/v1/metrics` — `_bucket` (3 events, correct cumulative counts with exemplars attached), `_count`, `_sum` populated for histogram and exponential histogram, summary quantiles stored, `asInt` counter stored as 42 instead of 0. - `cargo clippy -- -D warnings` clean; lib test failures are pre-existing local-environment issues that reproduce identically on main. |
||
 Harsh Mahajan
|
c260d9162d | fix: changing vendored ui-swagger (#12506) | ||
|
Hengfei Yang
|
3e865a3e6a | ci: update rust (#12394) | ||
 Don Kendall
|
75df625e06 | fix: clear Security audit (lettre CVE, rustls-pemfile migration, core2 ignore) (#11832) |