mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 16:09:05 +00:00
6ca0eec1075053d4e97b35dfbf8ededcdcd1791a
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Hengfei Yang
|
4231ca5833 |
perf(schema): share stream settings via Arc and centralize cache mutation (#13535)
Design at: #13465 PR 2 of the plan in #13465 (finding 1: the settings cache O(S) deep clone per mutation). ## Background The stream settings cache is a locked map (`STREAM_SETTINGS`) plus an ArcSwap read snapshot (`STREAM_SETTINGS_ATOMIC`) introduced in `405794db08` for lock-free ingestion reads. That architecture is kept — the read/write ratio justifies it. The problem was the constant factor: - every mutation republished the snapshot by **deep-cloning every cached `StreamSettings`** (vectors, UDS field lists, maps). The snapshot is republished on every schema change event cluster-wide (any new stream or field, on every node) plus every cache-miss backfill — at 100k+ streams this is a full deep copy of the map at minute-level frequency; - every point read (`get_settings`, called on each ingestion request) deep-cloned the returned `StreamSettings`, including UDS lists that can hold hundreds of strings; - the two maps each held a complete deep copy of all settings. ## Changes **Arc-ified values.** Both maps now store `Arc<StreamSettings>`. Snapshot republish is a shallow clone (String keys + refcount bumps), point reads return an `Arc` clone, and the maps share one set of values, roughly halving the cache's memory. **Centralized mutation.** The map statics are private; all writes go through `put_stream_settings` / `put_stream_settings_batch` / `remove_stream_settings`, which republish the snapshot under the write lock so snapshots cannot be published out of order. Startup load collects settings and publishes once via the batch API. Callers can no longer accidentally rebuild the snapshot — the pattern that caused the original quadratic startup. Lock-free reads stay available via `get_stream_settings_atomic`; the status API uses a new `get_stream_settings_stats()` accessor. **API ripple, kept minimal.** `get_settings()` now returns `Option<Arc<StreamSettings>>`. Field access and `unwrap_or_default()` call sites compile unchanged; the few paths that edit a local copy before persisting (`update_stream_settings`, LLM/gen-ai settings, dashboard distinct fields) do an explicit `(*s).clone()`. The `get_stream_setting_*` helpers are generic over `Borrow<StreamSettings>`, so both `&Option<StreamSettings>` and `&Option<Arc<StreamSettings>>` callers work — including the downstream enterprise call sites, which were checked and need no changes. **`SchemaCache::fields_map` Arc-ified** — cloning a cached schema entry no longer deep-copies the field-name index, which holds thousands of entries for wide streams. ## Measurements (criterion bench used during development, not kept in-tree) | operation | result | |---|---| | bulk load 10k / 50k / 100k streams | 0.80ms / 4.4ms / 8.7ms — linear, ~87ns per stream | | point write against 100k-entry cache | 3.5ms (shallow snapshot republish) | | point read against 100k-entry cache | 7.9ns (lock-free) | For comparison, the field report in #13465 measured ~27ms per stream insert at 58k cached streams under the old per-mutation deep clone; reads previously deep-cloned the settings value on every call. ## Testing - new unit test for the mutation API (put/batch/remove, snapshot sharing via `Arc::ptr_eq`, republish keeps other keys) - full test suites pass for all touched crates (infra, db, schema, stream, core, api-search, api-management, search_service, compaction, enrichment-data); the only failures are pre-existing local-environment ones also failing on main (sqlite-path retention tests, one env-dependent index test, one enterprise doctest) - `cargo clippy --workspace --all-targets`: no findings in touched files Finding 2 (grouped stream enumeration) landed in #13536; this branch has been rebased onto main (the #13531 overlap is resolved). |
||
 Huaijin
|
3d96c9670c |
chore: regenerate Rust CycloneDX SBOMs (#13534)
## Summary - regenerate Rust CycloneDX SBOMs with cargo-cyclonedx cyclonedx - update 9 existing SBOMs and add 32 workspace crate SBOMs - exclude all generated files under src/enterprise ## Validation - cargo-cyclonedx version: 0.5.9 - all 41 retained XML files pass xmllint --noout - malformed vcs_url PURL pattern from 0.5.7 is absent - git diff --check passes - src/enterprise has no changes |
||
|
Hengfei Yang
|
93b3fec5cf |
perf(cache): remove per-org full scans and hot-path shrink_to_fit calls (#13531)
Design at: #13465 PR 1 of the plan in #13465 (the mechanical fixes: findings 3, 4, 5). ## Changes **Metadata metrics job (finding 3)** — `update_metadata_metrics` counted users by scanning all of `ORG_USERS` once per organization, with a `format!` allocation inside the inner loop: O(orgs × org_users) iterations and allocations every 60 seconds on the compactor. Replaced with a single pass that aggregates counts per org (`count_org_users()`); per-org metric output is unchanged, including 0 for organizations with no users. Covered by a unit test. **Incremental stats (finding 4)** — `update_stats_from_file_list` checked every enumerated stream against `updated_streams: Vec<String>` with a linear `contains`, O(streams × updated) and effectively quadratic when most streams changed. The list is now collected into a `HashSet` once; the empty-set-means-update-all semantics are preserved. **Per-operation `shrink_to_fit` (finding 5)** — removed from: - the schema watcher delete path, which shrank four global maps (`STREAM_SCHEMAS`, `STREAM_SCHEMAS_LATEST`, `STREAM_RECORD_ID_GENERATOR`, `STREAM_SETTINGS`) after every single-stream deletion — bulk deletes rehashed a large table repeatedly while holding write locks; - the WAL search-lock release paths (`release_files`, `release_request`), which shrank `SEARCHING_FILES`/`SEARCHING_REQUESTS` on every request completion. `shrink_to_fit` rehashes the entire table but only reclaims the bucket array — the entry payloads are already freed by `remove()` — so these calls cost O(map size) under a write lock for a few MB upper-bound of savings. The shrink inside the rare full-clear `clean()` path is kept. **Test flakiness fix (pre-existing)** — the WAL locking tests mutate the shared global map and run in parallel; `clean_lock_files()` from one test could clear locks another test had just asserted on. Verified flaky on main (2 of 3 runs failed). The tests that assert on lock presence now serialize on a test-local mutex; 5 consecutive full runs pass. ## Testing - `cargo test -p common infra::wal` — 6 passed, stable across 5 runs - `cargo test -p compaction stats` — 10 passed - `cargo test -p openobserve-jobs job::metrics` — new `test_count_org_users` passes - `cargo clippy --all-targets` clean on the four touched crates Findings 1 (settings cache Arc-ification + mutation centralization) and 2 (grouped stream enumeration) follow in separate PRs per the plan. |
||
 udev
|
8ed0f95ea0 |
fix: resolve AI session schema and AnyValue handling (#13459)
## Summary - remove legacy `tool_name` and `tool_args` references from the session span SQL projection - use the canonical `gen_ai_tool_name` field for session tool calls - handle `gen_ai.input.messages` and `gen_ai.output.messages` as OTEL `AnyValue` payloads in backend session summaries and frontend turn rendering - support structured values, JSON-encoded span values, plain strings, scalar values, and unstructured object fallbacks without dropping message content - add regression coverage for the SQL schema and message normalization paths ## Testing - frontend unit tests: 145 passed - backend session extraction tests: 10 passed - frontend application type-check passed - targeted ESLint passed - Rust formatting and diff checks passed ## Reference - https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-events.md |
||
|
Huaijin
|
d165299bcd |
refactor: extract TLS and node services from core (#13449)
## Summary - extract TLS certificate loading and client configuration from openobserve-core into openobserve-tls - extract node discovery, cluster information, ingester gRPC channel selection, and Tokio runtime metrics into openobserve-node - update Core, API, API Management, Jobs, and the root binary to depend on the new crates directly - preserve one-way dependencies: openobserve-core depends on openobserve-node, while openobserve-node does not depend on Core - register both crates in the enterprise workspace overlay ## Validation - cargo test -p openobserve-node --lib - CARGO_INCREMENTAL=0 cargo clippy -p openobserve-node -p openobserve-core -p openobserve-jobs -p openobserve-api-management -p openobserve-api -p openobserve -- -D warnings - RUSTFLAGS="--cfg tokio_unstable" cargo check -p openobserve-node - enterprise overlay: CARGO_INCREMENTAL=0 cargo clippy -p openobserve --no-default-features --features enterprise -- -D warnings - cargo deny check licenses --hide-inclusion-graph - cargo fmt --all -- --check - git diff --check Companion enterprise PR: openobserve/o2-enterprise#2267 |
||
|
Huaijin
|
1624a6ff24 |
fix(schema): restore persistence guards on auto-UDS settings write (#13431)
## Summary #13405 rewired `handle_diff_schema`'s auto-UDS persistence from `stream::save_stream_settings` to a direct `db::schema::update_setting` call (required by the new crate dependency direction: `schema` cannot depend on `stream`). The direct write dropped the persistence guards that path provided. This PR restores them inside the schema crate, without reintroducing a dependency on the stream service: - skip persisting when the stream is being deleted (`db::compact::retention::is_deleting_stream`), matching the old `save_stream_settings` early-return - enforce `cfg.limit.user_defined_schema_max_fields` on the persisted field list (previously, with `ZO_SCHEMA_MAX_FIELDS_TO_ENABLE_UDS` set above the cap, settings that were never persisted before #13405 would now be written); skipped writes are logged at warn level instead of silently dropped - sort `defined_schema_fields` before persisting so the stored order is deterministic (was `HashSet` iteration order, nondeterministic across nodes/restarts) Also restores the `calculate_files_size` unit tests that were dropped when `src/core/src/file_list.rs` was deleted in #13405 — they now live next to the function in `infra::file_list`. ## Validation - `cargo check -p schema -p infra` - `cargo test -p schema --lib` (7 passed) - `cargo test -p infra --lib file_list::tests::test_calculate_files_size` (5 passed) - `cargo clippy -p schema -p infra` - `cargo fmt -p schema -p infra -- --check` |
||
|
Huaijin
|
15275ae7cf |
refactor: split core foundation services (#13405)
## Summary This PR begins decomposing `openobserve-core` into focused service crates while preserving the existing application behavior and dependency direction. It extracts six independently owned areas: - `ingestion-common`: shared ingestion contracts and response types - `schema`: schema inference, validation, evolution, and persistence helpers - `compaction`: merge, retention, and compaction services - `enrichment-data`: enrichment-table lifecycle and runtime support - `promql-service`: PromQL service orchestration - `stream`: stream metadata, settings, retention, and lifecycle operations ## What changed - registered the extracted crates in the workspace and propagated enterprise/cloud/vectorscan/vortex features - moved implementations out of Core and updated API, Jobs, query, and queue consumers to use the new crates - removed Core facade aliases for extracted crates and converted remaining callers to direct crate paths - retained the small cross-service stream-delete coordinator in Core for now because it still coordinates Pipeline and Alert cleanup - restored required root/Core dependencies after validating the full enterprise build - added the missing `anyhow` dependency for enterprise-only `search_service::search_jobs` - updated the lockfile for the new crate dependency graph ## Dependency direction The extracted crates do not depend on `openobserve-core`. Core remains a temporary upper-level consumer while the remaining domain services are split incrementally. ```text ingestion-common -> common/config schema -> ingestion-common/db/infra stream -> schema/enrichment-data/db/infra compaction -> schema/search/search-service/db/infra openobserve-core -> ingestion-common/schema/stream/compaction/... api/jobs -> openobserve-core + extracted services ``` ## Enterprise compatibility The private OpenObserve manifest was updated in openobserve/o2-enterprise#2257 to register the extracted workspace members and dependencies. Related Enterprise PR: https://github.com/openobserve/o2-enterprise/pull/2257 ## Validation OSS manifest: - `cargo fmt --all -- --check` - `cargo clippy -- -D warnings` Using `o2-enterprise/Cargo.toml.openobserve` as the OpenObserve root manifest: - `cargo check --no-default-features --features enterprise` - `cargo check --no-default-features --features cloud` - `cargo clippy --workspace --no-default-features --features enterprise -- -D warnings` - `cargo clippy --workspace --no-default-features --features cloud -- -D warnings` Enterprise repository: - `cargo fmt --all -- --check` - `cargo clippy -- -A clippy::question-mark -D warnings` All checks above pass in the development profile. |