mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-05 00:09:35 +00:00
74df120055e523fa00a3aef378b8a0e2142ff2bb
136
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Hengfei Yang
|
2d3dcf0dcc | fix(metrics): accept exemplars from Prometheus remote-write (#15282) | ||
|
Hengfei Yang
|
610cd70a13 | fix: generate and store the ext auth salt on a new install instead of refusing to start (#15191) | ||
  Ashish Kolheandgithub-actions[bot]
|
46678ea8da |
fix: tighten input validation, auth and outbound request checks (#15054)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
 Subhra264
|
60f92cfc2d | fix: install the cipher master key at startup and ack undecodable on-call queue messages (#15160) | ||
   
|
951ded3684 |
feat(iam): configurable password policy over api, enforced instance wide. (#14214)
Co-authored-by: omkark06 <omkar@zinclabs.io> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Bhargav <BJP232004@GMAIL.COM> Co-authored-by: Shrinath Rao <shnath@openobserve.ai> |
||
 Prabhat Sharma
|
9a77c6b878 | feat: Slack destination setup via OAuth and app manifest (#14020) | ||
|
Prabhat Sharmaandgithub-actions[bot]
|
0dba67d638 |
feat: composite alerts (#13797)
Design at : alerts/composite_alerts/composite_alerts.md ## Summary - add composite alert graph persistence, evaluation, scheduling, lifecycle APIs, and deletion safeguards - add composite alert UI forms, preview, references, list/detail integration, and localization - add Rust, API, Vue, and Playwright contracts ## Verification - cargo fmt --all -- --check - cargo check -p openobserve-core - cargo check -p openobserve-api-management - npm run type-check:composite-alerts - npm run type-check:app - npm run lint:ci -- --quiet - design/token/style checks - targeted Rust/Vitest suites - browser QA against local alerts UI ## Follow-ups (intentionally deferred, documented in the spec) - `RunOutcome::Error`/`NotifyFailed` outcome records on composite delivery failure (§11.5) - Incident detail live-definition resolver for composites + "definition unavailable" retention (§9.6) - `alert_composite_*` observability metrics (§16) - SLO/stream delete-conflict 409 bodies with readable-vs-hidden parent split (§10.3) - Super-cluster startup preflight (§18) <img width="1650" height="979" alt="image" src="https://github.com/user-attachments/assets/3def5246-7bf7-48fe-9a42-c0ac08271ea9" /> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
 Huaijin
|
015a20a184 |
refactor: remove HTTP API compatibility re-exports (#13560)
## Summary - remove the HTTP request/model compatibility re-exports introduced by the domain crate split - migrate router, OpenAPI, enterprise action-server, and integration-test callers to the crates that own each API - remove the ingest Content-Type re-export and declare the direct root/test dependencies - delete the pure `handler::http::models` facade module ## Validation - `cargo fmt --all -- --check` - `cargo clippy -p openobserve-api-ingest -p openobserve-api-http -- -D warnings` - `cargo test --test integration_test --no-run` - `cargo check -p openobserve --no-default-features` - enterprise overlay: `cargo test --offline --workspace --no-default-features --features enterprise --no-run` - cloud overlay: `cargo test --offline --workspace --no-default-features --features cloud --no-run` - `git diff --check` |
||
|
Hengfei Yang
|
4231ca5833 |
perf(schema): share stream settings via Arc and centralize cache mutation (#13535)
Design at: #13465 PR 2 of the plan in #13465 (finding 1: the settings cache O(S) deep clone per mutation). ## Background The stream settings cache is a locked map (`STREAM_SETTINGS`) plus an ArcSwap read snapshot (`STREAM_SETTINGS_ATOMIC`) introduced in `405794db08` for lock-free ingestion reads. That architecture is kept — the read/write ratio justifies it. The problem was the constant factor: - every mutation republished the snapshot by **deep-cloning every cached `StreamSettings`** (vectors, UDS field lists, maps). The snapshot is republished on every schema change event cluster-wide (any new stream or field, on every node) plus every cache-miss backfill — at 100k+ streams this is a full deep copy of the map at minute-level frequency; - every point read (`get_settings`, called on each ingestion request) deep-cloned the returned `StreamSettings`, including UDS lists that can hold hundreds of strings; - the two maps each held a complete deep copy of all settings. ## Changes **Arc-ified values.** Both maps now store `Arc<StreamSettings>`. Snapshot republish is a shallow clone (String keys + refcount bumps), point reads return an `Arc` clone, and the maps share one set of values, roughly halving the cache's memory. **Centralized mutation.** The map statics are private; all writes go through `put_stream_settings` / `put_stream_settings_batch` / `remove_stream_settings`, which republish the snapshot under the write lock so snapshots cannot be published out of order. Startup load collects settings and publishes once via the batch API. Callers can no longer accidentally rebuild the snapshot — the pattern that caused the original quadratic startup. Lock-free reads stay available via `get_stream_settings_atomic`; the status API uses a new `get_stream_settings_stats()` accessor. **API ripple, kept minimal.** `get_settings()` now returns `Option<Arc<StreamSettings>>`. Field access and `unwrap_or_default()` call sites compile unchanged; the few paths that edit a local copy before persisting (`update_stream_settings`, LLM/gen-ai settings, dashboard distinct fields) do an explicit `(*s).clone()`. The `get_stream_setting_*` helpers are generic over `Borrow<StreamSettings>`, so both `&Option<StreamSettings>` and `&Option<Arc<StreamSettings>>` callers work — including the downstream enterprise call sites, which were checked and need no changes. **`SchemaCache::fields_map` Arc-ified** — cloning a cached schema entry no longer deep-copies the field-name index, which holds thousands of entries for wide streams. ## Measurements (criterion bench used during development, not kept in-tree) | operation | result | |---|---| | bulk load 10k / 50k / 100k streams | 0.80ms / 4.4ms / 8.7ms — linear, ~87ns per stream | | point write against 100k-entry cache | 3.5ms (shallow snapshot republish) | | point read against 100k-entry cache | 7.9ns (lock-free) | For comparison, the field report in #13465 measured ~27ms per stream insert at 58k cached streams under the old per-mutation deep clone; reads previously deep-cloned the settings value on every call. ## Testing - new unit test for the mutation API (put/batch/remove, snapshot sharing via `Arc::ptr_eq`, republish keeps other keys) - full test suites pass for all touched crates (infra, db, schema, stream, core, api-search, api-management, search_service, compaction, enrichment-data); the only failures are pre-existing local-environment ones also failing on main (sqlite-path retention tests, one env-dependent index test, one enterprise doctest) - `cargo clippy --workspace --all-targets`: no findings in touched files Finding 2 (grouped stream enumeration) landed in #13536; this branch has been rebased onto main (the #13531 overlap is resolved). |
||
|
Huaijin
|
ef3617f157 |
refactor: split HTTP APIs by domain (#13479)
## Summary
- organize API crates under
`src/api/{common,grpc,http,ingest,management,pipelines,search}`
- keep ingest, management, pipelines, and search independent from one
another; compose them only in the HTTP frontend
- move pattern APIs into search and simplify `main.rs` wiring
- update workspace manifests, lockfile, integration tests, and
cargo-deny entries
- keep `web/` unchanged
## Validation
- `cargo fmt --all -- --check`
- `cargo clippy --all-targets -- -D warnings`
- `cargo deny check licenses`
- enterprise overlay: `cargo build --no-default-features --features
enterprise`
- enterprise overlay: `cargo build --no-default-features --features
cloud`
- enterprise overlay: `cargo clippy --all-targets -- -D warnings`
The matching `o2-enterprise` overlay PR will be linked after creation.
|
||
|
Huaijin
|
3f9ce27542 |
refactor: split service dependencies out of core (#13446)
## Summary - decouple stream cleanup from enrichment-data and remove the transitive compaction/DataFusion dependency from stream - extract sourcemaps and organization storage services from core into dedicated crates - extract GitHub built-in regex access and model pricing into openobserve-builtins - rename pipeline and organization-storage persistence modules from store to db - remove now-unused direct dependencies from core - synchronize the enterprise workspace overlay in openobserve/o2-enterprise#2266 ## Validation OSS: - cargo check -p openobserve-builtins -p openobserve-core -p openobserve-jobs -p openobserve-api - cargo test -p openobserve-builtins --lib - cargo test -p openobserve-builtins --doc - cargo clippy -p openobserve-builtins -- -D warnings - cargo check -p openobserve-org-storage --features enterprise Enterprise overlay: - copied ../o2-enterprise/Cargo.toml.openobserve over Cargo.toml - cargo fmt --all -- --check - cargo check --all-targets --no-default-features --features enterprise - cargo clippy --all-targets --no-default-features --features enterprise -- -D warnings The default enterprise + vectorscan build was attempted and reached vectorscan-rs-sys, but the validation host does not have cmake installed. The enterprise-only build and clippy run completed successfully. |
||
|
Huaijin
|
15275ae7cf |
refactor: split core foundation services (#13405)
## Summary This PR begins decomposing `openobserve-core` into focused service crates while preserving the existing application behavior and dependency direction. It extracts six independently owned areas: - `ingestion-common`: shared ingestion contracts and response types - `schema`: schema inference, validation, evolution, and persistence helpers - `compaction`: merge, retention, and compaction services - `enrichment-data`: enrichment-table lifecycle and runtime support - `promql-service`: PromQL service orchestration - `stream`: stream metadata, settings, retention, and lifecycle operations ## What changed - registered the extracted crates in the workspace and propagated enterprise/cloud/vectorscan/vortex features - moved implementations out of Core and updated API, Jobs, query, and queue consumers to use the new crates - removed Core facade aliases for extracted crates and converted remaining callers to direct crate paths - retained the small cross-service stream-delete coordinator in Core for now because it still coordinates Pipeline and Alert cleanup - restored required root/Core dependencies after validating the full enterprise build - added the missing `anyhow` dependency for enterprise-only `search_service::search_jobs` - updated the lockfile for the new crate dependency graph ## Dependency direction The extracted crates do not depend on `openobserve-core`. Core remains a temporary upper-level consumer while the remaining domain services are split incrementally. ```text ingestion-common -> common/config schema -> ingestion-common/db/infra stream -> schema/enrichment-data/db/infra compaction -> schema/search/search-service/db/infra openobserve-core -> ingestion-common/schema/stream/compaction/... api/jobs -> openobserve-core + extracted services ``` ## Enterprise compatibility The private OpenObserve manifest was updated in openobserve/o2-enterprise#2257 to register the extracted workspace members and dependencies. Related Enterprise PR: https://github.com/openobserve/o2-enterprise/pull/2257 ## Validation OSS manifest: - `cargo fmt --all -- --check` - `cargo clippy -- -D warnings` Using `o2-enterprise/Cargo.toml.openobserve` as the OpenObserve root manifest: - `cargo check --no-default-features --features enterprise` - `cargo check --no-default-features --features cloud` - `cargo clippy --workspace --no-default-features --features enterprise -- -D warnings` - `cargo clippy --workspace --no-default-features --features cloud -- -D warnings` Enterprise repository: - `cargo fmt --all -- --check` - `cargo clippy -- -A clippy::question-mark -D warnings` All checks above pass in the development profile. |
||
|
Huaijin
|
6adcae3a1f |
refactor: remove crate compatibility re-exports (#13381)
## Summary
- remove compatibility re-exports introduced during the recent crate
refactors
- migrate callers to the crates that own each API
- delete five pure facade modules and add the required direct crate
dependencies
- keep OSS and enterprise manifests aligned
## Validation
- cargo fmt --all -- --check
- cargo clippy --workspace -- -D warnings
- cargo test --locked --workspace --no-run
- enterprise overlay: cargo test --offline --workspace
--no-default-features --features enterprise --no-run
- enterprise overlay: cargo clippy --offline --workspace
--no-default-features --features enterprise -- -D warnings
- git diff --check
Companion enterprise PR uses the same branch name:
codex/remove-refactor-reexports.
## TODO / Follow-ups
- [ ] **Re-establish the pipeline write-guardrail lost with
`db::pipeline`.** That module was not a pure facade: it re-exported only
the six read-only accessors from `infra::pipeline` and its doc comment
recorded the invariant that *mutations must go through
`service::pipeline::store`* (which keeps the `ExecutablePipeline` caches
and the coordinator/super-cluster watch events in sync). Deleting it
points callers straight at `infra::pipeline`, which also exposes
`put`/`delete`/`drop_table` with no guardrail. Decide the layering and
enforce it:
- **Reads** may hit `infra` directly (raw persistence, no side effects).
- **Writes** must go through the wrapper that persists *and* emits the
coordinator/super-cluster event + refreshes the in-memory cache. That
wrapper is `db::<entity>::{set,delete}` for most entities (e.g.
`db::alerts::alert::set` →
`infra::coordinator::alerts::emit_put_event`), but for pipeline it is
`core::pipeline::store` (not `db`), because the
`STREAM_EXECUTABLE_PIPELINES` cache it maintains is a core-level
construct the `db` crate cannot build. The super-cluster receiver
(`super_cluster_queue::pipelines`) is the one sanctioned caller of
`infra::pipeline::{put,delete}` directly, and it manually pairs each
with `emit_*_event` to avoid a re-broadcast loop.
- Net invariant to protect: never call `infra::pipeline::{put,delete}`
without emitting the coordinator event — i.e. nowhere outside
`core::pipeline::store` and the super-cluster handler. A bare `infra`
write leaves in-memory caches stale until restart (see the warning
comment still live at `src/core/src/org_cleanup.rs:484`).
- [ ] Consider whether the pipeline write-wrapper living in
`core::pipeline::store` while every other entity's lives in
`db::<entity>` is worth unifying, or at least documenting as an
intentional exception.
|
||
|
Huaijin
|
e236c7e653 |
refactor: extract database layer into db crate (#13340)
## Summary - extract persistence, cache, watcher data access, and DB service modules from openobserve-core into a standalone workspace db crate - keep application orchestration that depends on search, organization provisioning, and other Core services behind the existing Core db facade - preserve existing call paths through re-exports while making the dependency direction openobserve-core -> db - move HTTP error mapping back to the Core/API boundary so db has no direct axum dependency - avoid provider/provisioner callback traits and function-pointer hooks ## Validation - cargo fmt --all - cargo check --all-targets - cargo check -p db --all-targets - Enterprise workspace: cargo check --all-targets --offline - Enterprise workspace: cargo check -p db --all-targets --features enterprise --offline Companion Enterprise manifest PR: openobserve/o2-enterprise#2230. |
||
|
Huaijin
|
3b44da3eee |
refactor: extract shared common crate (#13252)
## Summary - introduce a `common` workspace crate for shared metadata, infrastructure helpers, and utilities - keep service-backed code in `openobserve-core` while preserving the existing `common` compatibility namespace - flatten `src/core/src/service/` into the standard `src/core/src/` module layout without using `#[path]` - retain `service::*` as compatibility re-exports from `service.rs` - move cipher request/response types from core into the search crate while preserving `openobserve_core::cipher` - remove unused Cargo dependencies from workspace crates Enterprise companion: https://github.com/openobserve/o2-enterprise/pull/2202 ## Validation - `cargo check --workspace --all-targets` - `cargo clippy -p common -p openobserve-core -p search --lib -- -D warnings` - common crate unit tests: 271 passed, 4 ignored - enterprise overlay: `cargo check -p openobserve-core --features cloud` - migrated cipher tests: 8 passed - `cargo deny check licenses` - `cargo fmt --all -- --check` - `git diff --check` |
||
|
Huaijin
|
66ca3d2439 |
refactor: extract PromQL evaluator into dedicated crate (#13239)
## Summary - add a dedicated promql workspace crate for parsing and evaluation - keep cluster routing, cache, WAL, file-list, and request orchestration in openobserve-core - preserve existing service::promql call sites through a compatibility facade; the facade re-exports an explicit item list (not a glob), and the promql search orchestration in core depends on the promql crate directly - inject enterprise query cancellation through TableProvider instead of coupling the evaluator to core search state - replace the core regex-UDF dependency with DataFusion regexp_like expressions ## Companion PR - openobserve/o2-enterprise#2198 updates the enterprise workspace override for the new promql crate ## Validation - cargo fmt --all -- --check - cargo check --workspace - cargo clippy -p promql --all-targets -- -D warnings - cargo clippy -p openobserve-core - cargo test -p promql: 363 passed - verified the promql dependency tree has no openobserve-core or search dependency - cargo check -p o2_enterprise --lib in the paired private workspace ## Notes The evaluator moves atomically in this PR. The existing service::promql::search subtree remains in core because it owns OpenObserve-specific orchestration rather than PromQL evaluation. The two web/ file changes only update source-path references in comments that pointed at the moved Rust files (src/service/promql/... -> src/promql/src/...). |
||
  
|
0aab1ae261 |
feat: online evaluations (#12352)
- **feat(online-eval): add PipelineKind::Evaluation and dual realtime pipeline support** - **feat(online-eval): add provider, score_config, scorer, and online_eval_jobs DB entities** - **feat(online-eval): add config CRUD services and API handlers** - **feat(online-eval): reconcile jobs into hidden evaluation pipelines** - **feat(online-eval): add scorer preparation and execution backends** - **feat(online-eval): write scores and evaluator traces to system streams** - **refactor(online-eval): remove legacy eval templates backend** - **test(online-eval): add API integration coverage for eval resources** --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
 yujinqiu
|
833b2bdc8e | feat(security): enforce strong password policy (#12327) | ||
 Loakesh IndiranandHengfei Yang
|
379087b550 |
chore: add sync unit tests (#11588)
## Summary - Add `#[cfg(test)] mod tests` blocks to ~525 files across all workspace crates - Covers: openobserve, infra, config, ingester, wal, flight, tantivy_utils, cli - All tests are synchronous (`#[test]` only) — struct defaults, serde roundtrips, enum variants, pure utility functions - No async tests, no logic changes — purely additive --------- Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
|
85d261b5d0 |
fix: vulnerability scan fixes (#11249)
| # | Severity | Category | File | | :--- | :--- | :--- | :--- | | 1 | **High** | SSRF bypass (IPv6) | `ssrf_guard.rs` | | 2 | **High** | SSRF bypass (retry) | `enrichment_table/mod.rs` | | 3 | **High** | SSRF defense-in-depth | `alert.rs` | | 4 | **Critical** | CORS + credentials | `router/mod.rs` | | 5 | **Critical** | SQL Injection | `postgres.rs`, `sqlite.rs` | | 6 | **High** | Privilege escalation (token) | `users.rs` | | 7 | **Medium** | Panic/DoS | `users.rs` | | 8 | **Low** | Integer overflow | `alerts/mod.rs` | | 9 | **Medium** | XSS | `alertSummaryGenerator.ts`, `colorizeQuery.ts` | | 10 | **Medium** | Credential exposure in logs | `validator.rs`, `organization.rs`, `users.rs` | --------- Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Shrinath Rao <shnath@openobserve.ai> Co-authored-by: Prabhat Sharma <hi.prabhat@gmail.com> |
||
 ByteBakerandHengfei Yang
|
b948361769 |
chore: update copyright year to 2026 (#11166)
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
 
|
e419449ddb |
feat: implement prebuilt alert destination (Backend) (#10074)
Design at : observability/alert-prebuilt-destinations/alert-prebuilt-destinations-plan.md ## 🤖 Auto-Implementation (Backend) **Feature:** implement prebuilt alert destination **Source Issue:** openobserve#10055 ## Changes - ✅ Backend code (Rust) - ✅ Unit + integration tests - ✅ All quality gates passed ## Related PRs Frontend PR: #FRONTEND_PR_NUMBER (to be linked) Closes openobserve#10055 --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> --------- Co-authored-by: claude-code-bot <claude-code-bot@openobserve.ai> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
Hengfei Yang
|
1fc4270f54 |
fix: payload limit (#10021)
1. We only need `ZO_PAYLOAD_LIMIT` now, removed `ZO_JSON_LIMIT` 2. Config axum log level `RUST_LOG = "info,config::axum=warn"` |
||
|
Hengfei Yang
|
33eecaefe3 | refactor: migrate actix-web to axum (#9912) | ||
|
Ashish KolheandClaude Opus 4.5
|
ce09cd0a4d |
feat: add js for claim parser (#9873)
### **User description** Design at : security/sso-claim-parser/configuration.md ___ ### **PR Type** Enhancement, Tests ___ ### **Description** - Introduce rquickjs-based JS runtime for functions - Add compile_js_function and apply_js_fn utilities - Extend pipeline to support JS via CompiledFunctionRuntime - Update web UI: select JS/VRL transforms, show JS errors ___ ### Diagram Walkthrough ```mermaid flowchart LR A["FunctionNode"] -->|JS| B["compile_js_function"] A -->|VRL| C["compile_vrl_function"] B --> D["apply_js_fn"] C --> E["apply_vrl_fn"] D --> F["send to children"] E --> F ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>13 files</summary><table> <tr> <td><strong>function.ts</strong><dd><code>Add optional `trans_type` to TestFunctionPayload</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-c6766f6c671adce05f9eef5d4a57a32371fa7da35628ef5c142fa503d191cb16">+16/-0</a> </td> </tr> <tr> <td><strong>js.rs</strong><dd><code>Introduce JS runtime, compile and apply functions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-88e490bed5fa6c9c8e0aa2738b95c5284bb97899298f54b9cd587713e3ef0109">+756/-0</a> </td> </tr> <tr> <td><strong>batch_execution.rs</strong><dd><code>Extend pipeline execution to handle JS functions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-3f6824a6eddaf1603e5cb0f1a4311ac6a45e1ffb967266ccc1f0bba944625e69">+465/-96</a></td> </tr> <tr> <td><strong>functions.rs</strong><dd><code>Validate and test-run JS transforms, restrict to _meta</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-ed5611a9aa6b5c033e70d7631ba72fafad3cc842672077a7fcf965d11a4fef23">+319/-35</a></td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Forbid JavaScript functions in pipelines globally</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-c186c796a7bdfd9d1301b1429df11456430f1d0d4758783810b9e2e44384e3b2">+39/-1</a> </td> </tr> <tr> <td><strong>jwt.rs</strong><dd><code>Add JS claim parser execution function</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-232cab74f4cbf135af424744272e17732fbda8ac005d33c0acb4597e65479d02">+25/-2</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Wrap JS compile/apply in ingestion API</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-7c7b5bca85f0ced652965cae66540c96f56dfcca4f4c16b77bf7b45aac203d1b">+24/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Include `trans_type` in test_function API payload</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-a5e1b3644cef9feb689dfe66382e44e5434ecadbff3469127346a333d4733102">+8/-3</a> </td> </tr> <tr> <td><strong>AddFunction.vue</strong><dd><code>UI: add JS/VRL transform type selector</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-ebfd6c538f3cd3ecb9ab17550fd76e2e89e7a1abccc83a7505077e4af3eee110">+42/-25</a> </td> </tr> <tr> <td><strong>FunctionsToolbar.vue</strong><dd><code>Toolbar: transform type radio buttons</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-8283d2ff9c4a66b4b59eac306672aca2d54655a462e2ddb88e14e039d79dcdaf">+51/-3</a> </td> </tr> <tr> <td><strong>DomainManagement.vue</strong><dd><code>Track claim-parser function changes for save button</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-5e47cd01fcb63f7ef27dc64d2ecf66c39e17097e29944951c3ae1bfe715bb97c">+17/-9</a> </td> </tr> <tr> <td><strong>TestFunction.vue</strong><dd><code>Support JS error messaging in test UI</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-b9af9a2d1bf04d23422c26a13e6a763c6953c8713357a2f64ed8073ce6c926a3">+15/-5</a> </td> </tr> <tr> <td><strong>PipelineEditor.vue</strong><dd><code>Filter out JS functions in pipeline editor UI</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-68c21f647336067eb79737ce9aca2d49632f2fa1c0878bcfdae55607e3473a15">+12/-4</a> </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>function.rs</strong><dd><code>Extend `Transform` model for JS support</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-27c9f3d93723397a0e234bc45c9f543b43eff5fd518d76e735f78d95a58f3f1c">+16/-6</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Register new `js` utils module</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-2aa59ce4464e7dbabdb9367fc8bb8c1b489c914b8690577d4916870463ab12db">+1/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Bug fix</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>errors.rs</strong><dd><code>Rename `MemoryTableOverflowError` display message</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-bf45932466265b732f13d0cd9b9cffbaefbe37c5861ace9315449eeca9250713">+1/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>3 files</summary><table> <tr> <td><strong>result-array-functions.spec.js</strong><dd><code>Add E2E tests for `#ResultArray#` functions in UI</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-e9911625ad2bab76f2b760f0ad254791331993a6a874b96f6a11fc6d88038b10">+470/-0</a> </td> </tr> <tr> <td><strong>cleanup.spec.js</strong><dd><code>Clean up JS function test data in UI tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-c214275a512fa285a7e4f2ab61dfcb42d71a0436b0e4241f94d7b45693587983">+31/-0</a> </td> </tr> <tr> <td><strong>page-manager.js</strong><dd><code>Register `FunctionsPage` in `PageManager`</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-565eab8b3c767ae50c9d05e591621d14f66b81af84d6b5ea6efe136c1c3933df">+6/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Dependencies</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>Cargo.toml</strong><dd><code>Add `rquickjs` dependency for JS runtime</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542">+2/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>18 files</summary><table> <tr> <td><strong>test_functions_meta_org_restriction.py</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-99ef1207d6ae6d7eeaed37d70c8cf0c4770ba78f6f51fbdc067d35492d2fe9ea">+401/-0</a> </td> </tr> <tr> <td><strong>test_js_enrichment.py</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-c5547d46bec60b888d947d1e9b7c5445aee72cf8e837182f7138e0fc903eedbb">+356/-0</a> </td> </tr> <tr> <td><strong>test_js_result_array.py</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-9ed1363a1f58c94e018bb2b0aaa9d08fa71f89e7e55a6ff6be4a82bbeed8985c">+390/-0</a> </td> </tr> <tr> <td><strong>functionsPage.js</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-7bab9cc2eb8f36d3ee5d0aab453c353ec6d6dfecbc401cf401322ff57faa8440">+326/-0</a> </td> </tr> <tr> <td><strong>javascript-functions.spec.js</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-bdae6529346a7222bdec19f75f490cb9d0f3578a260ba835225efcd27db3f64e">+314/-0</a> </td> </tr> <tr> <td><strong>meta-org-js-restriction.spec.js</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-16cc2373f6f1955f8344a59d5f2edcd6ce121c4fdee1d432b1f0ec8e27f25881">+483/-0</a> </td> </tr> <tr> <td><strong>FunctionList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-6ff9af50a1597504b7f0dca91e61cf30c800b4a17b4be2ed2c3a4f7a7b2d7068">+1/-0</a> </td> </tr> <tr> <td><strong>de.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-78f24ecb2dbcfc38fdf3ec22b2559a5b40d236ea689d1c55f5d3e87168310264">+30/-18</a> </td> </tr> <tr> <td><strong>en.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+15/-7</a> </td> </tr> <tr> <td><strong>es.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-ac2a2167b8b973e3f776f9fe3b43bef4576470d8ad33f3d7cfaff6edff0e5778">+30/-18</a> </td> </tr> <tr> <td><strong>fr.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-db80e762b88ddf183820c050cdc6b215e6b02b365b5e73f5152af46b0d26f1a2">+30/-18</a> </td> </tr> <tr> <td><strong>it.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-2de78074a5f887afddce60d694ead9322e6ab223eb70251c13819f3028bdc70d">+30/-18</a> </td> </tr> <tr> <td><strong>ja.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-092e49c5f9fbc1b539ab9c2866d108d0ce1d314022b4c294eb4611b285e7bbd3">+30/-18</a> </td> </tr> <tr> <td><strong>ko.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-50e27ca1b4825a5026f3a251bfcf79c97cab07cfade2ebd512d332453d707c48">+30/-18</a> </td> </tr> <tr> <td><strong>nl.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-a1706021687ae5d8a1c006c49eab3946174fecd895f9b70a89c2e76d97250906">+30/-18</a> </td> </tr> <tr> <td><strong>pt.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-38e804f64cb5e0e451849fc84e9b46e53c954a4a4263d51c18665ad45f4b4815">+30/-18</a> </td> </tr> <tr> <td><strong>tr.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-6f0f131e9e2591480c4c5d1eaffc72d02dc31cbd8d3f90d8f012f08cabac8818">+30/-18</a> </td> </tr> <tr> <td><strong>zh.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-8818055db58f83230aeea4ebfd4cec31d924e8c47b6f91298e356959ab25339e">+254/-242</a></td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
Ashish Kolhe
|
0f4dcbe971 | fix: 9882 add option to limit max series return for promql (#9896) | ||
|
Subhra264
|
509bf7b6fb |
feat: pipeline backfill (#9462)
Design at : pipeline-gap-filler
# feat: Add Pipeline Backfill Jobs Feature
This PR introduces a complete pipeline backfill system that allows users
to reprocess historical data through existing pipelines.
## Key Features
### Backend
- **Dual-table architecture**: Separate `backfill_jobs` table for static
config and `scheduled_jobs` for scheduling state
- **RESTful API endpoints** under
`/api/{org_id}/pipelines/{pipeline_id}/backfill`:
- Create, list, pause, resume, update, and delete backfill jobs
- **Robust error handling**: Retry mechanism matching derived stream
pattern (immediate retries → scheduled retries, never delete jobs)
- **Progress tracking**: Chunk-based processing with real-time position
updates
- **Optional deletion phase**: Delete existing data before backfilling
with separate status tracking
- **Super cluster support**: Proper trigger synchronization across
regions
### Frontend
- **Backfill Jobs List Page**: Table view with status/pipeline filters,
action buttons (pause/resume/edit/delete)
- **Create Dialog**: Time range picker (absolute dates only),
configurable chunk periods and delays
- **Edit Dialog**: Update paused/completed jobs with reset to fresh
state
- **Job Details Dialog**: Real-time progress, timeline visualization,
deletion status tracking
- **Dark mode support**: Full theme compatibility across all components
- **DateTime integration**: Reuses existing DateTime component with
microsecond precision
## Technical Highlights
- Cross-database support (PostgreSQL, MySQL, SQLite)
- Frontend-only filtering for performance
- Status mapping (waiting/pending → "running" display)
- Timestamp formatting: `YYYY-MM-DDTHH:mm:ssZ`
- Default chunk period matches pipeline schedule frequency
- Navigation integration from PipelinesList component
## Files Changed
### Backend
- **Migration**:
`src/infra/src/table/migration/m20250109_000001_create_backfill_jobs.rs`
- **Table CRUD**: `src/infra/src/table/backfill_jobs.rs`
- **Service logic**: `src/service/alerts/backfill.rs`
- **Scheduler handlers**: `src/service/alerts/scheduler/handlers.rs`
- **API routes**: `src/handler/http/request/pipelines/backfill.rs`
- **Router integration**: `src/handler/http/router/mod.rs`,
`src/handler/http/router/openapi.rs`
- **Super cluster sync**: `src/super_cluster_queue/scheduler.rs`
### Frontend
- **Components**:
- `web/src/components/pipelines/BackfillJobsList.vue`
- `web/src/components/pipelines/CreateBackfillJobDialog.vue`
- `web/src/components/pipelines/EditBackfillJobDialog.vue`
- `web/src/components/pipelines/BackfillJobDetails.vue`
- **API service**: `web/src/services/backfill.ts`
- **DateTime enhancement**: `web/src/components/DateTime.vue`
## Testing
- Tested across PostgreSQL, MySQL, and SQLite databases
- Verified dark mode compatibility
- Tested pause/resume/edit/delete operations
- Validated progress tracking and completion status
- Confirmed super cluster synchronization
|
||
|
Subhra264
|
659a5439d1 |
fix: Make cron handler DST aware (#8952)
# Add DST-Aware Timezone Handling for Alert Scheduling ## Summary Alerts previously used static timezone offsets that became outdated during DST transitions, causing alerts to fire at incorrect times (e.g., an alert configured for "9 AM Pacific" would fire at 10 AM after DST change). ## Solution - Added `chrono-tz` dependency for IANA timezone database support - Implemented DST-aware offset calculation at scheduling time using timezone strings (e.g., `America/Los_Angeles`) - Updated `get_next_trigger_time_non_aligned()` and `align_time()` to dynamically calculate offsets based on the scheduled timestamp - Maintains backward compatibility: alerts without timezone strings continue using fixed offsets |
||
|
ByteBaker
|
77ab0cff5c |
fix: resolve flaky integration tests
- Ensure memtable isolation to prevent cross-test data contamination - Fix dashboard API version compatibility (v5 -> version-agnostic dashboard_id) - Fix indentation bug causing function/dashboard/stream deletions to run inside alert loop - Add 3-second wait after dashboard deletion for propagation |
||
|
Subhra264
|
57902884f2 |
feat: use primary region if specified when caching the enrichment tables on node start (#8878)
### **User description**
In a multi-region super cluster setup, when a node starts up and caches
enrichment tables, it generally queries all regions for enrichment table
data. However, if user is sure that enrichment table is stored in one
primary region, then querying all regions is inefficient and might even
fail if other region queriers return error for some reason. Which can
eventually cause node start failure.
### Solution
This commit adds a new `apply_primary_region_if_specified` boolean
parameter throughout the enrichment table data fetching flow to control
whether to use only the primary region when fetching data.
### Key Changes
1. **New Parameter Added** (`enrichment_table.rs:40`, `mod.rs:190`):
- Added `apply_primary_region_if_specified` boolean parameter to:
- `get_enrichment_table_data()`
- `get_enrichment_table()`
- `get_enrichment_table_json()`
2. **Primary Region Logic** (`enrichment_table.rs:52-68`):
- When `apply_primary_region_if_specified` is true and enterprise
features are enabled:
- Checks if super cluster is enabled
- Reads the `enrichment_table_get_region` configuration
- If a primary region is specified, uses only that region for the search
request
- Otherwise, uses empty regions array (queries all regions)
3. **Cache Initialization** (`schema.rs:588-591`):
- When caching enrichment tables during node startup
(`cache_enrichment_tables()`), passes `true` for the parameter
- This ensures node startup only queries the primary region for
enrichment data
4. **Runtime Watch Operations** (`enrichment_table.rs:310-328`):
- Watch operations pass `false` for the parameter
- This maintains existing behavior for runtime updates
5. **Code Cleanup**:
- Removed unused `get()` function (lines 105-146 deleted)
- Simplified logging by removing redundant empty/non-empty checks
- Added better debug logging showing number of records fetched
### Impact
- **Performance**: Node startup is faster as it only queries the
designated primary region for enrichment table data
- **Correctness**: Ensures enrichment tables are fetched from the
authoritative source (primary region) during initialization
- **Flexibility**: Runtime operations can still query all regions if
needed
### Configuration Required
This feature requires enterprise edition with super cluster enabled and
the `enrichment_table_get_region` configuration set to specify which
region should be considered the primary source for enrichment table
data.
___
### **PR Type**
Enhancement
___
### **Description**
- Add primary-region-aware enrichment fetch
- Thread control flag across fetch APIs
- Use primary region on node startup
- Preserve all-regions for watch updates
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["schema::cache_enrichment_tables()"] -- "get_enrichment_table(..., true)" --> B["enrichment::get_enrichment_table_inner(..., true)"]
B -- "get_enrichment_table_data(..., true)" --> C["db::enrichment_table::search_inner(regions=primary)"]
D["db::watch()"] -- "get_enrichment_table(..., false)" --> B
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>enrichment_table.rs</strong><dd><code>Region-aware
enrichment data retrieval and watch updates</code> </dd></summary>
<hr>
src/service/db/enrichment_table.rs
<ul><li>Add <code>_apply_primary_region_if_specified</code> to
<code>get_enrichment_table_data</code><br> <li> Determine
<code>regions</code> based on enterprise config and flag<br> <li> Call
<code>search_inner</code> with computed regions and enable consistency
flag<br> <li> Update watch flow to call enrichment getter with
<code>false</code></ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8878/files#diff-6ce27ddf8a0d8b4e790bda90d088ff692a5da31a8d330beea8a9dd843294572b">+35/-6</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>schema.rs</strong><dd><code>Startup caching uses
primary region when specified</code>
</dd></summary>
<hr>
src/service/db/schema.rs
<ul><li>Cache startup uses <code>get_enrichment_table(...,
true)</code><br> <li> Add comment clarifying primary-region-only
behavior</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8878/files#diff-4696a44abde2df4ac48fed716f930bfdcc01759bdfcfd8d802821c6dd857bff9">+4/-1</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>mod.rs</strong><dd><code>Plumb primary-region flag
through enrichment APIs</code>
</dd></summary>
<hr>
src/service/enrichment/mod.rs
<ul><li>Plumb <code>apply_primary_region_if_specified</code> through
public APIs<br> <li> Document flag semantics and usage caveats<br> <li>
Pass flag to DB layer when fetching remote data</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8878/files#diff-0ea1823eb8a3a9845cde8bb2264ccdb749fad5be6d9acda8b17d76b2dbfdcc9f">+16/-3</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
ByteBaker
|
50e0b64aa9 |
feat: add MCP protocol endpoints
Integrate Model Context Protocol server endpoints at `/api/{org_id}/mcp`
for both enterprise and non-enterprise builds.
Changes:
- Add POST and GET handlers for MCP requests with streaming support
- Extract and pass through Authorization header for API authentication
- Add non-enterprise stub endpoints that return "not available" error
- Update router to register MCP endpoints conditionally
Endpoints:
- POST `/api/{org_id}/mcp`: Single MCP request/response
- GET `/api/{org_id}/mcp`: Streaming MCP requests
|
||
|
Huaijin
|
456f4286fa | perf: store enrichment table in disk as parquet format (#8686) | ||
|
ByteBaker
|
c868605736 | chore: add tests, fmt, clippy | ||
|
ByteBaker
|
0cf0923a59 | fix: flaky integration test | ||
|
ByteBaker
|
934aaadbfc | fix: flaky integration tests | ||
|
ByteBaker
|
075168ab8a | chore: clippy delint | ||
|
Subhra264
|
bcdace5459 |
fix: Root user should not be added to any organization (#8632)
### **User description**
Root user by default can access all the organizations and does not need
to be added to any organizations through api.
___
### **PR Type**
Bug fix, Tests
___
### **Description**
- Prevent adding root user to orgs
- Return Forbidden for unauthorized add attempts
- Expand integration tests for user/org flows
- Validate user listing includes expected emails
___
### Diagram Walkthrough
```mermaid
flowchart LR
addUser["add_user_to_org handler"] -- "user is root" --> rejectRoot["400 Bad Request: root no org needed"]
addUser -- "not root & unauthorized" --> forbid["403 Forbidden"]
tests["Integration tests"] -- "root add attempt" --> expect400["expect failure"]
tests -- "create non-root & add to org" --> expect200["expect success"]
tests -- "list users" --> assertUsers["assert expected emails present"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Bug
fix</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>users.rs</strong><dd><code>Block root user org add; use
403 for denial</code>
</dd></summary>
<hr>
src/service/users.rs
<ul><li>Short-circuit if <code>existing_user.is_root</code>, return
400.<br> <li> Change unauthorized response from 401 to 403.<br> <li> Add
explanatory comments for root access.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8632/files#diff-98da2588a28d62bb2e301d989836125f89f89b76272cd7f75e08ec4deeea5f38">+13/-2</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>integration_test.rs</strong><dd><code>Extend
integration tests for user/org behaviors</code>
</dd></summary>
<hr>
tests/integration_test.rs
<ul><li>Import <code>UserList</code> and validate list users
content.<br> <li> Add <code>e2e_add_root_user_to_org</code> expecting
failure.<br> <li> Add <code>e2e_add_user_to_org</code> to create and add
admin user.<br> <li> Update test sequence to include new cases.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8632/files#diff-7ff815b68943f08bef79f083c09e55ca8c18db8e25033034b12fa31f1d43dab4">+79/-2</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
Subhra264
|
1077ad0461 |
fix: enrichment table delete event must be watched (#8578)
### **PR Type**
Bug fix, Tests
___
### **Description**
- Fix delete to watch enrichment deletions
- Add GeoIP config and parsing tests
- Add multipart CSV extraction tests
- Add derived stream timerange edge-case tests
- Add enrichment table integration tests
___
### Diagram Walkthrough
```mermaid
flowchart LR
DeleteFn["DB delete for enrichment tables"] -- "enable prev_kv for watch" --> Watcher["Cluster watcher reacts to deletions"]
GeoIP["GeoIP module"] -- "unit tests" --> Confidence["Parsing/config defaults validated"]
Multipart["Multipart CSV extraction"] -- "unit tests" --> CSVRecords["Robust CSV handling verified"]
Pipeline["Derived stream scheduler"] -- "integration tests" --> TimeRange["Invalid timerange scenarios covered"]
Enrichment["Enrichment tables"] -- "integration tests" --> CacheSchema["Schema, caches, disk cache validated"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Bug
fix</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>enrichment_table.rs</strong><dd><code>Enable prev_kv on
enrichment table delete</code>
</dd></summary>
<hr>
src/service/db/enrichment_table.rs
<ul><li>Change delete call flag to enable previous value<br> <li>
Ensures delete events are observable via watch</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8578/files#diff-6ce27ddf8a0d8b4e790bda90d088ff692a5da31a8d330beea8a9dd843294572b">+1/-1</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>geoip.rs</strong><dd><code>Add GeoIP config and parsing
tests</code>
</dd></summary>
<hr>
src/service/enrichment_table/geoip.rs
<ul><li>Add comprehensive unit tests for DatabaseKind parsing<br> <li>
Test defaults, config clone/eq, serialization<br> <li> Validate error on
invalid DB path</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8578/files#diff-e9b58b850f67eed76a5e8afb0bf98dbbc6e09216558dcdc4441663f58998f800">+116/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>mod.rs</strong><dd><code>Add multipart CSV extraction
test suite</code>
</dd></summary>
<hr>
src/service/enrichment_table/mod.rs
<ul><li>Add extensive tests for multipart CSV extraction<br> <li> Cover
headers-only, malformed, whitespace, specials<br> <li> Test multiple
files and append behavior</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8578/files#diff-3e2b6d3b8da1f9e73aca279052a97e683184d68256c23215b18ffee885836bbf">+381/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>integration_test.rs</strong><dd><code>Add pipeline and
enrichment integration tests</code>
</dd></summary>
<hr>
tests/integration_test.rs
<ul><li>Add helpers and integration tests<br> <li> Cover derived stream
invalid timeranges (delay/cron)<br> <li> Add enrichment table end-to-end
tests and cleanup<br> <li> Refactor pipeline retrieval helper and minor
logs</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8578/files#diff-7ff815b68943f08bef79f083c09e55ca8c18db8e25033034b12fa31f1d43dab4">+830/-18</a></td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
Huaijin
|
4a182220d2 | chore: upgrade datafusion to v50.0.0 (#8526) | ||
|
Hengfei Yang
|
4f49af792f | fix: shutting down DDL connection (#8398) | ||
|
Hengfei Yang
|
1989432ba7 |
feat: remove syslog (#8227)
### **User description** close #8186 ___ ### **PR Type** Enhancement, Documentation ___ ### **Description** - Remove syslog feature across codebase - Clean config and routing references - Update metadata prefixes and SBOM files - Refresh i18n and OpenAPI assets ___ ### Diagram Walkthrough ```mermaid flowchart LR SYS["Syslog feature"] -- "deprecated & removed" --> CODE["Code paths"] CODE -- "delete flags/routes" --> CFG["Configuration"] CODE -- "remove storage keys" --> META["Metadata prefixes"] CODE -- "update specs" --> SBOM["SBOM/XML assets"] CODE -- "update docs/i18n" --> WEB["Web locales & OpenAPI"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>13 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Remove syslog config import from job module</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+3/-26</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Stop glob importing config; drop syslog usage</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-ca09c9c9b020049bdbc3382c466bfcca80daac92e98760d9a9a76172abf34049">+1/-3</a> </td> </tr> <tr> <td><strong>meta.rs</strong><dd><code>Drop `/syslog` from metadata item prefixes</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-1797798cd07803457a6273ef793d56568d8c91e743f6c4649c6db733325fb0af">+1/-2</a> </td> </tr> <tr> <td><strong>config.rs</strong><dd><code>Remove syslog-related configuration options</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+0/-34</a> </td> </tr> <tr> <td><strong>config.rs</strong><dd><code>Purge syslog feature flag and defaults</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-cda0d9179ed982fa5f0989c9b26f21cfa49dd1de28195a98135650ffcc7c602a">+0/-25</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Clean references tied to syslog ingestion</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-5061e662a8899b7e00e5744bd72f808b28ac479a8921ac519f7409bffdc4438a">+0/-60</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Delete syslog router endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+0/-5</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Detach syslog handler wiring</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-99620cff6123acf0598ed2ec9d1c06f02cb0ddece1bef898047eb303d0b71fc2">+0/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Remove syslog-specific log ingestion paths</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-ce72aea3fa4c44904cbd9ec07bd844bec71b49c9676d667b14056aa5798f96b2">+0/-2</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Clean DB interactions for syslog keys</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-b7e3bf7f0ac23dd084c0c817a310a4d5e7a64ff532746d182986ef940fa03fd0">+0/-2</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Adjust request handling without syslog</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-a1cfd40da1ab6cd4db0181fd8664965034298047b3b7c3320c5385897a7c4aed">+0/-2</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Unexport syslog services</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-1b15316505c31acccb26ba66d920fdb0894567d08d2bf1e3457716d128a67b83">+0/-2</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Remove syslog metadata definitions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-fbf797574839e69491ea0429141c038c0ff96297bdf3d8c60d60cdc4acec74b1">+0/-2</a> </td> </tr> </table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>integration_test.rs</strong><dd><code>Update tests to exclude syslog paths</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-7ff815b68943f08bef79f083c09e55ca8c18db8e25033034b12fa31f1d43dab4">+0/-57</a> </td> </tr> <tr> <td><strong>useIngestionRoutes.spec.ts</strong><dd><code>Update UI tests to remove syslog routes</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-3b7ebacf9481f4643a2b56ffe53ab161fc89d416e280d45b48568866a4dee40c">+0/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>20 files</summary><table> <tr> <td><strong>openapi.rs</strong><dd><code>Remove syslog routes from OpenAPI</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-edb8ce232c0cd08366879f642788a75d99f6bab64b4c0e77391281fdb9241e90">+0/-5</a> </td> </tr> <tr> <td><strong>config.cdx.xml</strong><dd><code>Update config component index (remove syslog)</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-41ee11320f37e20fc96f754d35a890df5068af5072fd5388169d1cf305a1ed3b">+15011/-4611</a></td> </tr> <tr> <td><strong>wal.cdx.xml</strong><dd><code>Update WAL component index references</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-3d72e8b949a91b0bd78e97aa3ebf69e2887e2851223ea3d10fedeb0125c35a19">+19476/-358</a></td> </tr> <tr> <td><strong>openobserve.cdx.xml</strong><dd><code>Refresh SBOM root; remove syslog entries</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-454f38f03cf69f6b5358f5a528be4214ec2d33081b9e98ebe5d92c8bef47dc48">+14015/-7932</a></td> </tr> <tr> <td><strong>infra.cdx.xml</strong><dd><code>Update infra component index, no syslog</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-e51bf18198bc65a604b53ee18152ad9f2453ab7870d5be949022058e054ada73">+14127/-6086</a></td> </tr> <tr> <td><strong>proto.cdx.xml</strong><dd><code>Update proto index; purge syslog mentions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-a15fa99b53917ea85b4e268bb7612d43ee881dbe4ecf70a6682bb919b1f2f219">+8000/-1513</a></td> </tr> <tr> <td><strong>ingester.cdx.xml</strong><dd><code>Adjust ingester index excluding syslog</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-e72a9be4ba2aaff1a6b837a7f7149489603f8fe720766757c4291082e98b5a65">+15845/-4615</a></td> </tr> <tr> <td><strong>flight.cdx.xml</strong><dd><code>Add/refresh flight component index</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-f41393a0191b0560da893a5e24b2e4529f7c9bb34fc71231a532b8790630cba2">+19900/-0</a></td> </tr> <tr> <td><strong>hi.json</strong><dd><code>Remove syslog strings from Hindi locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-1e7b52b5c4e762db2c923d0d3efb6d693702825ed3a1a112d3069f26b0d5f605">+0/-13</a> </td> </tr> <tr> <td><strong>fr.json</strong><dd><code>Remove syslog strings from French locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-db80e762b88ddf183820c050cdc6b215e6b02b365b5e73f5152af46b0d26f1a2">+0/-13</a> </td> </tr> <tr> <td><strong>tr.json</strong><dd><code>Remove syslog strings from Turkish locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-6f0f131e9e2591480c4c5d1eaffc72d02dc31cbd8d3f90d8f012f08cabac8818">+0/-13</a> </td> </tr> <tr> <td><strong>de.json</strong><dd><code>Remove syslog strings from German locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-78f24ecb2dbcfc38fdf3ec22b2559a5b40d236ea689d1c55f5d3e87168310264">+0/-13</a> </td> </tr> <tr> <td><strong>ko.json</strong><dd><code>Remove syslog strings from Korean locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-50e27ca1b4825a5026f3a251bfcf79c97cab07cfade2ebd512d332453d707c48">+0/-13</a> </td> </tr> <tr> <td><strong>it.json</strong><dd><code>Remove syslog strings from Italian locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-2de78074a5f887afddce60d694ead9322e6ab223eb70251c13819f3028bdc70d">+0/-13</a> </td> </tr> <tr> <td><strong>nl.json</strong><dd><code>Remove syslog strings from Dutch locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-a1706021687ae5d8a1c006c49eab3946174fecd895f9b70a89c2e76d97250906">+0/-13</a> </td> </tr> <tr> <td><strong>es.json</strong><dd><code>Remove syslog strings from Spanish locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-ac2a2167b8b973e3f776f9fe3b43bef4576470d8ad33f3d7cfaff6edff0e5778">+0/-13</a> </td> </tr> <tr> <td><strong>pt.json</strong><dd><code>Remove syslog strings from Portuguese locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-38e804f64cb5e0e451849fc84e9b46e53c954a4a4263d51c18665ad45f4b4815">+0/-13</a> </td> </tr> <tr> <td><strong>zh.json</strong><dd><code>Remove syslog strings from Chinese locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-8818055db58f83230aeea4ebfd4cec31d924e8c47b6f91298e356959ab25339e">+0/-13</a> </td> </tr> <tr> <td><strong>ja.json</strong><dd><code>Remove syslog strings from Japanese locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-092e49c5f9fbc1b539ab9c2866d108d0ce1d314022b4c294eb4611b285e7bbd3">+0/-13</a> </td> </tr> <tr> <td><strong>en.json</strong><dd><code>Remove syslog strings from English locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+0/-13</a> </td> </tr> </table></details></td></tr><tr><td><strong>Dependencies</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>Cargo.toml</strong><dd><code>Update dependencies/features after syslog removal</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542">+0/-3</a> </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>10 files</summary><table> <tr> <td><strong>syslog.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-dc6813e31e32c23aae5760091460d2ec4023d37ae73e49d29052d9f4e9f2e1fa">+0/-43</a> </td> </tr> <tr> <td><strong>websocket.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-48f05dcd4b6f1d8345cb5707e1a3c34eedd76d94a3c3904eeabd799c7820789d">+0/-90</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-58010c6e78f69dc57c353890a807371cd43c2b4cceeeb1e9378130f26d898397">+0/-143</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-66207bd699343342402f88cdd331f4c549783754e4921d0c8d43e026e2d8e575">+0/-135</a> </td> </tr> <tr> <td><strong>syslog_server.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-cb7610142582b44885f673583ab16262c3b9ca6de1f59a9def65e503360665a0">+0/-137</a> </td> </tr> <tr> <td><strong>report_server.cdx.xml</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-d7a2fd1b635919119af9187ef44188c384bd01e444e74a9d15eaa21b0840b7ed">+20472/-0</a></td> </tr> <tr> <td><strong>syslog.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-4c0e9282e3d2352f9a382099901fe034bae2961eed01d764f16ad52f27f55db4">+0/-180</a> </td> </tr> <tr> <td><strong>syslog.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-4bc46f36fad79f5df8c42b8a8d5d8c4a70b1320b966bcff3421deeb96a1981fa">+0/-559</a> </td> </tr> <tr> <td><strong>syslogs_route.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-325df35568c993fc0727bfb0499b682e6bd30af781687706306b3bdb429c49e2">+0/-191</a> </td> </tr> <tr> <td><strong>syslog.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-69ec0272ddac65f2cc96c79d84ca3933fc335a8d21ab5dd6909d71ea02a46486">+0/-37</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ |
||
|
sktrpathi
|
75fc81cecf |
chore: remove bogus unit tests (#8172)
### **User description** A significant chunk of API unit tests were not testing any actual business logic, but performing trivial actions such as struct-field-access, derive trait validation such as `Eq`, `PartialEq`, `Clone`, `Serialize` etc, and many of the tests were also just comparing constant values. They added no value to the test suite and have now been removed from the codebase, making the tests more focused. Additionally, - A lot of flaky tests have been corrected. These used to fail due to occasional race conditions when ran locally. - Some mock setup and teardown functions have also been cleaned up, which makes tests a lot more consistent. ___ ### **PR Type** Tests, Enhancement ___ ### **Description** - Remove trivial, non-value unit tests - Relax flaky assertions and panics handling - Improve test isolation via DB/cache resets - Minor test output formatting improvements ___ ### Diagram Walkthrough ```mermaid flowchart LR tests["Remove trivial/boilerplate tests"] -- cleaner suite --> stability["Stabilize assertions and panics"] stability -- reduce flakiness --> isolation["DB/Cache resets in tests"] isolation -- consistent runs --> outputs["Improve test log formatting"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><details><summary>11 files</summary><table> <tr> <td><strong>organization.rs</strong><dd><code>Remove struct-field and default-value trivial tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-4f089715a7ac4f78b45039d773f2fccbb4fbc6a409e0ab9ceee1fcf7911bc44b">+4/-325</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Drop JSON roundtrip serialization tests; tidy imports</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-c4bd542e4e07ed7706b0643086f5a7d4edbb3647c7ceedbe4bea516c5274d8bd">+1/-183</a> </td> </tr> <tr> <td><strong>auth.rs</strong><dd><code>Remove Debug/PartialEq trivial tests; format strings</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+2/-104</a> </td> </tr> <tr> <td><strong>main.rs</strong><dd><code>Make tracing test tolerant; ignore enterprise tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-42cb6807ad74b3e201c5a7ca98b911c5fa08380e942be6e4ac5807f8377f87fc">+23/-60</a> </td> </tr> <tr> <td><strong>event.rs</strong><dd><code>Remove constants and simple struct creation tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-689769712cbd6bd02beba078c1e1dfc188e2be39ab19a0e232b8bd12bc7f1e78">+1/-95</a> </td> </tr> <tr> <td><strong>validator.rs</strong><dd><code>Ensure test isolation with ORM init and cache clears</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-f4d4b51574b378a8c372345ea48875834a4eadacfac33fa4d0ceca1e0ef96b35">+28/-22</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Make base URI tests robust to env errors</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-d7987e62dcc7b1ec1bd4aeabef94f85d8c41b2ea33de5ab78660794fa4d51a9e">+25/-4</a> </td> </tr> <tr> <td><strong>users.rs</strong><dd><code>Use best-effort cleanup; clear caches; improve logs</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-98da2588a28d62bb2e301d989836125f89f89b76272cd7f75e08ec4deeea5f38">+12/-9</a> </td> </tr> <tr> <td><strong>integration_test.rs</strong><dd><code>Compact println formatting and panic messages</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-7ff815b68943f08bef79f083c09e55ca8c18db8e25033034b12fa31f1d43dab4">+3/-3</a> </td> </tr> <tr> <td><strong>config.rs</strong><dd><code>Remove strict zero-length assertions on globals</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-cda0d9179ed982fa5f0989c9b26f21cfa49dd1de28195a98135650ffcc7c602a">+1/-13</a> </td> </tr> <tr> <td><strong>schema.rs</strong><dd><code>Remove unused constant import in tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-a5ea87a7766cb24619f64a6100f385f4b42c47af7d33b8e2f060229c536d7418">+1/-8</a> </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>26 files</summary><table> <tr> <td><strong>authz.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-28a815d1c4728aff854a9996de0ba3d3dafe1bea6bfb0cf2a9f4186c40f985b6">+0/-52</a> </td> </tr> <tr> <td><strong>loki.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-121c166af5eab8b9cb83716afa7b65fa7856c006c809b3d910ced184c52ec587">+0/-7</a> </td> </tr> <tr> <td><strong>proxy.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-7f12a4f2483e737671ac41f9c35b411e7323e0ff2731ad92a3b309e144d1316a">+0/-46</a> </td> </tr> <tr> <td><strong>service.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-61d456e7217d19bcec3920a4d1c17fd396a257c81e5394655db3632a057bdd9e">+0/-7</a> </td> </tr> <tr> <td><strong>service_account.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-878ad520e1c84500cc21ff6fc4088c8da64ff2febd79e60c0d4cc8c695c49798">+0/-80</a> </td> </tr> <tr> <td><strong>traces.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-bda8182c1f17a8360337c769edade0cb9478ff1443dd6aedce7f526afbe97afe">+0/-106</a> </td> </tr> <tr> <td><strong>user.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-e13b3a981fd70182b9738a6b0ecfdee915c5b36757703a87741e1eddf3f6e786">+0/-101</a> </td> </tr> <tr> <td><strong>auth_tests.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-2688cbdc0b009b3f0dd829c69ab86c6a7d0778ab3e65e916a410a76aa6ae6480">+0/-2961</a></td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-2aa59ce4464e7dbabdb9367fc8bb8c1b489c914b8690577d4916870463ab12db">+0/-1</a> </td> </tr> <tr> <td><strong>action.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-fc87c75a7df9539d7e7ee6184dfab02d4a8932323e81f95d57c05363aa44752d">+0/-18</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-5bf716db7e1d91dfbd427a2e320ce407c4f0c92d55d68f8e636daed1b02abd56">+0/-13</a> </td> </tr> <tr> <td><strong>reports.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-d2117a3d67d4ec47b0017861845c92143baf917e10e6746df5dd97f925f81eeb">+0/-10</a> </td> </tr> <tr> <td><strong>destinations.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-e0ba33d95ef7f0eac13bdac7e750462907a0308f01c027ea206d058e282c977b">+0/-103</a> </td> </tr> <tr> <td><strong>inverted_index.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-8bf9d9403f05f53f0c6fd5e6a6dbf0204ea80465b233567552cb7395d08d1c3c">+0/-22</a> </td> </tr> <tr> <td><strong>meta_store.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-45018c838da377fa9ded9305a1ee752adde06acdb607addf26c95f5747537615">+0/-6</a> </td> </tr> <tr> <td><strong>ratelimit.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-fa7dc03fb526c09d5eb58021872d821fcd72eeb4d6a670224f5286012485bc3d">+0/-10</a> </td> </tr> <tr> <td><strong>timed_annotations.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-f94b4d6d12d68e7054ad33601e666b08e4594586013dc34384fa21b1df5fe6e8">+0/-12</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-ca09c9c9b020049bdbc3382c466bfcca80daac92e98760d9a9a76172abf34049">+0/-34</a> </td> </tr> <tr> <td><strong>m20250611_000002_populate_reports_table.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-8247af6d6f8ccb6a5a51daff4cb1e352585be2d1c785d86ae5ebfea1dbc1099c">+0/-1</a> </td> </tr> <tr> <td><strong>folders.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-402482c22fda5dff8f33382ac11d0908f693975dec7c08b37b1c5fe24a20486a">+0/-1</a> </td> </tr> <tr> <td><strong>bulk.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-2017dbe9076c78a7e7919ed5c8738729123d630038803cef0ce2c356d6fa2c48">+0/-9</a> </td> </tr> <tr> <td><strong>ingestion.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-3edd5f4ed1eed0f765b7b450ef0fbf587632e6b22302006a5e397049e297057f">+0/-7</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-2f7f0bac01afc6e37908521d1d33d05fcd650b7a08792062976e1e7465b4a1f9">+0/-42</a> </td> </tr> <tr> <td><strong>footer_cache.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-f9278e53ed7131c953fc30f8b15cda801b74b81fb2d570fe7f80a3b7fc1de872">+0/-7</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-1602f85399080c1e2838d1ef1d62af23d96f353379e49a7770026e892c954018">+0/-20</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-66f3b4f5e7927092fd6e57738687a1091b6f316638f7c58f8d7be9addb72bc30">+0/-31</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ |
||
|
udev
|
b4eac8f607 |
fix: Inc UT coverage (#8128)
increase UT coverage improve the email regex to catch `doubledotsequence` in email |
||
|
sktrpathi
|
9a11a680ae |
chore: add test cases (#8095)
### **User description** - also clear over 400 clippy issues ___ ### **PR Type** Tests, Enhancement ___ ### **Description** Add extensive unit tests for enrichment storage Expose helpers for testability in storage module Modernize tests: clippy-friendly string formatting Normalize constants in time-based tests |
||
|
Subhra264
|
04f54a54cd |
test: Add tests for derived streams (#8084)
### **PR Type**
Tests, Enhancement
___
### **Description**
- Add derived stream e2e tests
- Create/delete pipelines via API helpers
- Validate trigger lifecycle and retries
- Cover invalid SQL evaluation behavior
___
### Diagram Walkthrough
```mermaid
flowchart LR
Create["Create test pipeline"] -- "POST /api/{org}/pipelines" --> List["List pipelines"]
List -- "GET /api/{org}/pipelines" --> TriggerOK["Trigger success path"]
List --> TriggerMaxRetries["Trigger max retries path"]
InvalidPipe["Create invalid pipeline"] -- "POST" --> DeleteField["Delete field from stream"]
DeleteField -- "PUT /streams/.../delete_fields" --> TriggerFail["Trigger evaluation failure"]
Cleanup["Cleanup pipelines"] -- "DB delete" --> End["Done"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>integration_test.rs</strong><dd><code>Add derived
stream pipeline E2E tests and helpers</code>
</dd></summary>
<hr>
tests/integration_test.rs
<ul><li>Import pipeline, derived stream, trigger types<br> <li> Add
helper to POST pipelines via API<br> <li> Add e2e tests for derived
stream success, not found, max retries<br> <li> Add e2e test for
evaluation failure with invalid SQL and cleanup</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8084/files#diff-7ff815b68943f08bef79f083c09e55ca8c18db8e25033034b12fa31f1d43dab4">+480/-1</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
  Yashodhan JoshiandYashodhan Joshi
|
65eb741f5c |
fix: cherry pick hec changes (#7595)
### **User description** cherry-picks some extra changes done in https://github.com/openobserve/openobserve/pull/7555 from branch rc7 to main ___ ### **PR Type** Enhancement, Tests ___ ### **Description** - Add default HEC stream configuration option - Set default stream to "_hec" when unspecified - Ingest handler uses default for missing index - Update integration test for missing index success ___ ### **Changes diagram** ```mermaid flowchart LR A["EnvConfig: ZO_INGEST_DEFAULT_HEC_STREAM"] B["Set default_hec_stream to '_hec' if empty"] C["HEC ingest uses default when index missing"] D["Integration test asserts success"] A --> B B --> C C --> D ``` ___ ### **Changes walkthrough** 📝 <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Configuration changes</strong></td><td><table> <tr> <td> <details> <summary><strong>config.rs</strong><dd><code>Add default HEC stream config logic</code> </dd></summary> <hr> src/config/src/config.rs <li>Add <code>ZO_INGEST_DEFAULT_HEC_STREAM</code> env_config field in <code>Common</code><br> <li> Set <code>default_hec_stream</code> to "_hec" when empty in <code>check_common_config</code> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/7595/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+6/-0</a> </td> </tr> </table></td></tr><tr><td><strong>Error handling</strong></td><td><table> <tr> <td> <details> <summary><strong>hec.rs</strong><dd><code>Handle missing HEC index with default</code> </dd></summary> <hr> src/service/logs/hec.rs <li>Import <code>get_config</code> to access runtime config<br> <li> Retrieve <code>default_hec_stream</code> in <code>ingest</code> handler<br> <li> Use default when <code>HecEntry.index</code> is missing<br> <li> Log error if default is empty and return invalid index </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/7595/files#diff-5434f454898e66d4a982801761a349ac7e6862836e15eefaf643b31d0d2933dd">+16/-3</a> </td> </tr> </table></td></tr><tr><td><strong>Tests</strong></td><td><table> <tr> <td> <details> <summary><strong>integration_test.rs</strong><dd><code>Update HEC test for missing index</code> </dd></summary> <hr> tests/integration_test.rs <li>Update test comment for missing index case<br> <li> Change assertion to expect success status </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/7595/files#diff-7ff815b68943f08bef79f083c09e55ca8c18db8e25033034b12fa31f1d43dab4">+2/-2</a> </td> </tr> </table></td></tr></tr></tbody></table> ___ > <details> <summary> Need help?</summary><li>Type <code>/help how to ...</code> in the comments thread for any questions about PR-Agent usage.</li><li>Check out the <a href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a> for more information.</li></details> --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> |
||
|
ByteBaker
|
25ce21cc3e | chore: clippy delints (#7446) | ||
|
ByteBaker
|
3aad96721d | chore: maintenance and cleanup (#7289) | ||
|
Yashodhan JoshiandYashodhan Joshi
|
54a3fda236 |
feat: support hec format for ingestion (#7043)
closes #6996 This adds support for HEC data format in ingestion via `/org_id/_hec` api path. See the issue for more details. --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> |
||
+2    
|
0d9f47966b |
feat: Organization revamp (#5579)
Fixes #5131 Addresses #5408 ### Revamp - Created a new table to store organizations in the `organizations` table. Earlier, we never stored organizations in any table, rather we used to store the organizations in the meta table as part of the `user` json. - The users and organization membership relations are now moved to its own separate tables - `users` and `org_users`. ### New changes **API changes** - - `/organizations` - `POST` api to create an org and save it in the meta table. In non-enterprise version, currently only the root user can create the org. For enterprise version with rbac enabled, anyone with `CREATE` access for organizations can create the organization and becomes an admin. Root user can always create an organization. - `/{org_id}/rename` - `PUT` api to rename the existing org. Only root user and the admins of the org can access this API. The payload for following APIs are changed - - `POST api/{org_id}/roles` - `{"name":"new_role"}` is changed to `{"role":"new_role"} - `POST api/{org_id}/users/{user}` - `{"role":"viewer","custom_role":["role_name"]}` is the new payload where `custom_role` is optional and is array of custom roles (only works in enterprise with rbac enabled) which will be assigned to the user. The value of `role` can be standard role or as well as any custom role (when it is custom role, the standard role assigned to the user is decided based on the enterprise env variable - `O2_DEX_DEFAULT_ROLE`). - `POST api/{org_id}/users`- the payload - ``` {"email":"user@user.com","role":"viewer","custom_role":["create_org_permission"],"password":"12345678", "first_name": "", "last_name":""} ``` - `PUT api//{org_id}/users/{email_id}` the payload - ``` {"first_name":"Hello","last_name":"j","role":"viewer","isExternal":false,"change_password":false,"password":"","custom_role":["create_org_permission"]} ``` A new ENV variable is introduced - `ZO_CREATE_ORG_THROUGH_INGESTION` (default `false`) which if true, an org will automatically created when ingesting data to a new org. If false, user can not create org by ingesting data to new org, they need to first call the `POST /organizations` API. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes **New Features** - Added a `cloud` feature for enhanced organizational management capabilities. - Introduced new methods for managing organizational users and asynchronous message processing. - Enhanced user invitation and membership features, including improved role handling. - Implemented configurable ingestion URL support across various components. - Added new functionality for user management and organization handling in the super cluster architecture. **Improvements** - Refined user authentication and authorization processes, including error handling. - Updated database schema to support multi-organization architecture. - Enhanced pagination settings and UI components for better user experience. - Improved handling of organization names and user roles in notifications and alerts. - Streamlined service account token management with visibility enhancements. - Enhanced the `AddUpdateOrganization` component with improved user input handling and validation. - Updated the `AppOrganizations` component to dynamically load the appropriate organization management interface. - Improved the `ServiceAccountsList` component with updated pagination options and token handling logic. **Bug Fixes** - Resolved issues with user role assignments and organization management inconsistencies. - Corrected routing and navigation logic for login and logout processes. - Addressed various error handling scenarios in user and organization processes. - Fixed issues with the visibility of service account options in the `EditGroup` and `EditRole` components based on the cloud configuration. **Breaking Changes** - Significant restructuring of user and organization data models. - Changes to authentication workflows and configuration handling. - Removal of certain components related to user and organization management. - Updated the logic for determining the ingestion URL across various components. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Bhargav <BJP232004@GMAIL.COM> Co-authored-by: Taiming Liu <liutaiming3@gmail.com> Co-authored-by: hengfei yang <hengfei.yang@gmail.com> Co-authored-by: harshiv-7 <harshiv@openobserve.ai> Co-authored-by: Huaijin <haohuaijin@gmail.com> Co-authored-by: Zhao.wei <weizhao029@gmail.com> Co-authored-by: ShyamOOAI <shyam@openobserve.ai> Co-authored-by: oasisk <ashish.j.kolhe@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> Co-authored-by: Yashodhan Joshi <yashodhan@openobserve.ai> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> |