diff --git a/.agents/engineering-guidelines.md b/.agents/engineering-guidelines.md new file mode 100644 index 0000000000..0fd84f8dd3 --- /dev/null +++ b/.agents/engineering-guidelines.md @@ -0,0 +1,231 @@ +# How to build things here + +These rules apply in every chat and every repo. They are not a style preference. They exist because +an agent can write thousands of lines in an hour, some of them invented, and a human then has to +review all of it. Every rule below either cuts invention or makes the code easier to review. + +**Precedence.** A repo's own instruction file (`AGENTS.md` / `CLAUDE.md`) wins on everything +mechanical: formatting, item order, comment rules, lint thresholds, schema rules, PR conventions. +This file governs judgment: what to build, how much of it, and how to prove it works. If the two +conflict, follow the repo file and say so in your report. + +The definitions and tests apply to every language in the repo. The examples in §2 and §3 are written +for the Rust backend; a frontend list belongs next to them when the frontend owners write one. + +## 0. Pick the tier first + +Every task is one of three sizes. Say which one you are in before touching code; it decides how +much of the rest applies. + +| Tier | What it is | Spec gate | Tests | Review | +|---|---|---|---|---| +| **Patch** | One cause, one place. No new public API, table, flag or dependency. Typically under ~50 changed lines. | None. One sentence: cause → fix. | One failing test that reproduces the cause, then green. | Self-review against the three tests in §2, then the gates. | +| **Change** | New behaviour inside the existing architecture: a new branch of logic, a new field, a new error path. | Numbered rules and open questions (§5.2), a few lines each. | One per rule. | Full loop (§5). | +| **Feature** | A new component, endpoint, table, background job or config flag. | Design doc first, per the repo's PR rules; the numbered rules derive from it. | One per rule, plus a failure-path test for every `[strong]` / `[weak]` function. | Full loop, security depth where §5 says so. | + +If a Patch outgrows its definition while you work, stop, say so, and re-enter as a Change. Do not +keep going. + +## 1. Never invent facts + +- **A claim about existing code needs a `path:line` you opened in this session.** A function, + table, config flag, env var or endpoint you have not opened does not exist yet. Search for it + before you use it. This applies to specs, reviews and reports, not to every sentence in chat. +- **A claim about an outside service** (Twilio, Slack, a cloud API) needs a link to the vendor's + docs. "Twilio probably returns X" is not allowed. +- **"I don't know" is a correct answer.** Write it down as an open question. Do not fill the gap + with something that sounds right. +- **If the spec does not decide something, stop and ask.** An invented decision is how a feature + turns into an endless fix loop. Anything new you discover while building becomes a new numbered + rule, and the human decides whether it is in scope. + +## 2. The three things we refuse + +Each one has a definition, a test you can apply mechanically, and examples. The cold reviewer (§5) +applies the same three tests, so there is one vocabulary for writer and reviewer. + +### 2.1 Hacks + +**Definition.** Code that makes a symptom disappear while the underlying statement about the system +stays false. + +**Test.** Delete the new code: is the thing it "fixed" still wrong, for the same reason? Then it was +a hack. Second test: does the fix only work because of a value, name, order or timing that is not +part of any contract? + +**Examples, all banned:** + +- A branch keyed on one input, test name, org id or environment so that one case passes. +- Turning an error into success where the caller needed to know: `.ok()`, `.unwrap_or_default()`, + `let _ = …`, `if let Ok(x)` with no `else`, a handler that logs at debug and continues. +- A sleep, retry or re-ordering that makes a race rarer instead of removing it. +- Widening a type (`String`, `serde_json::Value`, `i64` for everything) so the compiler stops + complaining, instead of modelling the value. +- `#[allow(clippy::…)]`, `#[ignore]`, a weakened assertion, or a raised lint threshold to get green. +- A flag that defaults the new behaviour off so the bug stops reproducing. +- A "temporary" anything with no issue number and no removal condition. + +**Not a hack:** a fallback the spec names and that is visible when it fires (a `warn!` with context, +a metric, an error field in the response). A default is a decision; a silent default is a hack. + +**When the real cause is out of scope:** stop and report it. Do not patch the symptom to finish the +task. + +### 2.2 Over-engineering + +**Definition.** Structure that exists for a future the spec does not contain. + +**Test.** Name the second concrete user that exists today, or that the spec names. If you cannot, +inline it. + +**Examples, all banned until the second user exists:** + +- A trait with one impl; a generic with one instantiation; a type parameter "for testing". +- A config option, env var or feature flag nobody asked for ("in case someone wants to tune it"). +- A newtype or wrapper that enforces no invariant. +- Layers that only forward: `handler → service → manager → repository` with logic in one of them. +- A builder for a struct with three fields; a registry or plugin point with one entry. +- An error enum with variants nothing matches on; one variant per call site when every caller + just logs. +- A new module or file for one function. + +**Counts as a second user:** another call site in the same diff, another provider the spec names, +or an existing trait that every implementation in the repo already follows. **Does not count:** "the +UI will need it", "we may add Postgres", or a test that is the only other caller. + +### 2.3 Over-complication + +**Definition.** Control flow or state a reviewer cannot predict from the signature and the first +screen of the body. Over-engineering is too much structure around the code; over-complication is +too much happening inside one piece of it. + +**Test.** Cover everything below the first screen of the function. Can you say what it returns for +each kind of input? If not, split it. + +**Examples, each needs a split or a name:** + +- A `bool` parameter that switches behaviour (`send(msg, true, false)`): two functions, or an enum. +- More than ~5 parameters: a named struct. +- Nesting deeper than three: an early return or a helper. +- `Option>`, `Result>>`, `(bool, bool, Option)` as a return + type: a named enum. +- An iterator chain whose closures contain branches: a loop. +- A function that is only correct if another one ran first: return the value and pass it in, or + make the ordering a type. +- `Arc>` to share what could be passed by value; `async` or `spawn` for work that is not + I/O; a macro where a function works. +- A `&str` or `i32` that is really an enum (`status == "running"`). +- A mutable local that is reassigned across more than one screen. + +**Default when unsure:** the version with fewer names and fewer branches wins. If two versions both +meet the rules, ship the shorter diff. + +## 3. Build in logical components with predictable behaviour + +Split each Change or Feature into **logical components**. A logical component is one unit with one +job, such as "decide who to page", "render the SMS text" or "send through the provider". + +- **One line, no exceptions.** You must be able to explain each component in one plain sentence + that has no "except when". If you cannot, it is two components. +- **Same input, same output.** Keep the decision logic pure: no I/O, and the clock (`now: i64`), + randomness and config are passed in as arguments. Side effects (database, network, queue) live in + a thin outer layer that only carries out what the pure core decided. In practice: the core takes + the config *values* it needs; the outer layer is the one that reads the global config. +- **Deep is fine, wide is not.** A function may run many steps one after another. It should not + fan out into many branches. Prefer an exhaustive `match` on a project enum (no catch-all `_` + hiding a future variant; `_` is only for `#[non_exhaustive]` foreign enums), early returns, and + lookup tables over chains of `if/else`. If a decision has more than about four outcomes, give the + outcome a name: make it an enum. +- **Write the guarantee in the type wherever the type can say it.** Adapted from Audacity's + safety guarantees (https://audacity.gitbook.io/dev/getting-started/coding-standards#safety_guarantees_defined): + - *pure*: the signature says it. Takes values or `&`, returns a value, not `async`, touches no + global. No tag needed. + - *no-fail*: returns `T`, not `Result`. Any `unwrap`, `expect`, index or unchecked arithmetic + inside needs a one-line comment naming the invariant that makes failure impossible. + - `[strong]`, tagged at the end of the doc comment: either it fully succeeds, or it fails and + nothing changed. For example, one database transaction. + - `[weak: ]`, tagged: a failure can leave partial effects, but + everything stays consistent. Name what is left behind and what makes a retry safe. For example: + `[weak: the ledger row may exist without a send; a retry is a no-op]`. + + Only the outer I/O layer carries `[strong]` / `[weak]`. A tag is a promise: each one needs a test + of its failure path. + +## 4. TDD, without the bloat + +- **Test first, and see it fail.** A test written after the code, or one you never saw fail, + proves nothing. Run the narrowest target that contains it red, then make it green. + Exempt: a pure refactor (behaviour unchanged, the existing tests are the net and you do not edit + them) and a pure deletion. +- **One test per numbered rule or per real branch.** Use table-driven tests (inputs → expected + outputs) for pure components. Do not test the framework, getters or private helpers on their + own. Do not repeat an assertion that another test already makes. +- **Pass values, do not mock.** If a function needs a mock to be tested, its I/O is in the wrong + layer (§3). Mocks are for the thin outer layer only, and only when no real local dependency is + available. +- **Never change a test's expectation to get green** unless the PR states why the old expectation + was wrong. +- **Done = every numbered rule has a passing test and the repo's gates pass.** "It looks + complete" is not done. +- **Size each diff by rule, not by subsystem.** One rule (or a few small ones), its test and its + code, small enough for a person to review in one sitting. + +## 5. The working loop + +In repos that have the `o2-loop` skill (openobserve, o2-enterprise), the skill *is* steps 3–6 +below: it owns the round cap and the findings ledger. Elsewhere, run the steps by hand. + +1. **The main chat is the monitor.** It holds the spec, the rule list and the progress. It + delegates the work and does not do bulk reading or writing itself. +2. **Spec gate** (Change and Feature only). Before any code, write the numbered rules, the + components with their one-line jobs and guarantee tags, and the open questions. Ask the human + about every open question. +3. **Tests.** A subagent writes the tests for the next rule-sized slice. Before it reports, it + checks them once against this list: does each test map to a rule, did it fail first, is any + test a duplicate, is anything invented. +4. **Cold review.** A fresh reviewer (one that did not write the code) gets only the spec path + and the diff, not the conversation, and applies the three tests in §2. Each finding must give + `path:line` and a concrete failure: "input X gives wrong output Y". A finding with no concrete + failure is dropped. +5. **Confirm each finding before acting on it.** Read the cited lines, or run a test that shows + the failure. Reviewers make things up too. Fix only the findings you confirmed, and record the + ones you rejected along with the reason. +6. **Repeat step 4** until a round produces no confirmed finding. If the round cap is reached and + real problems remain, the spec is the problem: go back to the human. +7. **Run the repo's gates (fmt, clippy, tests) before every review round** so reviewers do not + spend tokens on what a tool would catch. + +**Review depth follows risk.** Pure logic with good table tests needs one cold round. Full depth +for anything that touches: unauthenticated endpoints, authorization, signature verification, +secrets, money, data deletion, migrations, anything that pages a human. + +**Stop and ask the human when:** + +- the spec does not decide something (§1); +- the only green path you can find is a hack (§2.1); +- the fix needs to touch files or layers outside the stated scope; +- an abstraction needs a second user and the spec has none (§2.2); +- a test can only pass by changing its expectation; +- the gates fail for a reason that predates your change. + +**Keeping the token cost down.** Give a subagent file paths and a narrow question, not pasted +contents, and ask for conclusions with citations, capped in length. Use read-only search agents +for searching. Continue an existing agent instead of starting one that has to re-read everything. +Do not re-read files you already read, and do not re-derive facts you already settled. + +## 6. UI + +- Follow the repo's UI rules. In openobserve that is the `ui-architect` skill: O2 components, + rem not px, design tokens. Mock the flow on the real product UI before building it. +- **Zero-config for 90% of users.** Defaults that work without any setup, controls whose meaning + is obvious from the screen, and no step that needs the docs. Advanced options stay out of the way + until someone needs them. + +## 7. How to write for the human + +- Write for a reader who has not followed the conversation. Define each term the first time you + use it, use plain words, and make the conclusion the first thing they read. +- When agents reach a decision, present it as: what we will build, how it behaves for the user, + what we decided not to do, and what still needs a human answer. +- A PR description states the tier (§0), lists the numbered rules with the test that covers each, + and lists the rejected review findings with their reasons. diff --git a/.claude/agents/o2-coder.md b/.claude/agents/o2-coder.md index 2fe33b1f4b..ceafcc734d 100644 --- a/.claude/agents/o2-coder.md +++ b/.claude/agents/o2-coder.md @@ -10,7 +10,7 @@ When the brief names a checkout path, every file you read or edit and every comm ## Ground rules -- Follow the repo's `CLAUDE.md` exactly: item ordering, one-line comments, tests module last, clippy thresholds. +- Follow the repo's `AGENTS.md` exactly: item ordering, one-line comments, tests module last, clippy thresholds. - Never run `git commit`, `git add`, `git reset`, `git stash`, or `git push`. The orchestrator's script freezes your working tree as a WIP commit; you only edit files. - Never spawn agents or split your work; if the spec needs more than one coder, that is the orchestrator's decision, and you say so in a `question:` line. - Never write to the ledger except the files named below, and never edit `verdict.json`. @@ -27,7 +27,7 @@ When the brief names a checkout path, every file you read or edit and every comm cargo fmt --all cargo clippy --workspace --all-targets -- -W clippy::too_many_lines -W clippy::cognitive_complexity -W clippy::excessive_nesting -D warnings ``` - If the change touches `#[cfg(feature = "enterprise")]` code, follow `CLAUDE.md`'s enterprise verification rule or say in the evidence that it was not checked. + If the change touches `#[cfg(feature = "enterprise")]` code, follow `AGENTS.md`'s enterprise verification rule or say in the evidence that it was not checked. 4. Write `/round-1/evidence.md`: the exact commands, pass or fail, test names, tails of any failure output, and anything skipped. 5. Reply to the orchestrator with: files changed, gates run, anything you deviated from in the spec and why. diff --git a/.claude/skills/o2-loop/prompts/backend-codex.md b/.claude/skills/o2-loop/prompts/backend-codex.md index 10adb4e17d..615d24c4a6 100644 --- a/.claude/skills/o2-loop/prompts/backend-codex.md +++ b/.claude/skills/o2-loop/prompts/backend-codex.md @@ -1,2 +1,2 @@ ## Notes for the Codex reviewer process -You run in Codex's read-only sandbox inside a disposable checkout of the commit under review. There is no `AGENTS.md` in this repository; `CLAUDE.md` at the checkout root is the rules file, so do not search for anything else. The ledger paths given below are absolute and readable; `rg` is not installed, use `grep`, `cat`, and `git`. Do not run cargo; the evidence section already carries the build and test results. +You run in Codex's read-only sandbox inside a disposable checkout of the commit under review. There is no `CLAUDE.md` in this repository; `AGENTS.md` at the checkout root is the rules file, so do not search for anything else. The ledger paths given below are absolute and readable; `rg` is not installed, use `grep`, `cat`, and `git`. Do not run cargo; the evidence section already carries the build and test results. diff --git a/.claude/skills/o2-loop/prompts/review.md b/.claude/skills/o2-loop/prompts/review.md index 1950fba27f..926ce47143 100644 --- a/.claude/skills/o2-loop/prompts/review.md +++ b/.claude/skills/o2-loop/prompts/review.md @@ -1,8 +1,9 @@ You are the second reviewer in a two-AI loop: the coder (a Claude agent) wrote this change, you (an independent reviewer process) review it, the coder fixes, you verify. Your job is to find real defects that a careful senior Rust engineer would block a PR on. Be precise, not exhaustive. ## Before you look at the diff -1. Read `CLAUDE.md` at the repo root. Its rules on item ordering, comments, and clippy thresholds are hard requirements for this repo. -2. Read the evidence file listed below. It contains the build, clippy, and test results the coder already ran. Do not re-run cargo; the sandbox is read-only. +1. Read `AGENTS.md` at the repo root. Its rules on item ordering, comments, and clippy thresholds are hard requirements for this repo. +2. Read §2 of `.agents/engineering-guidelines.md`: the definitions and tests for a hack, over-engineering and over-complication. Apply those tests, not your own taste, when you report one. +3. Read the evidence file listed below. It contains the build, clippy, and test results the coder already ran. Do not re-run cargo; the sandbox is read-only. ## What to review Only the change set described in the "Change set" section. Read surrounding code as needed to judge it, but do not report pre-existing problems outside the diff unless the diff makes them worse. @@ -14,7 +15,8 @@ Look for, in priority order: - Error handling: swallowed errors, `unwrap` on fallible paths, wrong status codes, silent truncation of results. - API and compatibility contracts: wire-format changes, enterprise `cfg` gates, behaviour differences between OSS and enterprise builds. - Performance regressions on hot paths: extra allocations or clones in loops, O(n²), needless full scans, metrics with high-cardinality labels. -- Repo conventions from `CLAUDE.md`: multi-line or narrating comments, functions placed above constants or types, tests module not last, `pub` and private functions interleaved. +- Hacks, over-engineering and over-complication as `.agents/engineering-guidelines.md` §2 defines them: a symptom patch, an abstraction with one user, a function whose result cannot be predicted from its first screen. +- Repo conventions from `AGENTS.md`: multi-line or narrating comments, functions placed above constants or types, tests module not last, `pub` and private functions interleaved. - Missing tests for new branches, or tests that cannot fail. Do not report: formatting, naming taste, anything `cargo fmt` or `cargo clippy` already enforces, or speculative issues you cannot point to a concrete line for. diff --git a/.claude/skills/o2-loop/prompts/verify.md b/.claude/skills/o2-loop/prompts/verify.md index d5016fc6bd..4c808a69be 100644 --- a/.claude/skills/o2-loop/prompts/verify.md +++ b/.claude/skills/o2-loop/prompts/verify.md @@ -1,7 +1,8 @@ You are the second reviewer in a two-AI loop: the coder (a Claude agent) wrote this change, an independent reviewer process (possibly you) reviewed it in a previous round, the coder responded, and now you verify. You have no memory of the previous round; everything you need is in the files listed below. ## Before you look at the code -1. Read `CLAUDE.md` at the repo root. Its rules are hard requirements for this repo. +1. Read `AGENTS.md` at the repo root. Its rules are hard requirements for this repo. + Apply §2 of `.agents/engineering-guidelines.md` (hack, over-engineering, over-complication) by its tests, not by taste. 2. Read the evidence file for this round (build, clippy, and test results the coder ran after fixing). Do not re-run cargo; the sandbox is read-only. 3. Read every earlier round's findings and the coder's per-finding responses, all listed below in order. A finding's severity, location, and defect description live in the round that first reported it; later rounds only carry its id and status. @@ -14,7 +15,7 @@ A prior finding is any id from any earlier round whose latest status is not `res Do not repeat a prior finding in `findings`; its status lives in `prior_findings`. ## Step 2: review what changed since the previous round -The delta between the previous round's commit and this round's commit is listed below. Review only that delta for new defects, using the same priorities as a first-round review: correctness, concurrency, security, error handling, API contracts, performance, `CLAUDE.md` conventions, missing tests. New findings get fresh ids continuing after the highest id used in any earlier round. +The delta between the previous round's commit and this round's commit is listed below. Review only that delta for new defects, using the same priorities as a first-round review: correctness, concurrency, security, error handling, API contracts, performance, `AGENTS.md` conventions, missing tests. New findings get fresh ids continuing after the highest id used in any earlier round. ## Output rules - `verdict` is `approve` only if no prior finding is `still_open` with an original severity of critical, high, or medium, and no new finding is critical, high, or medium. diff --git a/.gitignore b/.gitignore index 9c69de02e0..25a38a8e65 100644 --- a/.gitignore +++ b/.gitignore @@ -68,6 +68,7 @@ tests/ui-testing/playwright-tests/utils/auth/user.json # ---------- other agents ---------- /.agents/* +!/.agents/engineering-guidelines.md alert_mock # Local verification artifacts (browser screenshots, per-crate scratch DBs) diff --git a/AGENTS.md b/AGENTS.md index 5c2f7db9cb..919077ff86 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,5 +1,12 @@ # OpenObserve — project rules +## Engineering guidelines + +- [.agents/engineering-guidelines.md](.agents/engineering-guidelines.md) governs judgment: task + tiers, what counts as a hack / over-engineering / over-complication, TDD and the review loop. + Read it before any change bigger than a one-cause patch; reviewers apply its §2 tests. On + anything mechanical (ordering, comments, lint thresholds, schema, PR rules) this file wins. + ## Build & workflow - Build with `cargo build` (debug). Never use `--release` unless explicitly asked. diff --git a/Cargo.toml b/Cargo.toml index f45cea1a6a..015d785250 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -51,8 +51,14 @@ cloud = [ ] mimalloc = ["dep:mimalloc"] jemalloc = ["dep:tikv-jemallocator"] +cpu-profiling = [ + "openobserve-api-http/cpu-profiling", + "openobserve-jobs/cpu-profiling", + "openobserve-core/cpu-profiling", +] profiling = [ "jemalloc", + "cpu-profiling", "openobserve-api-http/profiling", "openobserve-jobs/profiling", "openobserve-core/profiling", diff --git a/deploy/build/Dockerfile.profiling.amd64 b/deploy/build/Dockerfile.profiling.amd64 index efc262062d..b7c91e0fa6 100644 --- a/deploy/build/Dockerfile.profiling.amd64 +++ b/deploy/build/Dockerfile.profiling.amd64 @@ -23,7 +23,7 @@ COPY . /openobserve COPY --from=webbuilder /web/dist web/dist RUN mkdir -p /openobserve/target/release/ -RUN --mount=type=cache,target=/root/.cache/sccache cargo build --profile release-profiling --features mimalloc,profiling --target x86_64-unknown-linux-gnu \ +RUN --mount=type=cache,target=/root/.cache/sccache cargo build --profile release-profiling --features mimalloc,cpu-profiling --target x86_64-unknown-linux-gnu \ && sccache --show-stats RUN mv /openobserve/target/x86_64-unknown-linux-gnu/release-profiling/openobserve /openobserve/target/release/openobserve diff --git a/scripts/translations/.translation_state.json b/scripts/translations/.translation_state.json index e872946d4e..ec18762b32 100644 --- a/scripts/translations/.translation_state.json +++ b/scripts/translations/.translation_state.json @@ -2361,8 +2361,8 @@ "actions": { "attachDestination": "e4ee81538188171d", "attachDestinationDesc": "79999248115223c2", - "enableDeduplication": "23b7113a61773230", - "enableDeduplicationDesc": "51b6f906f1e9112d", + "enableDeduplication": "58597d12e244abbb", + "enableDeduplicationDesc": "312e4a7af5d06ccf", "fixAlertQuery": "d5402a926641e540", "fixAlertQueryDesc": "dfb99a99051ee69b", "lengthenEvaluationWindow": "a77df21ce70c24c1", @@ -3337,6 +3337,8 @@ "refreshDestinations": "30bb2318f36e608d", "sectionTitle": "74a883a037bc227f", "sendsToStatus": "df6e0cc2cad0b7b6", + "suppressRepeats": "fb0d6c1b34ed1c65", + "suppressRepeatsTooltip": "cfaddd8a61012886", "targetsDestinationsGroup": "72eb63f032e47e34", "targetsWorkflowsGroup": "e2073645cb0dbbdd", "timezoneWarning": "6baa6a168e65198c" @@ -3654,27 +3656,6 @@ "confirmDeleteDestinations": "9acb690d79a0b4e5", "confirmDeleteTemplates": "e7e356f0b59c474c", "continue": "31fbef162594de01", - "correlation": { - "cancelButton": "19766ed6ccb2f4a3", - "defaultWindow": "75ae7c854a023384", - "defaultWindowDescription": "10dca533b41a0e87", - "defaultWindowPlaceholder": "be8da1a40343815a", - "defaultWindowTooltip": "10dca533b41a0e87", - "description": "3e56d5465d4efdbe", - "enableCrossAlert": "57396cfdc96e6c94", - "enableCrossAlertTooltip": "9c3aaf4cc07f9aee", - "enableOrgLevel": "56334c505798bed6", - "enableOrgLevelTooltip": "7227130ee5c954f6", - "fingerprintGroups": "4784ece2c2a26f88", - "fingerprintGroupsHint": "eb3787c49ca121d4", - "fingerprintGroupsRequired": "a9ffea79feeeeef4", - "fingerprintGroupsTooltip": "ad5c93ed4ffc68fe", - "saveButton": "1509f561f2416598", - "semanticFieldNote": "746705420935a034", - "settingsSaveError": "4924972f958737c6", - "settingsSaved": "51b7016346687749", - "title": "eb8f2d364eed82fa" - }, "createdAt": "3d44337010bfbdf7", "createdBy": "dfd60b9aa5ca29f4", "criticalIf": "d51f1d573fc13e14", @@ -3682,17 +3663,6 @@ "customDestination": "595795c3f2d32edd", "customDestinationDescription": "5966438ae41e979b", "customSql": "874d485cbc30fc67", - "dedup": "401f0a26aa07419c", - "deduplication": { - "fingerprintFields": "971a78dedba8e98c", - "fingerprintFieldsHelp": "3cae384fc3fc416f", - "fingerprintFieldsHint": "f4b085fa4f07283b", - "fingerprintFieldsTooltip": "e481b9fe6b452cdd", - "timeWindow": "671058294d73de1e", - "timeWindowHint": "01d49f5a8b894cd6", - "timeWindowTooltip": "04b798d6e4136819" - }, - "deduplicationConfiguredTooltip": "ac43a07ea8e2bd56", "delete": "e2d0a54968ead24e", "deleteAlertMessage": "b5bd997057b8e7db", "deleteAlertTitle": "73d4e7aa312ba1ac", @@ -5030,8 +5000,6 @@ "actions": { "actionsFor": "c9cb1c707b38a9b8", "closeActions": "821790b7ba7639e1", - "configureDedup": "f352a1749543adbd", - "configureDedupTooltip": "160ba9c7a3b6ae2f", "editAlert": "55d309d6da250466", "editAlertTooltip": "b2960b39285cbde9", "viewAlertHistory": "0e26841319ab8a3b", @@ -5134,9 +5102,6 @@ "noData": "af75117dfbb0edb7", "noTemplatesMsg": "e9e871d4bebbbe00", "noTriggersLoaded": "980bb294c56dd9e3", - "notificationSent": "02e87bf90c4c4431", - "occurrence": "1b12a9d74483571f", - "occurrencesDeduplicated": "46d74043371765f4", "oncallTeam": "b250e6b0b87af388", "oncallTeamAtFireTime": "80a154924b2e1c49", "oncallTeamFromRules": "6ca74cdb253f8fc9", @@ -5148,14 +5113,10 @@ "outputMessages": "ace70a9ce0d04504", "outputMessagesLabel": "ace70a9ce0d04504", "owner": "4b1b8aa3608a26da", - "passedLabel": "11abdb02ce12dc7e", "pause": "858e4ba7a29fd38b", - "pendingBatches": "5a61ad2dc761ebb3", - "pendingBatchesTooltip": "1851046da4cddf4a", "period": "b885b560b9174430", "placeholders": { "addTag": "221f0b0cd0dd4842", - "autoUsesCheckInterval": "580e20ef34637b15", "savedFunctions": "e26ec955b2ba4407", "selectColumn": "91db086469f6b553", "typeSomething": "680ec866a90e116c", @@ -5514,13 +5475,11 @@ "alertSettings": "507c5a3eed5763d3", "alertSetup": "cce2cf121808b3fc", "compareWithPast": "2b534069633d462f", - "conditions": "6303136617bd2f68", - "deduplication": "b1136c8463de1038" + "conditions": "6303136617bd2f68" }, "steps": { "advanced": "9f088dbebd6c3c70", - "compareWithPast": "f72a7d32906c557c", - "deduplication": "68c30f88becadc79" + "compareWithPast": "f72a7d32906c557c" }, "streamConfig": "50009d375dd72fc0", "streamName": "4a25bb3cfd6fd66c", @@ -5588,8 +5547,8 @@ "summaryRecent": "5b047e7e0aea16e4", "summaryTotal": "c9b3c38247f744e1", "suppressed": "feb3abdafeb72a33", - "suppressedByDeduplication": "222f34152e79262d", - "suppressionRate24h": "84e7bccc111a1c19", + "suppressedAsRepeat": "e37596eb0cf6120b", + "suppression": "858c847b742b96bd", "tags": "1331275bc537b4c2", "tagsTooltip": "a56e372b90037999", "template": "84de89c26b0ff73f", @@ -5611,15 +5570,12 @@ "timestamp": "115a2cc92c1097ac", "timezoneRequiredLabel": "109c6d257c45352e", "title": "7e8cd2056da73a7f", - "total": "11239872d178729a", - "totalAlerts": "43ac1c2b6422ca65", "trainingWindow": "9480a67efe24c0ee", "triggerAlert": "8b9c643731c94119", "triggerDetection": "ad2fdbfe066f7a7e", "type": "baaddf70fb5d432b", "unknownHttpStatus": "b764cdc0eab71374", "useCurlyBracesNote": "40531e1b24e2da8b", - "usingDeduplication": "b9b769b6e61b4efa", "validation": { "aggregationColumnRequired": "c2a4688b43f9b7f7", "aggregationFunctionRequired": "4941ce4b9d80bcd5", @@ -6225,7 +6181,7 @@ "AddLibraryAlert": "babfb4ab4d1ddb7a", "AttachDestination": "3c4146ec72d1b65a", "CreateDetector": "f92e9e088ba7d13a", - "EnableDeduplication": "1f9ce350c9773ecf", + "EnableDeduplication": "eba2293ef985f5c3", "EvictLeakedConnections": "4d8d53793e2ad3b6", "FixAlertQuery": "39f58fd1741f97bd", "LengthenEvaluationWindow": "651154e9e09287fd", @@ -6243,7 +6199,7 @@ }, "connected": "22965568d22a14ee", "defaultsNote": "e87dd6ae47bb0736", - "gateHint": "fd019c81f3b1f9e7", + "gateHint": "186791c4a234cc95", "groupCalibration": "2cc6a26ef5e2484f", "groupCoverage": "879ed1aa0d5a6ce9", "groupCoverageHint": "6c192c1211614195", @@ -6385,11 +6341,11 @@ "why": "7d27ee7ad327d185" }, "L-302": { - "done": "c6a795b938e2583f", - "from": "154366e9201d07e3", + "done": "6d93312b7188de0d", + "from": "62bf37b09b005aee", "on": "294f3ee187954fa2", - "to": "6b86b273ff34fce1", - "why": "739fb6ef2a71c28e" + "to": "6979e49983feda16", + "why": "c5fea437c2f5558f" }, "L-303": { "done": "752f6289efad802d", @@ -6606,7 +6562,7 @@ "foundWatch": "9251b3f34a2e81d8", "fridaySummary": "62905f1ed1b16b86", "gatewayPaged": "58a7e729458969c0", - "latencyGrouped": "9a1ebc4193819ad7", + "latencyGrouped": "a81dff48ae005919", "loginPaged": "d8e4ed65191a0363", "newWatch": "f21fc04ec0805552", "poolApproved": "8ff3647e9840cc9c", @@ -8261,6 +8217,7 @@ "critical": "427dd2969bd140be", "degraded": "a8494c12f2243903", "down": "b86d11af79188e58", + "error": "54a0e8c17ebb21a1", "failed": "031a8f0f659df890", "healthy": "7f1e323b6272c620", "offline": "a1794783aab72d20", @@ -8944,6 +8901,9 @@ "bits": "6b4ef7a4f34f2685", "bitsPerSecond": "f3b0dd77d1d3149a", "bottom": "479ad21ac9096236", + "boxplotWhisker": "ace8ebbdaba90aaa", + "boxplotWhiskerIqr": "3217336eec297a44", + "boxplotWhiskerMinMax": "a9636b513c88e43b", "browserDeniedFileDownload": "c29bc3d4a1ceed2b", "bucketUnit": "9c255386ebf5b06b", "bucketUnitCustom": "a7ea7b936ab593f2", @@ -8966,12 +8926,14 @@ "chartSelection": { "alertList": "e4e74afbcd38eab5", "barGauge": "9e949a5820332fca", + "boxplot": "e4d59eb4ea943661", "customChart": "5502642e42fe9de9", "distribution": "72a23e549d099bbd", "flameGraph": "d776d0b395c67943", "funnel": "fc063af055a3a9f3", "gauge": "f6d8f4c86cebb44b", "html": "07239dbd2a1a1dd7", + "logs": "ea2100dc89ae9fe2", "markdown": "0e52f6b9d02515c5", "nodeGraph": "4d9a159f99ec9f79", "sankey": "362c638c72a060cd", @@ -9544,6 +9506,7 @@ "hours": "21e8492938abc179", "hstackedLabel": "0bf475d412d632f3", "htmlPanelEmbedBlocked": "34424226c6a45711", + "htmlPanelFieldTooltip": "94c0048448e73412", "hundredPlaceholder": "ad57366865126e55", "id": "12156e921bdb6085", "identifier": "9b10587f84a2c374", @@ -9675,6 +9638,16 @@ "lineThicknessRange": "ab9f0dadabe42cc2", "linear": "e6950b45d25c2027", "localeFormat": "b8017eeb489109d9", + "logsLevelField": "67352c54cca99a32", + "logsMessageField": "8a0e4ee365cd7b37", + "logsOrder": "6be090825c997103", + "logsOrderNewest": "ffb6f5764bddb68c", + "logsOrderOldest": "6e2ebdab3c02a3e6", + "logsOrderTooltip": "c60ccd9e4b72b11c", + "logsPrettifyJson": "c96fbb7e82c0c6e3", + "logsShowTime": "c53847ca2389f46e", + "logsTimeField": "4e1ee7767412cb49", + "logsWrap": "6727132ac799a256", "longitudeLabel": "6d80458fd17d33fc", "mapCluster": "158f734a97e87cd5", "mapClusterRadius": "5b37352fbfb9f34c", @@ -10100,6 +10073,7 @@ "dashboardDeletedAction": "9af86340789ed912", "dashboardMissing": "5a625c76e6145acc", "defaultNotAvailable": "f100490863c82d99", + "disabledPaused": "4a436bdcf4fbf5d4", "editAccessRequired": "085cd7d51f0b9062", "editForbidden": "3a3a55d6afcabfbe", "editNote": "604f60afa00cb59c", @@ -10122,6 +10096,7 @@ "newLink": "39890d5b20848a6b", "openDashboard": "803f2313cdf44275", "openPublicPage": "193b919d4957430f", + "orgDisabledBanner": "b3c3f4ddc639449b", "orgDisabledHint": "76b67d3fe9918c0b", "pageSubtitle": "69e8f1d6eb64b08b", "pageTitle": "b4bee918655d4e11", @@ -10670,6 +10645,7 @@ "utils": { "addAtLeastOneFieldFor": "e6a070d000e6a750", "addAtLeastOneFieldOnEither": "c26044cb61956d35", + "addAtMostOneFieldFor": "15c467b3dfde01d6", "addExactlyOneFieldAreaStacked": "9307a7ecdb9a1807", "addExactlyOneFieldOnAreaStacked": "5e54b757bd6c1193", "addExactlyOneFieldStacked": "9728d7a031952d75", @@ -10695,6 +10671,13 @@ "argMustBeValidNumber": "333ba2eddd983a49", "argNumberNoValue": "1264a103288e24ce", "argStringNoValue": "cd79d486c995bca9", + "boxplotLowerWhisker": "3415f15325daa2cb", + "boxplotMax": "a1a5936d3b0f8a69", + "boxplotMedian": "daab7c435134742a", + "boxplotMin": "dea79332147ffe1f", + "boxplotOutlier": "c1f25bd5c2732041", + "boxplotSamples": "bc228b269ce0d87f", + "boxplotUpperWhisker": "8d54fdd65dbb7ecf", "configValueNotNegative": "1a5497bb6e9a2fff", "configValueNotNumeric": "c92a74f449a4d907", "configValueOutOfRange": "1a8f45a1d20d6cd4", @@ -12304,6 +12287,12 @@ "description": "b928f13366a82214", "title": "5cc7ab6259dc720f" }, + "noBlockedUsers": { + "action": "38398e7642a68580", + "actionDesc": "b83179b51b4348ea", + "description": "f37628054e521e56", + "title": "7ee90761f864fedc" + }, "noCipherKeys": { "action": "eda5496b306f05bc", "actionDesc": "1001dea858967963", @@ -12354,6 +12343,16 @@ "description": "58946c6145c03a73", "title": "29b0352f2dc58297" }, + "noDomainMappings": { + "action": "160cc87a108cd97f", + "actionDesc": "2d52feca2091f687", + "description": "8d2a1d3961a85569", + "title": "5e555e9dfc2f7305" + }, + "noDomainRequests": { + "description": "eec9ff2e2ca9e75f", + "title": "c242dd5f6af4b624" + }, "noDowntimes": { "action": "928a7246a60c12cf", "actionDesc": "b39e9d84e595a81e", @@ -12422,6 +12421,14 @@ "description": "f8d04da459e9beac", "title": "dbeed0f9193cc3e6" }, + "noLinkedDomains": { + "description": "fa12abdd3599d33a", + "dns": "4bed48849e63af07", + "dnsDesc": "c377b7285abb1d9d", + "request": "66151f5b1821c8d6", + "requestDesc": "6752b021a2d01da5", + "title": "f806dd9c9c9fccc4" + }, "noLlmProviders": { "action": "8cd1856b03dd6844", "actionDesc": "b67cc2375086b515", @@ -12695,10 +12702,13 @@ "anomalyConfigs": "3eec8e1c618c4e60", "apiLimits": "277e1f454b2456f5", "backfillJobs": "9e55cd82af3ec79c", + "blockedUsers": "baa1092cc5cce495", "cipherKeys": "a4ec922695af251c", "dashboards": "21b2e2095fd38f7d", "datasetItems": "5f3c4f8580d392e4", "discoveryItems": "1ee23025c9c0c5a9", + "domainMappings": "2820946d61f000b8", + "domainRequests": "2e76551da294e5e7", "enrichmentTables": "2523e745479b9694", "evalJobs": "6559acc79b8655cb", "evalTemplates": "3d717d23df0a3510", @@ -12708,6 +12718,7 @@ "incidents": "a6d1765ab64d82a7", "ingestionTokens": "94150066b0bceb34", "invitations": "33cd5e40bb5fc8a9", + "linkedDomains": "3478feb608d25877", "llmProviders": "437dbe28117905aa", "logs": "98f38f12db221a8c", "modelPricing": "960087629fb9d135", @@ -13754,13 +13765,34 @@ "all": "a52ace420f2175d0", "allModulesTitle": "7bd30494297e0f94", "bulkSelectColumn": "917098c49df7353d", + "changeCount": "36b588a4e74d7752", + "changeOtherGroup": "f97e9da0e3b879f0", "childResource": "916cd7c979a0b3e2", "clearFilter": "910bda6aed78b23e", "collapseModuleGroup": "35044c06f245c5fe", + "companionAdd": "9fd728c66c9a256b", + "companionAddAll": "05401f46f7cada88", + "companionAddItem": "19daf8f79286b612", + "companionNeeds": "6818176c00700328", + "companionNeedsThis": "63a7087c9444e6e6", + "companionOpenStreams": "ef836aa4f8f14513", + "companionOpenStreamsTooltip": "b4fc0385e294d732", + "companionReasonAlertDestinations": "7919e3c99e13467e", + "companionReasonAlertStreams": "c9bd36f5d3a37867", + "companionReasonDashboardStreams": "1aa7a8e2230242b5", + "companionReasonDestinationTemplates": "0b434e9829465f1f", + "companionStillNeedsThis": "1aabb4e04dc120a6", "dbmPresetNoMatch": "db8e3bdafce56e16", "dbmPresetSeeded": "5248e9a5b6aace4f", + "effectCan": "653af35fcda08995", + "effectCanLoadDestinations": "bde2e69f02f75fde", + "effectCanLoadTemplates": "dec9cf07c30bd11a", + "effectCannot": "f6e679263cfda4e6", + "effectCannotLoadDestinations": "53a21e80ef59aaad", + "effectCannotLoadTemplates": "c79800a8598665dd", + "effectStillAllowed": "5f13d4761582023c", "expandModuleGroup": "ed520c2c4b64e090", - "filterModules": "ceee269a8c7fd7c6", + "filterModulesTasks": "ceee269a8c7fd7c6", "forAllInstances": "6407662fdde3ff55", "grantedByWiderScope": "0dce4fb6dbbe656b", "grantedHereAndByWiderScope": "842472e92e47c946", @@ -13795,28 +13827,77 @@ "moduleScopeAll": "a52ace420f2175d0", "moduleScopeGranted": "57fd7a0cf33f2666", "modulesTitle": "76c86c4c32432be3", + "neededBy": "ac5a7811408bde24", "noGrantedModule": "fce6c29bb74e45f0", "noMatchingResources": "0ff4417804c27000", - "noModuleMatch": "ee7dcb6a716543e0", + "noModuleMatchHint": "76eaaeed65096525", + "noModuleMatchQuery": "7d2ec4c807d9baba", "noUpdatesDetected": "5b015412d0730e6f", + "notAddedYetTitle": "eb7ab14d0c4a7368", + "noteDestinationsFiltered": "ab2187bc5bf81cca", + "noteTemplatesFiltered": "406e4b9e6af308cc", "openFolder": "68b3f4bff6de90ae", "permissionCountSingular": "051f06305d595a6b", "permissions": "abccc78cc93c0793", + "presetAddButton": "914af4653212a900", + "presetAddsTooltip": "13dac67f1e227459", + "presetAlertManagerDescription": "2844c891cb1db276", + "presetAlertManagerTitle": "e86bbad1e2cc63ab", + "presetAlertViewerDescription": "73f45a2fd669376e", + "presetAlertViewerTitle": "649642cf0302439e", + "presetApplied": "430fbac3cb84d5e1", + "presetApplyNote": "bd39009c9fc74003", + "presetBack": "5d84c790bd898dd1", + "presetDashboardEditorDescription": "9136fcdd6f7915f2", + "presetDashboardEditorTitle": "906da4e8764c7d56", + "presetDashboardViewerDescription": "d317eb6722ac155d", + "presetDashboardViewerTitle": "4c576a48726aee1b", "presetDbmDescription": "6fa3fc84efd23374", "presetDbmTitle": "e2795b484db7c9dc", + "presetInRole": "9f31bb38d9b6e0d1", + "presetInRoleTooltip": "0dcd109291b0802a", "presetK8sDescription": "1ab5869177620a3c", "presetK8sTitle": "1348fce2efb5d899", + "presetLoadingStreams": "497dfcec1787bb0e", + "presetMissingItem": "18fcad9e28c393d3", + "presetModuleCount": "3e5b84b40963657c", + "presetNoMatchingStreams": "9e739c44f5232efc", + "presetNothingMissing": "f3a4df9f5f9524c6", + "presetNothingToAdd": "aa4db700a5dec3e1", + "presetPermissionCount": "574057eeed7f5ff4", + "presetPipelineEditorDescription": "de6e3add2840c75e", + "presetPipelineEditorTitle": "af77ceae37d95856", "presetReadonlyDescription": "65d35f1e5efff445", "presetReadonlyTitle": "fd523668f3222a15", + "presetReadsEveryStream": "779eeb4275b6a7f3", + "presetRestoreButton": "37496b30cafdbd3f", + "presetRestored": "98af67b8d9a42604", + "presetRetry": "942087cc2d41e013", "presetRumEditorDescription": "45af3f7b0636c328", "presetRumEditorTitle": "f44097ce697f860c", "presetRumViewerDescription": "397aac1005ba2e4a", "presetRumViewerTitle": "098cb30398d04ae9", + "presetStillMissing": "7b74472a3465f1d2", + "presetStreamsFailed": "770dea01f8e12907", + "presetSummaryAdds": "28ba936dbff7014a", + "presetSummaryHasAll": "bbe8cad8cacfb4f3", + "presetSummaryRestores": "22d18edbad0b14ae", + "presetSummarySkipped": "9952d82e7c251536", + "presetSummaryViewOnly": "3f60f63609742357", + "presetSummaryWrite": "f9265078106ee504", + "presetsClose": "f592612715ae2d33", + "presetsSectionBuiltin": "86d50abefe4a4533", + "presetsSectionJob": "8cd9b4e7688723aa", + "presetsShow": "55ed0bb59f5ff96d", + "presetsSubtitle": "261cf9df639b302a", + "presetsTitle": "954f93fe1c66ae44", "quickReference": "8dc01f004f155067", "reviewChanges": "d9764d6d64cedf78", + "reviewMissingTooltip": "84f3995080a989b0", "roleNotFound": "eb230bc586caaebf", "rumPresetNoMatch": "95873dff644884f3", "rumPresetSeeded": "f9b03957a5cad1c5", + "saveSuccessCounts": "a424d67969865677", "scopeAllOf": "c600e6cbe1bf6ad6", "scopeCoveredByParent": "f8e7e0b7e085d5a1", "scopeDowntimeHint": "2d5e550e6c592398", @@ -13833,9 +13914,10 @@ "show": "0df6f1cad36c49da", "specificInstanceOr": "2b209ead151a81a7", "stillGrantedByWiderScope": "b18dc7bb028b3b76", + "stillNeededBy": "ece32491c7520328", "summaryChangeAdded": "6b02e0d363a4af1c", "summaryChangeRemoved": "4118fb4fed0ecec9", - "summaryEmptyDescription": "bfe990d7ed130301", + "summaryEmptyDescription": "d4777882f82af0b1", "summaryEmptyTitle": "afef6f392c9add8d", "summaryGrantCount": "7ea21433a2c5ca22", "summaryGrantsLabel": "c30209075bd95b47", @@ -13849,13 +13931,41 @@ "summaryReachSome": "de344ba2ec6092cb", "summaryUndoRemove": "a8283ade31856f71", "table": "16d1c9050a0b32e8", + "targetAlertsEveryFolder": "1b30c61b34c2aae2", + "targetAlertsInFolder": "d1132b49171b0895", + "targetAllOf": "4f6fb834c57dacf6", + "targetDashboardsEveryFolder": "de65b7ac822f6f8e", + "targetDashboardsInFolder": "4ad30bcf340c009a", + "targetEveryDestination": "8d06c600c367bbb4", + "targetEveryStream": "41831f70a3de8a91", + "targetEveryTemplate": "33d89317e5d6565f", + "targetItemIn": "0a405945881e1c6a", + "targetOneAlert": "b07dd9e4c2b99b1d", + "targetOneDashboard": "60e538b12883ecbb", + "targetOneDestination": "728f70dab0c79cf2", + "targetOneReport": "7373c57a2afb80cd", + "targetOneStream": "bda1eea240ccdce6", + "targetOneTemplate": "bec08c395e9bcbc4", + "targetReportsEveryFolder": "5735ad2a2035e6e4", + "targetReportsInFolder": "36b71f4bfab131ee", + "targetTypeStreams": "697c8545d79c6c2a", "undoAllChanges": "0ee88bcfd6a2821d", "unsavedDot": { "title": "10ba25333a83c618" }, "updateError": "f63709346743dd32", "updateSuccess": "71bc5feee968e4c1", - "users": "6b0cc904d081a8c1" + "users": "6b0cc904d081a8c1", + "verbCreate": "fa8847b0c3318327", + "verbDelete": "6197595503f01ee2", + "verbEdit": "262121c5372be8af", + "verbList": "a330395cc0a53ad1", + "verbOpen": "2348f99874421257", + "verbSeeListOf": "6a3a48332191d4f9", + "verbView": "2bcb43cbc8f6b7ef", + "yourChanges": "ca00e53e8f4182b3", + "yourChangesCount": "6da14898b0687060", + "yourChangesNone": "98ca0d518997f8a4" }, "externalUserWarningMessage": "0140f11bb6c72a86", "externalUserWarningTitle": "88bc98107237635a", @@ -14058,9 +14168,14 @@ "memberUpdatedSuccessfully": "b5f31eeadc42a082" }, "user": { + "blockedButNotRemoved": "cc3e20dfda028109", + "blockedUsersNotRemoved": "da4d989021bf0d6e", + "blockedUsersPartial": "3396915b26488411", + "blockedUsersSuccess": "25d80ed957b338fd", "delete": "e2d0a54968ead24e", "deletedUsersPartial": "63dd9d50783eb53e", "deletedUsersSuccess": "ff64d76cb2cf3e31", + "errorBlockingUser": "8c9761a0644254f4", "errorDeletingUser": "1fdccd5e6ca1b3ac", "errorDeletingUsers": "4486d04a04ee6f34", "errorRevokingInvitation": "3a16c6ec8755e277", @@ -14075,6 +14190,7 @@ "pleaseWaitLoadingUsers": "fb6392d9b349425a", "unknownError": "27c2ccd962c2b8dc", "userAddedSuccess": "35cfd2ebb7ee4177", + "userBlockedSuccess": "51037bfae45b0ca4", "userDeletedSuccess": "b2d6db91aa0c838b", "userUpdatedSuccess": "c54ffa5dc3449c37" }, @@ -15429,6 +15545,7 @@ "dynamodbPipelineDesc": "f636a7855f10eac7", "dynamodbPipelineTitle": "1e6cafceaead0e9e", "dynamodbTagline": "64b74c679cbff83d", + "ebpfNetworkFlowsGuideLink": "f9c328e8b9c96f47", "ebpfTracingGuideLink": "927dc7edecfa2962", "ec2IamNote": "6110e1fc314f8c86", "enableSessionReplayDesc": "40519dfa9b2e2269", @@ -15546,6 +15663,9 @@ "k8sInstallerNote": "5f425a9e70ac3551", "k8sInternalEndpointNote": "f126f11d13c13953", "k8sLibrariesVariant": "0c527194c5b7ec36", + "k8sNetworkFlowsDesc": "ff40fa4596abcb07", + "k8sNetworkFlowsNote": "f317e2bba99a032f", + "k8sNetworkFlowsTitle": "78e2b5cf58d2ef0d", "k8sTroubleAnnotationA": "aaedaf1ae3dca70c", "k8sTroubleAnnotationQ": "f1d7d6ac036d9fd4", "k8sTroubleEndpointA": "9a233ffcb5cb08e9", @@ -15557,6 +15677,9 @@ "k8sTroubleWebhookA": "91f81f5fbca81dea", "k8sTroubleWebhookQ": "a911ec912d7a5a4d", "linuxAgentFixBody": "998a0d0a4fb72d7d", + "linuxNetworkFlowsDesc": "ce7edf9369ed6098", + "linuxNetworkFlowsNote": "89060e6b3b2e637c", + "linuxNetworkFlowsTitle": "6d919988371b8fb8", "linuxTracesNote": "0301ee038a3d5143", "linuxTracesVerifyNote": "0278ba0252e48cc9", "logsMetricsTracesVariant": "0036533de19fb57f", @@ -15579,6 +15702,9 @@ "mysqlAdminNote": "077314ff6018fa5f", "mysqlHostLabel": "9897c29e4ea26191", "mysqlTagline": "71e733e42a22a969", + "networkFlowsHelp": "5935d43be1041111", + "networkFlowsLabel": "d814b8c3c8c539d7", + "networkFlowsRequired": "ba7a0d62fddf54fb", "noLogsFoundYet": "b33c2d121bfd5962", "noMetricsFoundYet": "ccedc0cc2959d1d3", "noSpansFoundYet": "c6e7c48e1b4b65e2", @@ -19945,6 +20071,7 @@ "textbox": "449d1c52f1b8acf6" }, "weight": "81d27ef6d5033c3e", + "wellCategory": "292c06f0045a45d0", "wellGroup": "34ca0e76608842ff", "wellSize": "1af851907331c0ed", "wellSubgroup": "96fe12a20ed919a6", @@ -20773,6 +20900,72 @@ "unitPerSecond": "71f1adcb645b5392", "updateLimit": "0fc43ae1b62fbe3e" }, + "recentSearches": { + "aboutKeptDays": "31640629384cbf7e", + "aboutKeptForever": "2e08559121a01fa6", + "allPages": "903542125c003f8f", + "allStreams": "7ba01d16e615fc96", + "clear": "0d0ff3d78c31ee1a", + "clearBodyAll": "3979c4c7803210eb", + "clearBodyPage": "51a8842d6e7464e8", + "confirmClear": "496121494e4a967a", + "copyFunction": "6fb093137db6056d", + "copyLink": "dbf362d4f210c780", + "copyQuery": "8e5a75e93167b5a0", + "datePlaceholder": "038e472923ed356d", + "deletedQuery": "6d7f86f5dc466780", + "emptyTitle": "3dca22657bdc4672", + "failed": "031a8f0f659df890", + "failedCount": "cd8f5051cba3686a", + "filterByDateRun": "e78d12f2d00398bd", + "functionCopied": "2f4b48e8061978cc", + "functionLabel": "350c73d410d6a3bc", + "includeStarred": "3e622ea859a8790a", + "keyHints": "ed540677df5e80bc", + "lineCount": "e7b8886bd3dbee6c", + "linkCopied": "d12860c21e78baa5", + "loadError": "bee6fadd3aefbbe7", + "loadMore": "ac8991ef01019cf5", + "loadOnly": "d25f6d7876fb7647", + "loadToFix": "529d4a9b26d98d6b", + "noMatch": "074aee6c434f78e0", + "page": { + "dbm": "fa7fe67124e94375", + "logs": "ea2100dc89ae9fe2", + "metrics": "a58da793c7250f1b", + "profiles": "535e52e4a2616ebe", + "traces": "998741a58ed2b411" + }, + "pageTitle": { + "dbm": "228c84b5a56e6bce", + "logs": "21a9d8d924e102cd", + "metrics": "30219f1c45b8c972", + "profiles": "228c84b5a56e6bce", + "traces": "e1dc9727e9f50873" + }, + "queryCopied": "2b5f7fe902cb48bc", + "ranCount": "e505811bceef8aa8", + "recentRow": "7cb59bfc93961fa0", + "relativeWindow": "e35e9688dee4f7ee", + "rerunNow": "22de87ddeda82846", + "rerunOriginal": "781be17093338815", + "reset": "645982c52b7c11ae", + "resetFilters": "10afa98480f2d06c", + "retry": "942087cc2d41e013", + "searchPlaceholder": "7d854a995bf7dc35", + "star": "e357d396871d1012", + "starredOnly": "fbee73c784a49660", + "statusAll": "ac78229d6b822ce5", + "statusError": "031a8f0f659df890", + "statusOk": "6d9a6f97a5fdaf3b", + "title": "228c84b5a56e6bce", + "today": "2b065c7c9ce466e5", + "undo": "a8283ade31856f71", + "unstar": "318e826a75161b9f", + "viewAll": "30a64216eaeac5d3", + "warning": "d0f76e2e3abbb739", + "yesterday": "566181254b293aa6" + }, "refreshButton": { "dataFresh": "7c21db7765d42a41", "dataGettingStale": "4854378d5d270821", @@ -20782,6 +20975,8 @@ "lastRefreshed": "81969c7ddc3eb624", "minutesAgo": "33f4d52fdfa39099", "notYetRefreshed": "cde59065c4a2f1a1", + "refresh": "0e91610117029a62", + "refreshWithAge": "964aef05f56113a1", "secondsAgo": "8a421d3a725aacf9" }, "regex_patterns": { @@ -21096,6 +21291,7 @@ "loginCallback": "abc3ccb8b27e3974", "modelPricingEditor": "bcb792875fefde91", "notFound": "2d857a3660e0240b", + "orgDomainMapping": "91537c30fffb01dc", "passwordPolicy": "11831de5b6d62b7c", "pipelineBackfillJobs": "54a8767a761a58e0", "regexPatterns": "83a58bf362610444", @@ -21983,6 +22179,7 @@ "events": "8d14f6e72de8f18a", "eventsCount": "70894bd5fc64132e", "eventsUnit": "862417b9e7c3720b", + "exitFullScreen": "897e0cf78b808edd", "failedToDeleteSourceMaps": "c0d7f7d0ff020512", "failedToFetchTraces": "bdde58cab9c24864", "failedToTranslateStackTrace": "f9d33e143a021bff", @@ -22004,6 +22201,7 @@ "frustrationRageTap": "cf9680dedd1e387e", "frustrationSignalType": "ad5bb48dadd8e856", "frustrationSignalsDetected": "7ec6a4626da9715c", + "fullScreen": "674fe2acd0d5f47b", "handled": "1e7536d56c45d9cd", "health": "55898449eb74fb2e", "inTimeRange": "e07beeb7056f9845", @@ -22037,6 +22235,163 @@ "orClickToBrowse": "1c412f2d53be72cd", "osImageAlt": "343d012d58b2e98f", "overview": "d4b1ea5708dd5329", + "overviewPage": { + "asideLabel": "7f42198dd61617ad", + "bandAria": "5ba32c31f9967b1e", + "bandAtP75": "2a617709dcd228a3", + "bandGood": "c939327ca16dcf97", + "bandGoodPct": "103634e5ca02f3b9", + "bandNeeds": "738a3278e945747c", + "bandNeedsPct": "bb57abeb7367b245", + "bandPoor": "94dfd46d9dae63e6", + "bandPoorPct": "975fd9ed2d52b52e", + "chartInterval": "bb124e5867fdbc08", + "chartIntervalOther": "bb124e5867fdbc08", + "chartPerDay": "489b7ab47238e88a", + "chartPerFiveMinutes": "021a55c7a6d71a4c", + "chartPerHour": "6c72a5669544996a", + "chartPerSixHours": "b5df04a98ef1b95b", + "chartPerThirtyMinutes": "f489c4e09a100b11", + "chartTableCaption": "6aa2268c5bb0e779", + "chartTableHide": "a0879dad7c3f8a3d", + "chartTableToggle": "96fa2b6893c3aa4d", + "chartTime": "33b93476cf597a33", + "chartTitle": "45405319ec4a4cba", + "clearFilters": "7179ea0035fca783", + "deadClicks": "00d8ca7f888311a1", + "deltaChange": "9f0e28d14900e0b2", + "deltaChangePts": "e13f0a1ddee092fe", + "deltaFaster": "6a325085434f1240", + "deltaFlat": "3a4846c0d2b92e66", + "deltaHigher": "a8850998e2087257", + "deltaLower": "6379bccb1ae0895d", + "deltaNewMsg": "ec3bed16707958c0", + "deltaNoPrev": "7d233f38225a052d", + "deltaNone": "5d69f9dac5b02100", + "deltaSlower": "946227b5e1582391", + "diffBody": "30d7cce86b377ec7", + "diffLink": "2486d2e78f281c29", + "diffNone": "15c50d905a7115dc", + "diffSegment": "e14ff835a13bf68f", + "diffTitle": "ab83f728b5d37538", + "filterAll": "a52ace420f2175d0", + "filterEnv": "9e471951a1b4106e", + "filterService": "d677190e0a9990e7", + "filterVersion": "dd167905de0defca", + "forbiddenBody": "70052a9d798c22aa", + "forbiddenTitle": "1995b5ea7f6ce981", + "frustrationHint": "55abdb4890279913", + "frustrationReplayHint": "1aa9a318829058cf", + "frustrationReplayLink": "20a5e44483605bea", + "frustrationTitle": "56e60702769e4a73", + "kpiEmptyBody": "906be48ece2078ea", + "kpiEmptyTitle": "22be08ecd16a04a6", + "kpiErrorFree": "7d060fc5d0c5e396", + "kpiErrors": "5ea9d9389ee079aa", + "kpiRegion": "fd18cfaf42d52eae", + "kpiSessions": "6fa3cbf451b2a1d5", + "kpiUnhandled": "5b135a6d535f6e27", + "legendErrors": "cb702378f31507ef", + "legendSessions": "6fa3cbf451b2a1d5", + "markNeeds": "be5d23a3a45353b5", + "markPoor": "4e3795d05d9d3ed2", + "mixedBrowserNote": "004e723c7f422b4d", + "mixedMobileNote": "a9faf098a9c11cc8", + "pageErrorBody": "18f57a4129ed3f49", + "pageErrorTitle": "a4a790365c6aaad9", + "pageForbiddenTitle": "18fb9490a361a81b", + "pagesAbout": "78549e1bd0c69ec8", + "pagesAllLink": "903542125c003f8f", + "pagesAllLinkAria": "ebc1f3d358b3b82f", + "pagesColCls": "c47c6c6948ec5634", + "pagesColErrors": "cb702378f31507ef", + "pagesColImpact": "d1f23f0d13613e27", + "pagesColInp": "515b1a2be24746c8", + "pagesColLcp": "32106b7f555aa7de", + "pagesColPage": "0a30a815d67d7dd2", + "pagesColViews": "69c404591e99d86f", + "pagesEmptyBody": "bb22ebea13bc952d", + "pagesEmptyTitle": "bedda18bba81977d", + "pagesHint": "6c666de4002fc017", + "pagesImpactAria": "26b2708a79bc02ab", + "pagesLegend": "e9750ad656e68f14", + "pagesOpenErrors": "d17006b266c9da39", + "pagesOpenSessions": "2bda223ed7de6e96", + "pagesShowAll": "1177039dd3c91649", + "pagesShowFewer": "f8d72db5eef844c0", + "pagesTableLabel": "4ade2839c953451c", + "pagesTitle": "84cae825462bef43", + "panelFrustration": "a599d4cf692cd001", + "panelKpis": "5c512dd66195eed3", + "panelPages": "bfa062de040f55a1", + "panelRelease": "e0e8c0c197f8220c", + "panelSegments": "08fddbc83d3de346", + "panelVitals": "41d00ea2b1306bee", + "percentileLabel": "bb13af3d0ba72ee2", + "rageClicks": "c75d81aa8628a403", + "regionLabel": "d96040fed3bf811d", + "releaseBefore": "227c3d8e40305d8c", + "releaseEmptyBody": "a1971f4e7dcf7e6b", + "releaseEmptyTitle": "5962ee6e0edc2f96", + "releaseErrorFree": "d2a16e135b1bc49e", + "releaseHint": "a62a0366c41737f4", + "releaseImproved": "e88853216dc0a71e", + "releaseInp": "515b1a2be24746c8", + "releaseLcp": "32106b7f555aa7de", + "releaseLink": "2ceeb359b1e6871a", + "releaseMarker": "60368fcfa4e99557", + "releaseMetric": "2d275a74912cca2e", + "releaseNewErrors": "1b43c82ebc7ff5e2", + "releaseNote": "904272b628b800b2", + "releaseSessions": "6fa3cbf451b2a1d5", + "releaseTitle": "c84fabaf892a66b0", + "releaseWorse": "e760bdbe0be65db9", + "retry": "942087cc2d41e013", + "schemaErrorBody": "ddb5a959ca3f0192", + "schemaErrorTitle": "1bee2618fa07ae28", + "segBrowser": "d31de1a5c5c8ba2a", + "segConnection": "639a40e82b9a96f0", + "segCountry": "701d021d08c54579", + "segDevice": "6ba0bdeccb426d7b", + "segEmptyBody": "4ece6c1e99aa06d5", + "segEmptyTitle": "f20e5e6eeb43ab16", + "segInp": "95e0af44dc3b5f10", + "segNone": "72b7d7e92c4845f1", + "segOs": "343d012d58b2e98f", + "segShare": "f5c9b642aacbb5d9", + "segmentLabel": "29ee089ccecadf9d", + "sparkAria": "0908e97f1487e4c7", + "stateEmptyAction": "ee7646b3f423fb67", + "stateEmptyBody": "0e869c5ea6c07f51", + "stateEmptyFiltered": "ea6f94beb910cab9", + "stateEmptyQuiet": "c17f96e5fe52e329", + "stateEmptyTitle": "6f319360f9e15afc", + "stateErrorBody": "9fe5eb8b129794f1", + "stateErrorTitle": "0aa75faea4d1d91c", + "stateLoading": "eacd56e95b5f3c76", + "stateLowBody": "41b00a8813ab4166", + "stateLowTitle": "1f9b63a6d01336ee", + "stateRefreshing": "b79aa54865301ba9", + "tileOpenErrors": "3f6db05d68ba6874", + "tileOpenSessions": "61c01a80f711f64b", + "verdictAllPass": "02eabfdc35dffe20", + "verdictAllSome": "259b1a4fcf57bf51", + "verdictAllThree": "f5530a6cda768528", + "verdictCompare": "eeb12384c2ff9fa8", + "verdictFail": "bb95413b26462bc7", + "verdictNeedWork": "3a4c30caa999f317", + "verdictNeutral": "41d00ea2b1306bee", + "verdictNoData": "a621207696b9e79f", + "verdictNoDataLead": "c9de9a6ba7dd1ddf", + "verdictOver": "6a95f5e212faa9c1", + "verdictP75Note": "fbf2ee1626eb57ac", + "verdictPass": "f638962f3cc6807c", + "verdictSessions": "8cce09a9cd4caa12", + "vitalCls": "2592bbc7acc7fc0d", + "vitalInp": "33a2dbe6373afa87", + "vitalLcp": "a1ce627d3525a22c", + "vitalLinkFull": "cd27d4c527ac12cb" + }, "performance": "442aded87a55aa8c", "performanceEmptyDescription": "89af28b2afec4422", "performanceEmptyTitle": "b75c54a6d0ca045f", @@ -22484,6 +22839,7 @@ "queryPlan": "f44ad102b5dd5df1", "queryRangeHours": "1eb8a48ad2a585d1", "queryRangeRestrictionMsg": "ea63dde7a2a85e18", + "quickModeHiddenFields": "8be086af5b4a1a1a", "quickModeLabel": "6cd472b871517ce9", "quickPickMoreStreams": "4e1a29d3b10349d0", "quickPickStreamsLabel": "aa2db1e403a5ced8", @@ -22587,6 +22943,7 @@ "traceIdLabel": "13181f1360728438", "transformType": "658b33acde44acb7", "turnOffLiveMode": "c3dfdea2b88d959c", + "turnOffQuickMode": "a8830cbebeaf7cea", "turnOnLiveMode": "bcefde79656f0abc", "unknownError": "27c2ccd962c2b8dc", "unknownErrorOccurred": "43c4deec555b4c7e", @@ -22594,6 +22951,7 @@ "updateSavedView": "f0e9e5ac3af3b640", "updateSavedViewConfirm": "165c5f6398bd57f9", "updateSavedViewWithCurrent": "9ec1d2751c4654c4", + "usageCostHistory": "b97c1a6d83f75800", "userDefinedSchemaLabel": "ad895e214ecd7872", "viewCreatedSuccessfully": "6a620ddf5d4013a9", "viewDeletedSuccessfully": "6cc6f04153415865", @@ -22608,11 +22966,11 @@ "webUrlNotConfigured": "da2e1df8a17952bc" }, "search_history": { - "delayMessage": "33ba3ca60b3d278e", + "delayMessage": "b61c3cadbb0996d7", "executed_at": "5f3ff136b10e28d5", "get_history": "0cccb92604f6a427", "sql_query": "9a4f1370f2b72c82", - "title": "66d7e4a1c738f0b1" + "title": "b97c1a6d83f75800" }, "search_scheduler_job": { "actions": "ff8059dc6752afdd", @@ -22819,7 +23177,6 @@ "addFieldTooltip": "4607cc8ee893f1fa", "addGroup": "94901258ed029f18", "addGroupTooltip": "a01adc7d7cacdae4", - "alertCorrelationTab": "b9b26f1bed14a1cb", "autoSuggestedBanner": "2854b4d3462748aa", "configureServicesHint": "0e2d04c7b30c6a3b", "customizeFieldMappings": "9aa6f419e1527cc3", @@ -22939,28 +23296,6 @@ "domainManagementDesc": "88398e3ac40ce4d7", "domainNeedsBlockedEmails": "53170f72948b23a8", "domainNeedsEmails": "c68d07b51aa28f5a", - "domainOrgMappings": { - "addMapping": "160cc87a108cd97f", - "baseRole": "33551d66efcd283e", - "baseRoleHelp": "50482be96fa489a0", - "confirmRemove": "09bf5961ed02c7ae", - "createMissingRoles": "469ee08dedc79c24", - "deleteTitle": "06380e0419eca2ef", - "domain": "f865f1ee55b26840", - "domainHelp": "c927b335c093577b", - "duplicateDomain": "197ff12f947386be", - "editMapping": "d5ef1883268e1264", - "empty": "df0e529517206088", - "organization": "d764d42592fc9687", - "organizationHelp": "783574eea65c82ad", - "organizationPlaceholder": "d7f6cb5385e83aca", - "organizationRequired": "7d12b1ae42d8cc36", - "roleClaimName": "ed8728fb5ccdde85", - "roleClaimNameHelp": "3247090d0155c3cb", - "roleClaimNamePlaceholder": "2fc659f8dd586eba", - "subtitle": "031a53b7414811e0", - "title": "ac852cfa9baf6fa9" - }, "domainPlaceholder": "cc921aaa05754257", "domainRemoved": "2984d48362fadf0b", "domainRequired": "025f3d105c5c8f7c", @@ -23107,6 +23442,7 @@ "modelPricingList": { "subtitle": "511e0ecb589643d4" }, + "noAllowedEmailsDenyAll": "2fe011dd2aebac32", "noDomainMessage": "6e3b65f95acec5dd", "noRecentErrors": "6e11fa6ce48a8831", "noTextAvailable": "aed109960ed9410e", @@ -23122,6 +23458,145 @@ }, "notAllowedAction": "516e566b5d2c3fb6", "orgDesc": "4a1927826d0fa512", + "orgDomainMapping": { + "accessTab": "f8173cd6b94a1c2d", + "addMapping": "160cc87a108cd97f", + "alreadyLinked": "6ae3405ed1780f90", + "answeredBy": "1cbdadbedf5c01a5", + "approve": "6007acbe30b2cd98", + "approveBody": "b00d39614ef4322d", + "approveTitle": "366542c6d1f68b00", + "approvedLinkInfo": "b1cfd424b6a7445e", + "baseRole": "33551d66efcd283e", + "baseRoleHelp": "85a90ff83b26634c", + "block": "211d0bb8cf4f5b52", + "blockUsers": "38398e7642a68580", + "blockedCount": "194ba0dff38e03e3", + "blockedEmails": "34325ba4b11e09a4", + "blocklistDeleted": "e2fad69615931a95", + "blocklistDomainHelp": "5f3dbd2ee5132f73", + "blocklistEmailsHelp": "686ebe2ffc2bddc9", + "blocklistSaved": "eb876f029f0688fa", + "blocklistSection": "61d2a6f5d1a11c99", + "copyTxtName": "def0de935f302292", + "copyTxtValue": "569f923bc810a4cb", + "createMissingRoles": "469ee08dedc79c24", + "createsRoles": "12c02b1ee83f01dc", + "decisionFailed": "b4fc33bc42ac27d2", + "deleteMappingBody": "6a3206c520f864ab", + "deleteTitle": "45afe4152eb71013", + "deleted": "6abd8b1966fb2d45", + "details": "45989de49fb7f66d", + "detailsDialogTitle": "63acc3402811b23a", + "domain": "f865f1ee55b26840", + "domainHelp": "1be318ee4cb4e85e", + "domainHelpRequest": "edd3090574cd3cf4", + "domainNotVerified": "15133907259330da", + "domainRequired": "025f3d105c5c8f7c", + "duplicateMapping": "61dcad620b1e2340", + "editBlockedUsers": "ee186f1494ed6d9a", + "editMapping": "d5ef1883268e1264", + "enforceConfirmAction": "5a1f096a0d8d7674", + "enforceConfirmBody": "5acef29e5051a39d", + "enforceConfirmTitle": "2dccc6f1294e2267", + "enforceDesc": "4d5ef499aabc3eff", + "enforceSaved": "7e6799ca91bfa75f", + "enforceTitle": "49e1311912a28788", + "failureReason": { + "dnsFailed": "191653c1e32269f8", + "recordMissing": "44606b9b5761726f", + "valueMismatch": "8be355f637c38207" + }, + "filterAll": "a52ace420f2175d0", + "filterPending": "331551b0de4157c9", + "goToLinked": "032b897355406970", + "goToLinkedDesc": "708f38e4fa11734e", + "keepRecordHint": "d48e2893e3f293ff", + "lastCheckFailed": "fb0953690ba3562b", + "lastChecked": "0c0c351a971353b5", + "linkDomain": "c296ce16414c87a7", + "linkFailed": "4f298906f73e8190", + "linkMethod": "bf44e5102a62aa62", + "linked": "61375bb06624de6a", + "linkedTab": "7b80e95841c5b918", + "linkedVia": "fe93da5acf84729f", + "loadFailed": "ff3241d73e3ed1e1", + "mappingDomainHelp": "9be86be6ff2a9ceb", + "mappingsTab": "f64ec16b0d88b0da", + "method": { + "approved": "e0aa737c9d31c98a", + "dns": "ebda700dd30bb4e5", + "dnsDesc": "90d12954ebf11f00", + "dnsTitle": "e831fa84f01eda8c", + "request": "9f1faa720fa12bc5", + "requestDesc": "eeed9a515bfa9c53", + "requestTitle": "58c5afe53ea21557" + }, + "noDomainForBlocklist": "a28c7f62b38ab2f3", + "noVerifiedDomains": "4b2fbc7b41b4dbbb", + "notVerifiedYet": "86d22cfc659ee79a", + "organization": "d764d42592fc9687", + "organizationHelp": "103c714455e69979", + "organizationRequired": "7d12b1ae42d8cc36", + "pageSubtitle": "50fdfbf47a2b0743", + "pendingRequestInfo": "780463319de82427", + "reason": "f81ab834de5f8491", + "reasonHelp": "f846e8387152bb77", + "reasonRequired": "79f223934e006694", + "recordDialogTitle": "8aa73754d5a7ee28", + "recordInstructions": "4ca5d0b0f5b2c777", + "reject": "ab604a360777735f", + "rejectBody": "930940c39aa22759", + "rejectTitle": "20536d41e50d6fa7", + "rejectedRequestInfo": "80371f6d79594b87", + "removeBody": "4b6ebb95913dfb3e", + "removeDomain": "5ce435076ca12c56", + "removeTitle": "716dfac9ef62efb1", + "removeWithMappingsBody": "a81e80b80275a075", + "removed": "186cc53091ad1fd8", + "requestApproved": "fe79a279deff856a", + "requestFailed": "c6cdee5b0a95e612", + "requestRejected": "0fb6a196029117df", + "requestSent": "5b7f10f4000b2af2", + "requestedAt": "2d9e28289facab94", + "requestedBy": "9199a054972b1489", + "requestsTab": "635cd0e916595276", + "roleClaimName": "ed8728fb5ccdde85", + "roleClaimNameHelp": "cc0d33be97a5ae25", + "roleClaimNamePlaceholder": "2fc659f8dd586eba", + "ruleBlockedRequired": "cb376d7e54530b2a", + "saveFailed": "4924972f958737c6", + "saved": "e0bc9ee67876b410", + "searchDomains": "df3b2794a7b1fe92", + "searchMappings": "0076316b776d02c6", + "searchRequests": "4132cecef4287500", + "sendRequest": "3a69f897298e727e", + "state": { + "approved": "87b42e40c2a290e0", + "failed": "031a8f0f659df890", + "pending": "331551b0de4157c9", + "rejected": "aea4a04a80426ed8", + "requested": "ae25c9b1d366d159", + "verified": "4f7838402f37674e" + }, + "status": "920e413c7d411b61", + "tabDesc": "50fdfbf47a2b0743", + "tabLabel": "91537c30fffb01dc", + "txtNameCopied": "b353561fcad08b44", + "txtNameLabel": "e3f134323855ff3a", + "txtValueCopied": "c8dcad93b2b0d76f", + "txtValueLabel": "f7b42a22b9cedbfa", + "unblockAll": "1aec444ddd5eb456", + "unblockAllBody": "79b40d4f58e76828", + "unblockAllTitle": "3c8499510ce80365", + "unlinkFailed": "beae9c59b2f9f019", + "verified": "08e0df6712bb45ef", + "verifiedAt": "4f7838402f37674e", + "verifyDialogTitle": "91e82537dcacc7c5", + "verifyFailed": "f0f2383f8af8d23d", + "verifyHint": "32281a270a11f2f9", + "verifyNow": "db383efba6aef9d9" + }, "orgLabel": "c12331aa1a61f587", "orgStorageEditor": { "accessKeyLabel": "176f1c109e28fa68", @@ -23433,7 +23908,7 @@ "logsRowOpen": "4e62957ded0d0dd0", "logsRunQuery": "0b9252e840f32989", "logsSaveView": "af29df0f9ca1446c", - "logsSearchHistory": "145dd6f1992eb214", + "logsSearchHistory": "8962685dd0fd843e", "logsToggleHistogram": "bebe67d66ebc5b81", "logsToggleSidebar": "9cf94e5f212e1985", "metricsAddToDashboard": "81c5e9c7bb53cc85", @@ -23573,7 +24048,7 @@ "runningQueries": "a026d17f06f54cec", "scoreConfigs": "c3c78db2263139e4", "scorers": "966ddbb027958248", - "searchHistory": "66d7e4a1c738f0b1", + "searchHistory": "b97c1a6d83f75800", "searchSchedulers": "047df1ad4dee93bc", "sessions": "6fa3cbf451b2a1d5", "sourceMaps": "74e923a5b4614cf6", @@ -25148,6 +25623,7 @@ "durationFast": "cfd210c0d957d454", "durationMid": "6a8fa448ca025881", "durationSlow": "03e0b960ac4d0d17", + "error": "54a0e8c17ebb21a1", "failed": "031a8f0f659df890", "passed": "436fe71bb9561f05", "warning": "e981ddae45d8f4ca" @@ -25692,11 +26168,13 @@ "back": "7e411e6d1fb73865", "checkName": "dcd1d5223f73b3a9", "checksTitle": "10014025dca36813", + "loadFailed": "0715f69c63f61005", "noAgents": "53f30015179ba65a", "noChecks": "3709b844e6f621e3", "notFound": "8df056a568a44b88", "pool": "f13024272c385f73", "restartConfig": "39cd3ae99887b0c9", + "retry": "942087cc2d41e013", "title": "57d84785ba9c4e19", "version": "dd167905de0defca" }, @@ -25919,6 +26397,10 @@ "attempts": "06e70139fcf5e5c7", "detailAction": "64cff1319d2fd2cb", "detailSelector": "f02a172a35354ca2", + "detailSkipReason": "52bbd7b4a76b81ed", + "detailStatus": "920e413c7d411b61", + "detailStepId": "22e1beecfe5cf990", + "detailTargetUrl": "5b341c960fb7166e", "detailUrl": "e7a241debad56609", "downloadTrace": "0d79fab080c1efe2", "errorFullscreenTitle": "024c4be2f89daed1", @@ -25963,8 +26445,11 @@ "settleStaleNote": "0aee2b4a2b2be6a0", "showFullError": "4450d34301ab8d70", "showLess": "94ea9b1d33a02975", + "skippedCount": "28541aa9135d134e", "startLoadLabel": "db96939121284345", "stepOf": "9bd8548d86fd91f5", + "stepStatusSkipped": "12698ce1ea5cd4ab", + "stepStatusUnknown": "b764cdc0eab71374", "stepsCount": "6e336a2b2b738463", "title": "95ccb018c85dce04", "viewFullError": "e0fcf523d4155c70", @@ -26069,7 +26554,7 @@ "retriesOnFailure": "a2e021cdc7d91dbd", "retriesOnFailureSuffix": "ad5f0c777a2b470f", "retryDelay": "bbbdc7fae12b28e4", - "retryDelaySuffix": "48e77c7c4e24e2f4", + "retryDelaySuffix": "a2f9474ace454f9a", "schedule": "f4830a1dae298044", "scheduleLater": "058e3d9d22f8df88", "scheduleLaterTooltip": "946c56014f45896f", @@ -26457,7 +26942,6 @@ "aiAnalysisIsAlreadyRunningFor": "cff089f517a5afd5", "alertClonedSuccessfully": "93729e0e45c2f03b", "alertNotFound": "1cb1c54427acf236", - "alertNotFoundInAlerts": "8032bd1c0b1f7207", "alertPausedSuccessfully": "4bbb140b6866b255", "alertResumedSuccessfully": "83e2bd8646284c49", "alertSavedSuccessfully": "797986766aad6827", @@ -26507,7 +26991,6 @@ "noAlertsToPause": "e38354226ff0921c", "noAlertsToResume": "f8190fba75eb0c63", "noDestinationsSelectedForDeletion": "6f3d55af7be19703", - "openingDedupConfig": "8c2139ac0be96855", "pausingAlerts": "22bcfb597fcc0141", "pleaseSelectStreamName": "b7888dff292770a1", "pleaseSelectStreamType": "8dc36f136ae1dc78", @@ -27604,13 +28087,17 @@ "actions": "ff8059dc6752afdd", "add": "6384ffa7712cb2d5", "authType": "8eb3ea9bbde63159", + "block": "211d0bb8cf4f5b52", "cancel": "19766ed6ccb2f4a3", "changePassword": "a949f46230748d3f", + "confirmBlockHead": "e761ae1549da54aa", + "confirmBlockMsg": "90a9f36f481f86ee", "confirmDeleteHead": "4e920cc9cc7cd6e1", "confirmDeleteMsg": "3afa72b6b4978bf9", "currentPasswordRequired": "5000b74e76ec8b67", "customRole": "6e8453849bf2d267", "delete": "e2d0a54968ead24e", + "deleteUsersBlockNote": "783ff61096dbb22c", "deleteUsersMsg": "ce952715001e5703", "deleteUsersTitle": "329abf36a8bd020d", "description": "526e0087cc3f254d", diff --git a/src/api/common/src/auth/jwt.rs b/src/api/common/src/auth/jwt.rs index 3b1386026d..1693e11c1b 100644 --- a/src/api/common/src/auth/jwt.rs +++ b/src/api/common/src/auth/jwt.rs @@ -13,11 +13,6 @@ // You should have received a copy of the GNU Affero General Public License // along with this program. If not, see . -#[cfg(feature = "cloud")] -use openobserve_core::{ - organization::list_org_users_by_user, - self_reporting::cloud_events::{CloudEvent, EventType, enqueue_cloud_event}, -}; #[cfg(all(feature = "enterprise", not(feature = "cloud")))] use { crate::common::meta::user::RoleOrg, @@ -36,6 +31,25 @@ use { std::collections::HashMap, std::sync::LazyLock as Lazy, }; +#[cfg(feature = "cloud")] +use { + config::get_config, + config::{DEFAULT_ORG, meta::user::UserRole, utils::rand::generate_random_string}, + db::org_status, + db::{org_users, organization::get_org_setting}, + o2_enterprise::enterprise::cloud::org_invites, + o2_enterprise::enterprise::{cloud::billing_group, domain_management::meta::AccessDecision}, + o2_openfga::authorizer::authz::get_add_user_to_org_tuples, + o2_openfga::authorizer::roles::{ + check_and_get_crole_tuple_for_new_user, get_roles_for_org_user, + get_user_crole_removal_tuples, + }, + openobserve_core::org_domain_ownership::get_cached_org_for_domain, + openobserve_core::organization::list_org_users_by_user, + openobserve_core::self_reporting::cloud_events::{CloudEvent, EventType, enqueue_cloud_event}, + openobserve_core::users, + std::{collections::HashSet, str::FromStr as _}, +}; #[cfg(all(feature = "enterprise", not(feature = "cloud")))] use { config::meta::user::{UserOrg, UserRole}, @@ -45,19 +59,6 @@ use { }, std::str::FromStr, }; -#[cfg(feature = "cloud")] -use { - config::{DEFAULT_ORG, META_ORG_ID, meta::user::UserRole, utils::rand::generate_random_string}, - db::org_status, - db::{org_users, organization::get_org_setting}, - o2_enterprise::enterprise::cloud::org_invites, - o2_openfga::authorizer::authz::get_add_user_to_org_tuples, - o2_openfga::authorizer::roles::{ - check_and_get_crole_tuple_for_new_user, get_roles_for_org_user, - get_user_crole_removal_tuples, - }, - std::{collections::HashSet, str::FromStr as _}, -}; #[cfg(feature = "cloud")] use crate::common::meta::{ @@ -124,9 +125,9 @@ pub async fn process_token( let mut custom_roles: Vec = vec![]; let mut tuples_to_add = HashMap::new(); - // Try custom claim parsing first + // Try custom claim parsing first, always from _meta if let Some((parsed_orgs, parsed_roles)) = - process_custom_claim_parsing(&dec_token.claims).await + process_custom_claim_parsing(&dec_token.claims, "_meta").await { log::info!( "Using custom claim parsing results: {} orgs, {} custom roles", @@ -879,141 +880,262 @@ pub async fn process_domain_org_mapping( claims: HashMap, is_new_user: bool, ) -> Result { - let meta_settings = get_org_setting(META_ORG_ID).await?; - let mappings = meta_settings.domain_org_mappings; // split email to get the domain - if let Some((_, domain)) = user_email.to_lowercase().split_once("@") { - if let Some(mapped) = mappings - .into_iter() - .find(|m| m.domain.to_lowercase() == domain) - { - match org_status::read_active(&mapped.org_id).await { - Ok(true) => {} - Ok(false) => { - log::warn!( - "domain org mapping for user {user_email} points at inactive org {}, skipping", - mapped.org_id - ); - return Ok(false); + let Some((_, domain)) = user_email.split_once("@") else { + log::warn!( + "user email {user_email} could not be split correctly at @, skipping org domain mapping" + ); + return Ok(false); + }; + let domain = domain.trim().to_lowercase(); + + let Some(org_id) = get_cached_org_for_domain(&domain).await else { + log::info!("no cached domain org mapping found for user {user_email}, continuing normally"); + return Ok(false); + }; + + let org_settings = get_org_setting(&org_id).await?; + let mappings = org_settings.domain_org_mappings; + + let mut allowed_orgs: HashSet<_> = billing_group::list_billing_group_members_of(&org_id) + .await? + .into_iter() + .map(|v| v.member_org_id) + .collect(); + allowed_orgs.insert(org_id.clone()); + + let decision = org_settings.domain_management_config.evaluate(user_email); + if matches!(decision, AccessDecision::Deny) { + if is_new_user { + // for new user who is blocked via domain, we return false, so + // the normal flow can take place + return Ok(false); + } + let orgs = org_users::list_orgs_by_user(user_email).await?; + let mut orgs_to_remove = Vec::new(); + for org in orgs { + if allowed_orgs.contains(&org.org_id) { + orgs_to_remove.push(org.org_id); + } + } + + for org in orgs_to_remove { + match users::remove_user_from_org( + &org, + &user_email, + &get_config().auth.root_user_email, + users::RemovalTarget::User, + ) + .await + { + Ok(v) => { + if !v.status().is_success() { + log::error!( + "error user {user_email} was blocked from org {org_id}, but failed to remove from org {org} with status : {}", + v.status().as_str() + ); + } } Err(e) => { - log::warn!( - "domain org mapping for user {user_email}: cannot read the status of org {}, skipping: {e}", - mapped.org_id + log::error!( + "error user {user_email} was blocked from org {org_id}, but failed to remove from org {org} : {e}" ); - return Ok(false); } } + } - log::info!("found domain org mapping for user {user_email}, processing"); - let base_role = UserRole::from_str(&mapped.base_role).map_err(|e| { - anyhow::anyhow!( - "error parsing domain org role {} to user role : {e:?}", - mapped.base_role - ) - })?; + log::info!("user {user_email} blocked by domain management config of org {org_id}"); + return Ok(true); + } - let need_to_add_user = org_users::get_optional(&mapped.org_id, user_email) - .await? - .is_none(); + let mapped_orgs: Vec<_> = mappings + .into_iter() + .filter(|m| m.domain.to_lowercase() == domain && allowed_orgs.contains(&m.org_id)) + .collect(); - if is_new_user || need_to_add_user { - let mut new_tuples = Vec::new(); - get_add_user_to_org_tuples( - &mapped.org_id, - user_email, - &mapped.base_role, - &mut new_tuples, + if mapped_orgs.is_empty() { + log::info!( + "domain {domain} is owned by {org_id} but no mapping found, processing user {user_email} normally" + ); + return Ok(false); + } + + let mut added = 0; + + for mapped in mapped_orgs { + match org_status::read_active(&mapped.org_id).await { + Ok(true) => {} + Ok(false) => { + log::warn!( + "domain org mapping for user {user_email} points at inactive org {}, skipping", + mapped.org_id ); - update_tuples(new_tuples, vec![]).await?; - org_users::add( - &mapped.org_id, - user_email, - base_role, - &generate_random_string(16), - Some(format!("rum{}", generate_random_string(16))), - ) - .await?; + continue; } - if let Some(claim_name) = mapped.role_claim_name - && let Some(v) = claims.get(&claim_name) - && let Some(arr) = v.as_array() - { - let mut roles = HashSet::new(); - for claim in arr { - if let Some(role) = claim.as_str() { - roles.insert(format_role_name_only(role)); - } - } - let existing_roles = get_roles_for_org_user(&mapped.org_id, user_email).await; + Err(e) => { + log::warn!( + "domain org mapping for user {user_email}: cannot read the status of org {}, skipping: {e}", + mapped.org_id + ); + continue; + } + } - let mut add_roles = Vec::new(); - let mut add_tuples = Vec::new(); - let mut remove_tuples = Vec::new(); + if org_status::is_blocked(&mapped.org_id) { + log::info!( + "user {user_email} supposed to be mapped to org {} but org status is blocked, so skipping", + mapped.org_id + ); + continue; + } + log::info!( + "found domain org mapping {} for user {user_email}, processing", + mapped.org_id + ); + let base_role = UserRole::from_str(&mapped.base_role).map_err(|e| { + anyhow::anyhow!( + "error parsing domain org role {} to user role : {e:?}", + mapped.base_role + ) + })?; - for role in &existing_roles { - if !roles.contains(role) { - let name = format_role_name(&mapped.org_id, role); - get_user_crole_removal_tuples(user_email, &name, &mut remove_tuples); - } - } - for role in &roles { - if !existing_roles.contains(role) { - add_roles.push(format_role_name(&mapped.org_id, role)) - } - } - if !add_roles.is_empty() { - check_and_get_crole_tuple_for_new_user( - user_email, - &mapped.org_id, - add_roles, - mapped.create_missing_roles, - &mut add_tuples, - ) - .await; - } + let need_to_add_user = org_users::get_optional(&mapped.org_id, user_email) + .await? + .is_none(); - if (!add_tuples.is_empty() || !remove_tuples.is_empty()) - && let Err(e) = update_tuples(add_tuples, remove_tuples).await - { + if is_new_user || need_to_add_user { + let mut new_tuples = Vec::new(); + get_add_user_to_org_tuples( + &mapped.org_id, + user_email, + &mapped.base_role, + &mut new_tuples, + ); + update_tuples(new_tuples, vec![]).await?; + org_users::add( + &mapped.org_id, + user_email, + base_role, + &generate_random_string(16), + Some(format!("rum{}", generate_random_string(16))), + ) + .await?; + } + if let Some(claim_name) = mapped.role_claim_name + && let Some(v) = claims.get(&claim_name) + && let Some(arr) = v.as_array() + { + let mut roles = HashSet::new(); + for claim in arr { + if let Some(role) = claim.as_str() { + roles.insert(format_role_name_only(role)); + } + } + let existing_roles = get_roles_for_org_user(&mapped.org_id, user_email).await; + + let mut add_roles = Vec::new(); + let mut add_tuples = Vec::new(); + let mut remove_tuples = Vec::new(); + + for role in &existing_roles { + if !roles.contains(role) { + let name = format_role_name(&mapped.org_id, role); + get_user_crole_removal_tuples(user_email, &name, &mut remove_tuples); + } + } + for role in &roles { + if !existing_roles.contains(role) { + add_roles.push(format_role_name(&mapped.org_id, role)) + } + } + if !add_roles.is_empty() { + check_and_get_crole_tuple_for_new_user( + user_email, + &mapped.org_id, + add_roles, + mapped.create_missing_roles, + &mut add_tuples, + ) + .await; + } + + if !add_tuples.is_empty() || !remove_tuples.is_empty() { + if let Err(e) = update_tuples(add_tuples, remove_tuples).await { log::error!( "error updating claim based role tuples for user {user_email} org {} : {e}", mapped.org_id ); } } - + } + log::info!( + "user {user_email} mapped to org {} successfully via domain org mapping", + mapped.org_id + ); + if need_to_add_user { log::info!( - "user {user_email} mapped to org {} successfully via domain org mapping", + "deleting any invites for {user_email} in org {} as added via org mapping", mapped.org_id ); - if need_to_add_user { - log::info!( - "deleting any invites for {user_email} in org {} as added via org mapping", + if let Err(e) = + org_invites::delete_invites_for_user(&mapped.org_id, &user_email.to_lowercase()) + .await + { + log::error!( + "error in deleting invites for user {user_email} for org {} after joining via domain mapping : {e}", mapped.org_id ); - if let Err(e) = - org_invites::delete_invites_for_user(&mapped.org_id, &user_email.to_lowercase()) - .await - { - log::error!( - "error in deleting invites for user {user_email} for org {} after joining via domain mapping : {e}", - mapped.org_id - ); - } } - log::info!("domain org mapping for user {user_email} successfully processed"); - Ok(true) - } else { - log::info!("no domain org mapping found for user {user_email}, continuing normally"); - Ok(false) } - } else { - log::warn!( - "user email {user_email} could not be split correctly at @, skipping org domain mapping" - ); - Ok(false) + added += 1; } + + let mut removed = 0; + // if added = 0, it means no viable orgs were there for mapping, + // so removing the user would just keep recreating new orgs at each login + if org_settings.enforce_allowed_orgs_only && !is_new_user && added > 0 { + let orgs = org_users::list_orgs_by_user(user_email).await?; + let mut orgs_to_remove = Vec::new(); + for org in orgs { + if !allowed_orgs.contains(&org.org_id) { + orgs_to_remove.push(org.org_id); + } + } + + removed = orgs_to_remove.len(); + for org in orgs_to_remove { + match users::remove_user_from_org( + &org, + &user_email, + &get_config().auth.root_user_email, + users::RemovalTarget::User, + ) + .await + { + Ok(v) => { + if !v.status().is_success() { + log::error!( + "error user {user_email} was domain mapped from org {org_id}, but failed to remove from extra org {org} with status : {}", + v.status().as_str() + ); + } + } + Err(e) => log::error!( + "error user {user_email} was domain mapped from org {org_id}, but failed to remove from extra org {org} : {e}" + ), + } + } + } + + if added == 0 && removed == 0 { + log::info!( + "no mapped orgs were viable, so skipped processing domain mapping for user {user_email} from org {org_id}" + ); + return Ok(false); + } + + log::info!("domain org mapping for user {user_email} successfully processed"); + Ok(true) } #[cfg(feature = "cloud")] @@ -1040,8 +1162,8 @@ pub async fn check_and_add_to_org( vec![] } }; - let pending_invites = pending_invites - .into_iter() + let mut has_pending_invites = pending_invites + .iter() .any(|invite| invite.status == OrgInviteStatus::Pending && invite.expires_at > now); if db_user.is_none() { is_new_user = true; @@ -1069,7 +1191,7 @@ pub async fn check_and_add_to_org( } Err(e) => { log::error!("Error adding user to the database: {}", e); - return Ok((is_new_user, pending_invites)); + return Ok((is_new_user, has_pending_invites)); } } } @@ -1078,8 +1200,48 @@ pub async fn check_and_add_to_org( match process_domain_org_mapping(user_email, claims, is_new_user).await { Ok(processed) => { if processed { - // domain based adding is done, so do not treat as a new user - return Ok((false, pending_invites)); + let user_record = db::user::get_user_by_email(user_email).await; + if user_record.is_none() { + is_new_user = true; + has_pending_invites = false; + match create_new_user(DBUser { + email: user_email.to_owned(), + first_name: first_name.to_owned(), + last_name: last_name.to_owned(), + password: "".to_owned(), + salt: "".to_owned(), + organizations: vec![], // No org + is_external: true, + password_ext: Some("".to_owned()), + }) + .await + { + Ok(_) => { + let mut tuples = vec![]; + get_new_user_creation_tuple(user_email, &mut tuples); + tuples_to_add.insert(user_email.to_string(), tuples); + log::info!("User added to the database"); + } + Err(e) => { + log::error!( + "Error adding user to the database after domain mapping: {}", + e + ); + return Ok((is_new_user, has_pending_invites)); + } + } + } else { + // domain based adding is done, so do not treat as a new user + let orgs = org_users::list_orgs_by_user(user_email).await?; + if orgs.is_empty() { + is_new_user = true; + has_pending_invites = pending_invites.iter().any(|invite| { + invite.status == OrgInviteStatus::Pending && invite.expires_at > now + }); + } else { + return Ok((false, has_pending_invites)); + } + } } } Err(e) => log::error!( @@ -1108,8 +1270,8 @@ pub async fn check_and_add_to_org( existing_orgs[0].role.to_string(), ), // if there are some pending invites, we do not want to create any orgs for the user - _ if pending_invites => { - return Ok((is_new_user, pending_invites)); + _ if has_pending_invites => { + return Ok((is_new_user, has_pending_invites)); } _ => { // Create a default org for the user @@ -1209,7 +1371,7 @@ pub async fn check_and_add_to_org( .await; } - Ok((is_new_user, pending_invites)) + Ok((is_new_user, has_pending_invites)) } /// Process custom claim parsing and populate source_orgs and custom_roles @@ -1219,6 +1381,7 @@ pub async fn check_and_add_to_org( #[cfg(all(feature = "enterprise", not(feature = "cloud")))] async fn process_custom_claim_parsing( claims: &HashMap, + org_id: &str, ) -> Option<(Vec, Vec)> { use o2_openfga::config::get_config as get_openfga_config; @@ -1234,8 +1397,9 @@ async fn process_custom_claim_parsing( // Prepare callback functions for claim parser let iam_settings_fn = || { - Box::pin(async { - match db::organization::get_org_setting("_meta").await { + let org_id = org_id.to_owned(); + Box::pin(async move { + match db::organization::get_org_setting(&org_id).await { Ok(settings) => settings.claim_parser_function, Err(_) => "claim_parser".to_string(), } @@ -1257,11 +1421,12 @@ async fn process_custom_claim_parsing( let function_loader_fn = |function_name: &str| { let function_name = function_name.to_string(); + let org_id = org_id.to_owned(); Box::pin(async move { - match db::functions::get("_meta", &function_name).await { + match db::functions::get(&org_id, &function_name).await { Ok(func) => Ok(func), // Return full Transform object with trans_type Err(e) => Err(anyhow::anyhow!( - "Failed to load function '{}' from _meta org: {}", + "Failed to load function '{}' from {org_id} org: {}", function_name, e )), @@ -1279,7 +1444,7 @@ async fn process_custom_claim_parsing( // Use String as compiled program type for Send compatibility let vrl_compile_fn = |function_content: &str| -> Result { // Just validate that it compiles, but return the source for execution - transform::compile_vrl_function(function_content, "_meta") + transform::compile_vrl_function(function_content, org_id) .map_err(|e| anyhow::anyhow!("VRL compilation failed: {}", e))?; Ok(function_content.to_string()) }; @@ -1289,7 +1454,7 @@ async fn process_custom_claim_parsing( use vrl::compiler::{TargetValue, runtime::Runtime}; // Compile the VRL function - let vrl_config = transform::compile_vrl_function(function_content, "_meta") + let vrl_config = transform::compile_vrl_function(function_content, org_id) .map_err(|e| anyhow::anyhow!("VRL compilation failed: {}", e))?; let mut runtime = Runtime::default(); @@ -1319,12 +1484,12 @@ async fn process_custom_claim_parsing( let js_execute_fn = |function_content: &str, claims: Value| -> Result { // Compile the JavaScript function - let js_config = transform::js::compile_js_function(function_content, "_meta") + let js_config = transform::js::compile_js_function(function_content, org_id) .map_err(|e| anyhow::anyhow!("JavaScript compilation failed: {}", e))?; // Execute JavaScript with claims as input let (result, error) = - transform::js::apply_js_fn(&js_config, claims, "_meta", &[String::new()]); + transform::js::apply_js_fn(&js_config, claims, org_id, &[String::new()]); if let Some(err) = error { return Err(anyhow::anyhow!("JavaScript execution failed: {}", err)); diff --git a/src/api/http/Cargo.toml b/src/api/http/Cargo.toml index 4291696ccb..4e4f761924 100644 --- a/src/api/http/Cargo.toml +++ b/src/api/http/Cargo.toml @@ -42,7 +42,8 @@ vectorscan = [ "openobserve-core/vectorscan", "stream/vectorscan", ] -profiling = ["openobserve-api-management/profiling"] +cpu-profiling = ["openobserve-api-management/cpu-profiling"] +profiling = ["cpu-profiling", "openobserve-api-management/profiling"] [dependencies] common.workspace = true diff --git a/src/api/http/src/handler/http/router/mod.rs b/src/api/http/src/handler/http/router/mod.rs index b6e203df29..41bfdafb56 100644 --- a/src/api/http/src/handler/http/router/mod.rs +++ b/src/api/http/src/handler/http/router/mod.rs @@ -26,7 +26,9 @@ use axum::{ use config::get_config; use openobserve_api_common::{X_O2_ASSISTANT_SESSION_ID, X_O2_ASSISTANT_TURN_ID}; use openobserve_api_ingest::request::{clusters, logs, metrics, profiles, rum}; -#[cfg(feature = "profiling")] +#[cfg(feature = "cloud")] +use openobserve_api_management::request::org_domains; +#[cfg(feature = "cpu-profiling")] use openobserve_api_management::request::profiling; use openobserve_api_management::request::{ alerts, announcements, authz, dashboards, db_monitoring, downtimes, folders, ingest_rejections, @@ -952,12 +954,14 @@ pub fn basic_routes() -> Router { router = router.nest("/node", node_routes); // Debug/profiling routes with auth - #[cfg(feature = "profiling")] + #[cfg(feature = "cpu-profiling")] { - let debug_routes = Router::new() + let debug_routes = Router::new().route("/profile/cpu", get(profiling::cpu_profile)); + #[cfg(feature = "profiling")] + let debug_routes = debug_routes .route("/profile/memory", get(profiling::memory_profile)) - .route("/profile/stats", get(profiling::jemalloc_stats)) - .route("/profile/cpu", get(profiling::cpu_profile)) + .route("/profile/stats", get(profiling::jemalloc_stats)); + let debug_routes = debug_routes .layer(middleware::from_fn(password_policy_middleware)) .layer(middleware::from_fn(auth_middleware)); @@ -1376,7 +1380,7 @@ pub fn service_routes() -> Router { .route("/{org_id}/savedviews", get(search::saved_view::get_views).post(search::saved_view::create_view)) .route("/{org_id}/savedviews/{view_id}", get(search::saved_view::get_view).put(search::saved_view::update_view).delete(search::saved_view::delete_view)) - .route("/{org_id}/query_history", get(query_history::list).post(query_history::record).layer(DefaultBodyLimit::max(query_history::MAX_BODY_BYTES))) + .route("/{org_id}/query_history", get(query_history::list).post(query_history::record).delete(query_history::clear).layer(DefaultBodyLimit::max(query_history::MAX_BODY_BYTES))) .route("/{org_id}/query_history/{id}", patch(query_history::star).delete(query_history::delete).layer(DefaultBodyLimit::max(query_history::MAX_BODY_BYTES))) // Functions @@ -2171,6 +2175,18 @@ pub fn service_routes() -> Router { get(billing_group::check_membership).delete(billing_group::leave_group), ) .route("/{org_id}/billing_group/members", get(billing_group::check_members)) + .route( + "/{org_id}/settings/domains", + get(org_domains::get_linked_domain).post(org_domains::link_domain) + ) + .route("/{org_id}/settings/domains/requests",post(org_domains::raise_request).get(org_domains::list_requests)) + .route("/{org_id}/settings/domain/request/{id}/approve",post(org_domains::approve_request)) + .route("/{org_id}/settings/domain/request/{id}/reject",post(org_domains::reject_request)) + .route("/{org_id}/settings/domains/{domain}",delete(org_domains::delete_linked_domain)) + .route( + "/{org_id}/settings/domains/verify/{domain}", + post(org_domains::verify_domain), + ) .route( "/{org_id}/billing_group/members/{member_org_id}", delete(billing_group::remove_member), @@ -4356,6 +4372,29 @@ mod tests { ); } + #[test] + fn query_history_list_publishes_both_response_shapes() { + let spec = serde_json::to_value(super::openapi::ApiDoc::openapi()).unwrap(); + let ok = &spec["paths"]["/api/{org_id}/query_history"]["get"]["responses"]["200"]; + assert_eq!( + ok["content"]["application/json"]["schema"]["$ref"], + "#/components/schemas/QueryHistoryListResponse" + ); + let shapes = spec["components"]["schemas"]["QueryHistoryListResponse"]["oneOf"] + .as_array() + .expect("the list response is a oneOf"); + let shapes = serde_json::to_string(shapes).unwrap(); + assert!( + shapes.contains("#/components/schemas/QueryHistoryEntry"), + "{shapes}" + ); + assert!(shapes.contains("\"type\":\"array\""), "{shapes}"); + assert!( + shapes.contains("#/components/schemas/QueryHistoryCountResponse"), + "{shapes}" + ); + } + #[cfg(feature = "enterprise")] #[test] fn experiment_lifecycle_actions_are_published_in_the_openapi_surface() { diff --git a/src/api/http/src/handler/http/router/openapi.rs b/src/api/http/src/handler/http/router/openapi.rs index 25cdcfed5e..4795c10455 100644 --- a/src/api/http/src/handler/http/router/openapi.rs +++ b/src/api/http/src/handler/http/router/openapi.rs @@ -120,6 +120,7 @@ use crate::{ openobserve_api_search::search::saved_view::update_view, openobserve_api_management::request::query_history::record, openobserve_api_management::request::query_history::list, + openobserve_api_management::request::query_history::clear, openobserve_api_management::request::query_history::star, openobserve_api_management::request::query_history::delete, openobserve_api_management::request::folders::delete_folder, @@ -698,6 +699,9 @@ use crate::{ openobserve_api_management::request::query_history::QueryHistoryRequest, openobserve_api_management::request::query_history::QueryHistoryStarRequest, openobserve_api_management::request::query_history::QueryHistoryEntry, + openobserve_api_management::request::query_history::QueryHistoryCountResponse, + openobserve_api_management::request::query_history::QueryHistoryListResponse, + openobserve_api_management::request::query_history::QueryHistoryClearResponse, meta::user::UpdateUser, meta::user::UserRoleRequest, meta::user::PostUserRequest, @@ -1618,6 +1622,7 @@ mod tests { for (path, method) in [ ("/api/{org_id}/query_history", "post"), ("/api/{org_id}/query_history", "get"), + ("/api/{org_id}/query_history", "delete"), ("/api/{org_id}/query_history/{id}", "patch"), ("/api/{org_id}/query_history/{id}", "delete"), ] { diff --git a/src/api/management/Cargo.toml b/src/api/management/Cargo.toml index 59729546bf..6f2c87107a 100644 --- a/src/api/management/Cargo.toml +++ b/src/api/management/Cargo.toml @@ -42,7 +42,8 @@ vectorscan = [ "search_service/vectorscan", "stream/vectorscan", ] -profiling = ["openobserve-core/profiling"] +cpu-profiling = ["openobserve-core/cpu-profiling"] +profiling = ["cpu-profiling", "openobserve-core/profiling"] [dependencies] anyhow.workspace = true diff --git a/src/api/management/src/models/alerts/mod.rs b/src/api/management/src/models/alerts/mod.rs index 3dc0c2a989..0895929e88 100644 --- a/src/api/management/src/models/alerts/mod.rs +++ b/src/api/management/src/models/alerts/mod.rs @@ -241,6 +241,10 @@ pub struct Alert { #[serde(default)] #[schema(example = 0)] pub keep_firing_for: i64, + + /// Notify once per state change instead of on every firing evaluation. Enterprise only. + #[serde(default)] + pub suppress_repeats: bool, } /// Configuration for when and how an alert should be triggered. @@ -638,6 +642,7 @@ impl From<(meta_alerts::alert::Alert, Option)> for Alert { pending_period_sec: alert.pending_period_sec, notify_on_recovery: alert.notify_on_recovery, keep_firing_for: alert.keep_firing_for, + suppress_repeats: alert.suppress_repeats, } } } @@ -847,6 +852,7 @@ impl From for meta_alerts::alert::Alert { alert.notify_on_recovery = value.notify_on_recovery; alert.recovery_destinations = value.recovery_destinations; alert.keep_firing_for = value.keep_firing_for.max(0); + alert.suppress_repeats = value.suppress_repeats; alert.normalize_promql_rule_mode(); alert diff --git a/src/api/management/src/models/dashboards.rs b/src/api/management/src/models/dashboards.rs index 5eb6e08c87..402ce77e78 100644 --- a/src/api/management/src/models/dashboards.rs +++ b/src/api/management/src/models/dashboards.rs @@ -558,6 +558,14 @@ mod tests { "alert_list_folder": "", "flame_graph_stack_field": "folded", "flame_graph_value_field": "value", + "boxplot_whisker": "minmax", + "logs_message_field": "log", + "logs_level_field": "severity", + "logs_time_field": "_timestamp", + "logs_wrap": false, + "logs_prettify_json": true, + "logs_show_time": false, + "logs_order": "newest", "override_config": [{ "field": {"matchBy": "series", "value": "errors"}, "config": [ diff --git a/src/api/management/src/request/alerts/history.rs b/src/api/management/src/request/alerts/history.rs index 284e7e8802..e2c2b5de29 100644 --- a/src/api/management/src/request/alerts/history.rs +++ b/src/api/management/src/request/alerts/history.rs @@ -42,12 +42,13 @@ use crate::common::{ }; /// Columns a stream gains only once a row carried them, so [history_columns] adds each one present. -const OPTIONAL_HISTORY_COLUMNS: [&str; 5] = [ +const OPTIONAL_HISTORY_COLUMNS: [&str; 6] = [ "downtime_id", "acked_by", "ack_action", "ack_source", "taken_over_from", + "repeat_suppressed", ]; /// Every column the data query reads but [OPTIONAL_HISTORY_COLUMNS]. @@ -144,6 +145,9 @@ pub struct AlertHistoryEntry { pub ack_source: Option, #[serde(skip_serializing_if = "Option::is_none")] pub taken_over_from: Option, + /// `suppress_repeats` withheld this run because it repeated the last delivered state. + #[serde(skip_serializing_if = "Option::is_none")] + pub repeat_suppressed: Option, } #[derive(Debug, Clone, Serialize, Deserialize, ToSchema)] @@ -437,6 +441,7 @@ pub async fn get_alert_history( ack_action: None, ack_source: None, taken_over_from: None, + repeat_suppressed: None, } }) .collect(); @@ -845,6 +850,7 @@ pub async fn get_alert_history( ack_action: hit_text(&hit, "ack_action"), ack_source: hit_text(&hit, "ack_source"), taken_over_from: hit_text(&hit, "taken_over_from"), + repeat_suppressed: hit.get("repeat_suppressed").and_then(|v| v.as_bool()), }); } @@ -1256,6 +1262,7 @@ mod tests { ack_action: None, ack_source: None, taken_over_from: None, + repeat_suppressed: None, }; assert_eq!(entry.alert_name, "test_alert"); @@ -1315,6 +1322,7 @@ mod tests { ack_action: None, ack_source: None, taken_over_from: None, + repeat_suppressed: None, }; let response = AlertHistoryResponse { @@ -1363,6 +1371,7 @@ mod tests { ack_action: None, ack_source: None, taken_over_from: None, + repeat_suppressed: None, }; assert_eq!(entry.status, "error"); @@ -1593,6 +1602,7 @@ mod tests { ack_action: None, ack_source: None, taken_over_from: None, + repeat_suppressed: None, }; let json = serde_json::to_string(&entry).unwrap(); @@ -1637,6 +1647,7 @@ mod tests { ack_action: None, ack_source: None, taken_over_from: None, + repeat_suppressed: None, }; let response = AlertHistoryResponse { @@ -1688,6 +1699,7 @@ mod tests { ack_action: None, ack_source: None, taken_over_from: None, + repeat_suppressed: None, }; let json = serde_json::to_value(&entry).unwrap(); let obj = json.as_object().unwrap(); @@ -1729,6 +1741,7 @@ mod tests { ack_action: None, ack_source: None, taken_over_from: None, + repeat_suppressed: None, }; let json = serde_json::to_value(&entry).unwrap(); let obj = json.as_object().unwrap(); @@ -1826,4 +1839,17 @@ mod tests { assert_eq!(hit_text(&hit, "acked_by").as_deref(), Some("dana@acme.com")); assert_eq!(hit_text(&hit, "ack_action"), None); } + + #[test] + fn the_repeat_suppressed_column_is_read_once_the_stream_has_it() { + use arrow_schema::{DataType, Field}; + + let mut fields: Vec = HISTORY_COLUMNS + .split(", ") + .map(|name| Field::new(name, DataType::Utf8, true)) + .collect(); + assert!(!history_columns(&Schema::new(fields.clone())).contains("repeat_suppressed")); + fields.push(Field::new("repeat_suppressed", DataType::Boolean, true)); + assert!(history_columns(&Schema::new(fields)).ends_with(", repeat_suppressed")); + } } diff --git a/src/api/management/src/request/cloud/org_usage.rs b/src/api/management/src/request/cloud/org_usage.rs index 6640712cc4..49806c8a33 100644 --- a/src/api/management/src/request/cloud/org_usage.rs +++ b/src/api/management/src/request/cloud/org_usage.rs @@ -18,15 +18,10 @@ use axum::{ response::Response, }; use hashbrown::HashMap; -use o2_enterprise::enterprise::cloud::{ - billing_group, - billings::{self, MeteringProvider}, -}; +use o2_enterprise::enterprise::cloud::billings::MeteringProvider; -use super::IntoHttpResponse; use crate::{ - common::meta::http::HttpResponse as MetaHttpResponse, - models::billings::GetOrgUsageResponseBody, service::org_usage, + common::meta::http::HttpResponse as MetaHttpResponse, models::billings::GetOrgUsageResponseBody, }; /// GetUsageData diff --git a/src/api/management/src/request/fleet/mod.rs b/src/api/management/src/request/fleet/mod.rs index 39d9697e27..bfee817f75 100644 --- a/src/api/management/src/request/fleet/mod.rs +++ b/src/api/management/src/request/fleet/mod.rs @@ -1609,7 +1609,11 @@ pub async fn list_audit( if !allowed_on(FLEET_MANAGEMENT, &org_id, &user_email.user_id, "GET").await { return MetaHttpResponse::forbidden("Forbidden"); } - match openobserve_core::fleet::list_audit(&org_id, query).await { + let permitted = match permitted_fleets(&org_id, &user_email.user_id).await { + Ok(permitted) => permitted, + Err(refused) => return refused, + }; + match openobserve_core::fleet::list_audit(&org_id, query, permitted.as_ref()).await { Ok(mut list) => { list.total = list.items.len(); MetaHttpResponse::json(list) @@ -1808,7 +1812,7 @@ async fn desired_request( } } -/// D19c: the fleets whose items an org-wide list or aggregate shows; `None` is unfiltered. +/// D19c: the fleets an org-wide list or aggregate shows, whatever the list setting; `None` is all. #[cfg(feature = "enterprise")] async fn permitted_fleets( org_id: &str, @@ -1821,7 +1825,7 @@ async fn permitted_fleets( if let Some(objects) = tests::listed_objects(org_id, user_id) { return Ok(permitted_from(org_id, key, objects)); } - match db::authz::list_objects_for_user(org_id, user_id, "GET", key).await { + match db::authz::list_permitted_objects(org_id, user_id, "GET", key).await { Ok(objects) => Ok(permitted_from(org_id, key, objects)), Err(e) => Err(MetaHttpResponse::forbidden(e)), } @@ -2272,7 +2276,7 @@ mod tests { assert_eq!( permitted_fleets(ORG, custom).await.ok(), Some(None), - "OpenFGA off, or the list flag off: unfiltered" + "OpenFGA off: unfiltered" ); for (listed, want) in [ (None, None), diff --git a/src/api/management/src/request/mod.rs b/src/api/management/src/request/mod.rs index f50bb6e105..cd5e8d45be 100644 --- a/src/api/management/src/request/mod.rs +++ b/src/api/management/src/request/mod.rs @@ -58,7 +58,7 @@ pub mod organization; pub mod password_policy; #[cfg(feature = "enterprise")] pub mod playground; -#[cfg(feature = "profiling")] +#[cfg(feature = "cpu-profiling")] pub mod profiling; #[cfg(feature = "enterprise")] pub mod prompts; @@ -90,4 +90,7 @@ pub mod users; #[cfg(feature = "enterprise")] pub mod workflows; +#[cfg(feature = "cloud")] +pub mod org_domains; + use openobserve_api_common::request::{BulkDeleteRequest, BulkDeleteResponse}; diff --git a/src/api/management/src/request/org_domains/mod.rs b/src/api/management/src/request/org_domains/mod.rs new file mode 100644 index 0000000000..737b5098c4 --- /dev/null +++ b/src/api/management/src/request/org_domains/mod.rs @@ -0,0 +1,341 @@ +// Copyright 2026 OpenObserve Inc. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +use axum::{Json, extract::Path, response::Response}; +use common::meta::http::HttpResponse as MetaHttpResponse; +use config::{META_ORG_ID, meta::status_pages::CreateDomainRequest, utils::str::is_valid_domain}; +use infra::table::org_domain_requests::Status; +use openobserve_api_common::extractors::Headers; +use openobserve_core::{org_domain_ownership, org_domain_requests}; +use serde::Deserialize; + +use crate::service::auth::UserEmail; + +#[derive(Deserialize)] +pub struct RequestDecision { + reason: String, +} +pub async fn get_linked_domain(Path(org_id): Path) -> Response { + match org_domain_ownership::get_domains_for_org(&org_id).await { + Ok(v) => MetaHttpResponse::json(v), + Err(e) => { + log::error!("error listing domains for org {org_id} : {e}"); + MetaHttpResponse::internal_error(format!("error listing domains : {e}")) + } + } +} + +pub async fn link_domain( + Path(org_id): Path, + Headers(user): Headers, + Json(body): Json, +) -> Response { + let domain = body.domain.trim().to_lowercase(); + + if !is_valid_domain(&domain) { + return MetaHttpResponse::bad_request(format!( + "domain {domain} is not a valid domain to link" + )); + } + + match org_domain_ownership::get_verified_org_for_domain(&domain).await { + Ok(Some(v)) => { + log::info!( + "org {org_id} tried to link domain {domain} but it is already verified by org {v}" + ); + let org_info = if v == org_id { + "to this org" + } else { + "by some other org" + }; + return MetaHttpResponse::bad_request(format!( + "this domain is already linked {org_info}" + )); + } + Err(e) => { + return MetaHttpResponse::internal_error(format!( + "error getting org domain linking : {e}" + )); + } + _ => {} + } + + let existing = match org_domain_requests::get_req_by_org_domain(&org_id, &domain).await { + Ok(v) => v, + Err(e) => { + return MetaHttpResponse::internal_error(format!( + "error getting org domain requests : {e}" + )); + } + }; + if existing.is_some() { + return MetaHttpResponse::bad_request( + "there is already a request open for this domain by this org", + ); + } + + let attempt = match org_domain_ownership::get_domain_org_record(&org_id, &domain).await { + Ok(v) => v, + Err(e) => { + return MetaHttpResponse::internal_error(format!( + "error getting org domain requests : {e}" + )); + } + }; + + if attempt.is_some() { + return MetaHttpResponse::bad_request(format!( + "there is already a linking created for this domain by this org" + )); + } + + match org_domain_ownership::save_org_domain_mapping(&org_id, &domain).await { + Ok(_) => { + log::info!( + "successfully created org domain ownership entry for {org_id} domain {domain} by user {}", + user.user_id + ); + MetaHttpResponse::ok("successfully created mapping request") + } + Err(e) => { + log::error!("error saving domain {domain} link to org {org_id} : {e}",); + MetaHttpResponse::internal_error(format!("error saving org domain linking : {e}")) + } + } +} + +pub async fn raise_request( + Path(org_id): Path, + Headers(user): Headers, + Json(body): Json, +) -> Response { + let domain = body.domain.trim().to_lowercase(); + + if !is_valid_domain(&domain) { + return MetaHttpResponse::bad_request(format!("domain {domain} is not a valid domain",)); + } + + match org_domain_ownership::get_verified_org_for_domain(&domain).await { + Ok(Some(v)) => { + log::info!( + "org {org_id} tried to link domain {domain} but it is already verified by org {v}" + ); + return MetaHttpResponse::bad_request("this domain is already linked by some org"); + } + Err(e) => { + return MetaHttpResponse::internal_error(format!( + "error getting org domain linking : {e}" + )); + } + _ => {} + } + + match org_domain_requests::get_req_by_org_domain(&org_id, &domain).await { + Err(e) => { + log::error!("error getting org domain request for {org_id} domain {domain} : {e}"); + return MetaHttpResponse::internal_error(e); + } + Ok(Some(v)) => { + return MetaHttpResponse::bad_request(format!( + "org {org_id} already has a request for domain {domain} with status {:?}", + Status::from(v.status) + )); + } + _ => {} + } + + match org_domain_requests::save_org_domain_request(&org_id, &domain, &user.user_id).await { + Ok(_) => { + log::info!( + "successfully created org domain ownership request for {org_id} domain {domain} by user {}", + user.user_id + ); + MetaHttpResponse::ok("successfully created ownership request") + } + Err(e) => { + log::error!("error saving domain {domain} link request to org {org_id} : {e}",); + MetaHttpResponse::internal_error(format!( + "error saving org domain ownership request : {e}" + )) + } + } +} + +pub async fn list_requests(Path(org_id): Path) -> Response { + let requests = if org_id == META_ORG_ID { + org_domain_requests::list_all_req().await + } else { + org_domain_requests::list_by_org(&org_id).await + }; + + match requests { + Ok(v) => MetaHttpResponse::json(v), + Err(e) => { + log::error!("error listing org domain mapping requests for org {org_id} : {e}"); + MetaHttpResponse::internal_error(e) + } + } +} + +pub async fn approve_request( + Path((org_id, id)): Path<(String, String)>, + Headers(user): Headers, + Json(body): Json, +) -> Response { + if org_id != META_ORG_ID { + return MetaHttpResponse::forbidden("operation not allowed"); + } + let reason = body.reason.trim().to_string(); + + let req = match org_domain_requests::get_req_by_id(&id).await { + Ok(Some(v)) => v, + Ok(None) => { + return MetaHttpResponse::not_found("request with given id not found"); + } + Err(e) => { + log::error!("error getting org domain request by id {id} for approval : {e}"); + return MetaHttpResponse::internal_error(e); + } + }; + + if req.status != Status::Pending as i32 { + return MetaHttpResponse::bad_request("This request is not pending, cannot be approved"); + } + + match org_domain_ownership::get_verified_org_for_domain(&req.domain).await { + Ok(Some(v)) => { + return MetaHttpResponse::bad_request(format!( + "this domain is already linked to org {v}, request cannot be processed" + )); + } + Err(e) => { + return MetaHttpResponse::internal_error(format!( + "error getting org domain linking : {e}" + )); + } + _ => {} + } + + if let Err(e) = org_domain_ownership::link_from_meta(&req.org_id, &req.domain).await { + log::error!("error linking the domain to org after request approval : {e}"); + + return MetaHttpResponse::internal_error(format!( + "error linking domain from req {} : {e}", + req.id + )); + } + + match org_domain_requests::update_status(&id, &user.user_id, Status::Approved, reason).await { + Ok(_) => MetaHttpResponse::ok("request approved successfully"), + Err(e) => { + log::error!("error approving domain ownership request {id} : {e}",); + MetaHttpResponse::internal_error(e) + } + } +} + +pub async fn reject_request( + Path((org_id, id)): Path<(String, String)>, + Headers(user): Headers, + Json(body): Json, +) -> Response { + if org_id != META_ORG_ID { + return MetaHttpResponse::forbidden("operation not allowed"); + } + let reason = body.reason.trim().to_string(); + + let req = match org_domain_requests::get_req_by_id(&id).await { + Ok(Some(v)) => v, + Ok(None) => { + return MetaHttpResponse::not_found("request with given id not found"); + } + Err(e) => { + log::error!("error getting org domain request by id {id} for approval : {e}"); + return MetaHttpResponse::internal_error(e); + } + }; + + if req.status != Status::Pending as i32 { + return MetaHttpResponse::bad_request("This request is not pending, cannot be rejected"); + } + + if let Err(e) = + org_domain_requests::update_status(&id, &user.user_id, Status::Rejected, reason).await + { + log::error!("error rejecting domain ownership request {id} : {e}",); + return MetaHttpResponse::internal_error(format!( + "error approving org domain ownership request : {e}" + )); + } + log::info!( + "org domain request {id} for org {} successfully rejected by user {}", + req.org_id, + user.user_id + ); + MetaHttpResponse::ok("request rejected successfully") +} + +pub async fn delete_linked_domain( + Path((org_id, did)): Path<(String, String)>, + Headers(user): Headers, +) -> Response { + let domain = did.trim().to_lowercase(); + let record = match org_domain_ownership::get_domain_org_record(&org_id, &domain).await { + Ok(v) => v, + Err(e) => { + return MetaHttpResponse::internal_error(format!( + "error getting org domain linking : {e}" + )); + } + }; + + let Some(model) = record else { + return MetaHttpResponse::bad_request("org domain link not found"); + }; + + match org_domain_ownership::delete_linked_domain(model).await { + Ok(_) => { + log::info!( + "removed domain link of {domain} from org {org_id} by user {}", + user.user_id + ); + MetaHttpResponse::ok("successfully deleted mapping") + } + Err(e) => { + log::error!( + "error removing domain link of {domain} from org {org_id} by user {} : {e}", + user.user_id + ); + MetaHttpResponse::internal_error(format!("error deleting org domain linking : {e}")) + } + } +} + +pub async fn verify_domain( + Path((org_id, did)): Path<(String, String)>, + Headers(user): Headers, +) -> Response { + let domain = did.trim().to_lowercase(); + match org_domain_ownership::verify_domain_now(&org_id, &domain).await { + Ok(record) => MetaHttpResponse::json(record), + Err(e) => { + log::error!( + "error verifying domain {domain} for org {org_id} by user {}: {e}", + user.user_id + ); + MetaHttpResponse::bad_request(e) + } + } +} diff --git a/src/api/management/src/request/organization/billing_group.rs b/src/api/management/src/request/organization/billing_group.rs index c4644a4968..a1f6d532a5 100644 --- a/src/api/management/src/request/organization/billing_group.rs +++ b/src/api/management/src/request/organization/billing_group.rs @@ -316,6 +316,23 @@ pub async fn remove_member( if let Err(e) = billing_group::remove_billing_member(&org_id, &member_org_id).await { return HttpResponse::bad_request(format!("error in removing member: {e}")); } + + let org_id_copy = member_org_id.clone(); + + if let Err(e) = db::organization::update_org_setting(&org_id, move |settings| { + let before = settings.domain_org_mappings.len(); + settings + .domain_org_mappings + .retain(|m| m.org_id != member_org_id); + settings.domain_org_mappings.len() != before + }) + .await + { + log::error!( + "error updating domain mapping for {org_id_copy} after removing the membership from {org_id} : {e}" + ); + }; + HttpResponse::ok("successfully removed") } @@ -366,6 +383,23 @@ pub async fn leave_group( if let Err(e) = billing_group::remove_billing_member(&membership.payer_org_id, &org_id).await { return HttpResponse::bad_request(format!("error in leaving billing group: {e}")); } + + let org_id_copy = org_id.clone(); + + if let Err(e) = + db::organization::update_org_setting(&membership.payer_org_id, move |settings| { + let before = settings.domain_org_mappings.len(); + settings.domain_org_mappings.retain(|m| m.org_id != org_id); + settings.domain_org_mappings.len() != before + }) + .await + { + log::error!( + "error updating domain mapping for {} after removing the membership of {org_id_copy} : {e}", + &membership.payer_org_id + ); + }; + HttpResponse::ok("successfully left") } diff --git a/src/api/management/src/request/organization/settings.rs b/src/api/management/src/request/organization/settings.rs index 9bdedb504f..65bda8e3f9 100644 --- a/src/api/management/src/request/organization/settings.rs +++ b/src/api/management/src/request/organization/settings.rs @@ -19,6 +19,8 @@ use axum::{ http::StatusCode, response::{IntoResponse, Response}, }; +#[cfg(feature = "cloud")] +use config::utils::str::is_valid_domain; use db::organization::{get_org_setting, set_org_setting}; use infra::errors::{DbError, Error}; #[cfg(feature = "enterprise")] @@ -28,7 +30,6 @@ use { }; #[cfg(feature = "cloud")] use { - config::META_ORG_ID, o2_enterprise::enterprise::cloud::{ billing_group::list_billing_group_members_of, billings::MeteringProvider, customer_billings, }, @@ -78,13 +79,19 @@ pub async fn create( #[cfg(feature = "cloud")] let settings = match check_domain_org_mappings(&org_id, settings).await { Ok(settings) => settings, - Err(refusal) => return refusal, + Err(refusal) => return MetaHttpResponse::bad_request(refusal), }; #[cfg(feature = "cloud")] if let Err(response) = check_budget_config(&org_id, &settings).await { return response; } + #[cfg(feature = "cloud")] + let settings = match check_domain_management_config(settings).await { + Ok(settings) => settings, + Err(msg) => return MetaHttpResponse::bad_request(msg), + }; + match db::organization::update_org_setting(&org_id, |data| { apply_settings_payload(data, settings) }) @@ -262,64 +269,130 @@ fn validate_settings_payload(settings: &OrganizationSettingPayload) -> Result<() async fn check_domain_org_mappings( org_id: &str, mut settings: OrganizationSettingPayload, -) -> Result { +) -> Result { + use hashbrown::HashSet; let Some(mut mappings) = settings.domain_org_mappings.take() else { return Ok(settings); }; - if org_id != META_ORG_ID { - return Ok(settings); - } + let child_orgs = + o2_enterprise::enterprise::cloud::billing_group::list_billing_group_members_of(&org_id) + .await + .unwrap_or_default(); + + let mut allowed_orgs: HashSet<_> = child_orgs.into_iter().map(|v| v.member_org_id).collect(); + allowed_orgs.insert(org_id.to_string()); + + let mut seen_domains = HashSet::new(); for mapping in &mut mappings { + use openobserve_core::org_domain_ownership; + + let domain = mapping.domain.trim().to_lowercase(); + + if !is_valid_domain(&domain) { + return Err(format!("domain {domain} is not a valid domain",)); + } + + if seen_domains.contains(&(domain.clone(), mapping.org_id.clone())) { + return Err(format!( + "domain {domain} is linked to {} multiple times", + mapping.org_id + )); + } + seen_domains.insert((domain.clone(), mapping.org_id.clone())); + + if !allowed_orgs.contains(&mapping.org_id) { + return Err(format!( + "Not allowed to setup a mapping for org {}", + mapping.org_id + )); + } + + if openobserve_core::organization::get_org(&mapping.org_id) + .await + .is_none() + { + return Err(format!("No org with org id {} found", mapping.org_id)); + } + // A deleted org keeps its row as a tombstone, so the row alone does not make it a target. match db::org_status::read_active(&mapping.org_id).await { Ok(true) => {} Ok(false) => return Err(inactive_mapping_target(&mapping.org_id).await), Err(e) => { - return Err(MetaHttpResponse::internal_error(format!( + return Err(format!( "cannot read the status of org {}: {e}", mapping.org_id - ))); + )); } } - if mapping.domain.is_empty() || mapping.domain.contains(' ') { - return Err(MetaHttpResponse::bad_request(format!( + if domain.is_empty() || domain.contains(' ') { + return Err(format!( "domain cannot have space or be empty, bad domain '{}'", mapping.domain - ))); + )); } if !matches!( mapping.base_role.as_str(), "admin" | "editor" | "viewer" | "allowed_user" ) { - return Err(MetaHttpResponse::bad_request(format!( + return Err(format!( "base role {} for org {} is not a valid base role - only admin, editor, viewer, allowed_user are supported", mapping.base_role, mapping.org_id - ))); + )); } if let Some(claim) = mapping.role_claim_name.as_mut() { if claim.trim().is_empty() { - return Err(MetaHttpResponse::bad_request( - "role claim name must not be empty", - )); + return Err("role claim name must not be empty".into()); } *claim = claim.trim().to_string(); } - mapping.domain = mapping.domain.to_lowercase(); + match org_domain_ownership::get_domain_org_record(&org_id, &domain).await { + Ok(None) => { + return Err("This domain is not linked yet".to_string()); + } + Err(e) => { + return Err(format!("error getting domain mappings : {e}")); + } + Ok(Some(v)) => { + use infra::table::org_domain_ownership::OwnershipState; + + if v.verification_state != OwnershipState::Verified as i32 { + return Err("This domain is not verified yet".to_string()); + } + } + } + + mapping.domain = domain; } settings.domain_org_mappings = Some(mappings); Ok(settings) } +#[cfg(feature = "cloud")] +async fn check_domain_management_config( + mut settings: OrganizationSettingPayload, +) -> Result { + if let Some(mut config) = settings.domain_management_config { + config.blocked_emails = config + .blocked_emails + .into_iter() + .map(|v| v.trim().to_lowercase()) + .collect(); + + settings.domain_management_config = Some(config); + } + Ok(settings) +} /// The refusal for a mapping whose target org is not active: missing, blocked, or unreadable. #[cfg(feature = "cloud")] -async fn inactive_mapping_target(org_id: &str) -> Response { +async fn inactive_mapping_target(org_id: &str) -> String { match infra::table::organizations::find(org_id).await { - Ok(None) => MetaHttpResponse::bad_request(format!("No org with org id {org_id} found")), - Ok(Some(_)) => MetaHttpResponse::bad_request(format!("org {org_id} is not active")), - Err(e) => MetaHttpResponse::internal_error(format!("cannot read org {org_id}: {e}")), + Ok(None) => format!("No org with org id {org_id} found"), + Ok(Some(_)) => format!("org {org_id} is not active"), + Err(e) => format!("cannot read org {org_id}: {e}"), } } @@ -449,6 +522,16 @@ fn apply_settings_payload( field_found = true; data.budget_config = configs; } + #[cfg(feature = "cloud")] + if let Some(enforce) = settings.enforce_allowed_orgs_only { + field_found = true; + data.enforce_allowed_orgs_only = enforce; + } + #[cfg(feature = "cloud")] + if let Some(config) = settings.domain_management_config { + field_found = true; + data.domain_management_config = config; + } // this will always be taken from orgs table, never from settings data.free_trial_expiry = None; field_found diff --git a/src/api/management/src/request/profiling.rs b/src/api/management/src/request/profiling.rs index c1d791cb6e..14ec5c8c0c 100644 --- a/src/api/management/src/request/profiling.rs +++ b/src/api/management/src/request/profiling.rs @@ -15,11 +15,14 @@ //! HTTP handlers for profiling endpoints -use axum::{Json, extract::Query}; +#[cfg(feature = "profiling")] +use axum::Json; +use axum::extract::Query; use config::get_config; +use openobserve_core::self_profiles::dump_cpu_profile; +#[cfg(feature = "profiling")] use openobserve_core::self_profiles::{ - dump_cpu_profile, dump_memory_flamegraph, dump_memory_pprof, - jemalloc_stats as read_jemalloc_stats, + dump_memory_flamegraph, dump_memory_pprof, jemalloc_stats as read_jemalloc_stats, }; use serde::Deserialize; @@ -29,6 +32,7 @@ use serde::Deserialize; /// The profile can be analyzed with jeprof or converted to flamegraph. /// /// Returns: Binary profile data +#[cfg(feature = "profiling")] pub async fn memory_profile() -> Result { let pprof_data = dump_memory_pprof().await?; let timestamp = chrono::Utc::now().format("%Y%m%d_%H%M%S"); @@ -58,6 +62,7 @@ pub async fn memory_profile() -> Result { /// - allocated: Current allocated memory (bytes) /// - resident: Current resident memory (bytes) /// - metadata: Metadata overhead (bytes) +#[cfg(feature = "profiling")] pub async fn jemalloc_stats() -> Result, String> { Ok(Json(read_jemalloc_stats()?)) } diff --git a/src/api/management/src/request/public_dashboards/public.rs b/src/api/management/src/request/public_dashboards/public.rs index 71b0354871..3f5f4243a3 100644 --- a/src/api/management/src/request/public_dashboards/public.rs +++ b/src/api/management/src/request/public_dashboards/public.rs @@ -19,7 +19,7 @@ use std::{sync::LazyLock, time::Duration}; use axum::{ Json, - extract::{Extension, Path, Query}, + extract::{Extension, Path, Query, rejection::QueryRejection}, http::{HeaderMap, HeaderValue, StatusCode, header}, response::{IntoResponse, Response}, }; @@ -182,7 +182,7 @@ pub async fn config(ip: Option>, Path(slug): Path) -> pub async fn data( ip: Option>, Path(slug): Path, - Query(params): Query, + query: Result, QueryRejection>, headers: HeaderMap, ) -> Response { let if_none_match = headers @@ -191,6 +191,9 @@ pub async fn data( if rate_limited(ip.map(|e| e.0)) || shed_early(&slug, if_none_match) { return with_headers(too_many_requests(READ_WINDOW)); } + let Ok(Query(params)) = query else { + return with_headers(invalid_range()); + }; with_headers(serve_data(&slug, params.range.as_deref(), if_none_match).await) } @@ -430,6 +433,11 @@ fn expired() -> Response { (StatusCode::GONE, "expired").into_response() } +// The extractor's own text names serde internals, e.g. a repeated `range`. +fn invalid_range() -> Response { + (StatusCode::BAD_REQUEST, "invalid range").into_response() +} + // A failed read is temporary, so the viewer retries instead of treating the link as gone. fn server_error() -> Response { (StatusCode::INTERNAL_SERVER_ERROR, "Something went wrong.").into_response() @@ -833,6 +841,7 @@ mod tests { StatusCode::OK.into_response(), StatusCode::NOT_FOUND.into_response(), expired(), + invalid_range(), unavailable(), too_many_requests(READ_WINDOW), server_error(), @@ -846,6 +855,22 @@ mod tests { } } + #[tokio::test] + async fn a_query_the_extractor_refuses_is_a_plain_invalid_range() { + let uri: axum::http::Uri = "/data?range=r3600&range=r60".parse().unwrap(); + let resp = data( + None, + Path("Qz8Zt2mR7aB3cD4eF5gH6Q".into()), + Query::try_from_uri(&uri), + HeaderMap::new(), + ) + .await; + assert_eq!(resp.status(), StatusCode::BAD_REQUEST); + assert_eq!(resp.headers()[header::CACHE_CONTROL], "no-store"); + let body = axum::body::to_bytes(resp.into_body(), 1024).await.unwrap(); + assert_eq!(&body[..], b"invalid range"); + } + #[test] fn gate12_a_missing_settings_row_means_enabled_and_a_disabled_row_stops_serving() { assert_eq!(org_gate(None), (true, 0)); diff --git a/src/api/management/src/request/query_history.rs b/src/api/management/src/request/query_history.rs index 81be8fcbaf..b48d88d9d0 100644 --- a/src/api/management/src/request/query_history.rs +++ b/src/api/management/src/request/query_history.rs @@ -21,7 +21,10 @@ use axum::{ }; use infra::table::{ entity::query_history::Model, - query_history::{self, MAX_STARRED_PER_USER, SetStarred}, + query_history::{ + self, DEFAULT_SOURCE, ListFilter, MAX_STARRED_PER_USER, NewRun, SOURCES, STATUS_ERROR, + STATUS_OK, SetStarred, + }, }; use openobserve_api_common::extractors::Headers; use openobserve_core::auth::UserEmail; @@ -34,14 +37,27 @@ use crate::common::meta::http::HttpResponse as MetaHttpResponse; pub const MAX_BODY_BYTES: usize = 128 * 1024; const MAX_QUERY_BYTES: usize = 16 * 1024; const MAX_CONTEXT_BYTES: usize = 64 * 1024; +const MAX_STREAMS: usize = 50; const DEFAULT_LIMIT: u64 = 50; const MAX_LIMIT: u64 = 200; #[derive(Deserialize, Serialize, ToSchema)] pub struct QueryHistoryRequest { + /// `logs`, `traces`, `metrics`, `dbm` or `profiles`; defaults to `metrics`. + #[serde(default)] + pub source: Option, /// The query text, at most 16 KB and not blank. pub query: String, - /// Time range, step, chart type and the panel data that reopens the query, at most 64 KB. + /// Runs with the same source and key merge into one entry; defaults to the query text. + #[serde(default)] + pub query_key: Option, + /// Streams the query ran on, at most 50; each name non-empty and without a comma. + #[serde(default)] + pub streams: Vec, + /// `ok` or `error`; defaults to `ok`. + #[serde(default)] + pub status: Option, + /// What the page needs to restore the run, at most 64 KB; stored as given. pub context: serde_json::Value, } @@ -55,24 +71,43 @@ pub struct QueryHistoryStarRequest { pub struct QueryHistoryEntry { /// Entry id. pub id: String, + /// Page the query ran on. + pub source: String, /// The query text. pub query: String, - /// Time range, step, chart type and the panel data that reopens the query. + /// Hex sha256 of the merge key; null on entries recorded before keys existed. + pub query_key: Option, + /// Streams the query ran on. + pub streams: Vec, + /// What the page needs to restore the run. pub context: serde_json::Value, + /// `ok` or `error`, from the latest run; `ok` on entries recorded by v1.1.x. + pub status: String, /// Starred entries are kept until deleted. pub starred: bool, - /// When the query was last run, in microseconds since the epoch. + /// How many runs merged into this entry; 1 on entries recorded by v1.1.x. + pub run_count: i64, + /// First run, in microseconds since the epoch; the last run on entries recorded by v1.1.x. pub created_at: i64, + /// Latest run, in microseconds since the epoch. + pub updated_at: i64, } impl From for QueryHistoryEntry { fn from(m: Model) -> Self { Self { + streams: query_history::decode_streams(&m.streams), id: m.id, + source: m.source, query: m.query, + query_key: m.query_key, context: m.context, + status: m.status, starred: m.starred, + run_count: m.run_count, created_at: m.created_at, + // A v1.1.x node in a rolling upgrade writes updated_at = 0; only the response is fixed. + updated_at: m.updated_at.max(m.created_at), } } } @@ -80,18 +115,55 @@ impl From for QueryHistoryEntry { #[derive(Deserialize, IntoParams)] #[into_params(parameter_in = Query)] pub struct QueryHistoryListParams { + /// Comma list of sources (logs, traces, metrics, dbm, profiles); omitted means metrics. + pub source: Option, + /// `ok` or `error`. + pub status: Option, + /// Exact stream name the entry ran on. + pub stream: Option, + /// Case-insensitive substring of the query text or the stream names. + pub q: Option, /// Only starred (`true`) or only unstarred (`false`) entries. pub starred: Option, - /// Substring match on the query text. - pub q: Option, + /// Entries last run at or after this time, in microseconds. + pub ran_from: Option, + /// Entries last run at or before this time, in microseconds. + pub ran_to: Option, /// Defaults to 50, at most 200. pub limit: Option, - /// Entries to skip, for paging. Defaults to 0. + /// Entries to skip, for paging. Defaults to 0, at most 9223372036854775807. pub offset: Option, + /// Answer `{"count": n}`, the number of entries the filters match, instead of the entries. + pub count_only: Option, } -fn list_limit(limit: Option) -> u64 { - limit.unwrap_or(DEFAULT_LIMIT).clamp(1, MAX_LIMIT) +#[derive(Deserialize, IntoParams)] +#[into_params(parameter_in = Query)] +pub struct QueryHistoryClearParams { + /// One source or a comma-separated list; all sources when omitted. + pub source: Option, + /// Also delete starred entries. Defaults to false. + pub include_starred: Option, +} + +#[derive(Deserialize, Serialize, ToSchema)] +pub struct QueryHistoryCountResponse { + /// Entries matching the filters, ignoring limit and offset. + pub count: u64, +} + +/// The entries, or with `count_only=true` how many entries match. +#[derive(Serialize, ToSchema)] +#[serde(untagged)] +pub enum QueryHistoryListResponse { + Entries(Vec), + Count(QueryHistoryCountResponse), +} + +#[derive(Deserialize, Serialize, ToSchema)] +pub struct QueryHistoryClearResponse { + /// Number of entries deleted. + pub deleted: u64, } /// RecordQueryHistory @@ -103,7 +175,7 @@ fn list_limit(limit: Option) -> u64 { tag = "Query History", operation_id = "RecordQueryHistory", summary = "Record a query history entry", - description = "Records a query the caller ran. Running the same query as the caller's latest entry refreshes that entry instead of adding one. Stores nothing and answers 204 when ZO_QUERY_HISTORY_ENABLED is false.", + description = "Records a query the caller ran. A run with the same source and query_key as an existing entry of the caller merges into it: the run count goes up and the latest run time, context and status replace the old ones. Stores nothing and answers 204 when ZO_QUERY_HISTORY_ENABLED is false.", security( ("Authorization"= []) ), @@ -114,7 +186,7 @@ fn list_limit(limit: Option) -> u64 { responses( (status = 200, description = "Success", content_type = "application/json", body = QueryHistoryEntry), (status = 204, description = "Query history is disabled; nothing was stored"), - (status = 400, description = "Query empty, or query or context too large", content_type = "application/json", body = ()), + (status = 400, description = "Unknown source or status, query empty, query or context too large, too many streams, or an empty stream name or one with a comma (the message lists them)", content_type = "application/json", body = ()), (status = 413, description = "Request body larger than 128 KB"), (status = 500, description = "Failure", content_type = "application/json", body = ()), ), @@ -146,7 +218,7 @@ pub async fn record( tag = "Query History", operation_id = "ListQueryHistory", summary = "List query history", - description = "Lists the caller's query history, newest first. Empty when ZO_QUERY_HISTORY_ENABLED is false.", + description = "Lists the caller's query history, most recently run first. With count_only=true it answers {\"count\": n} (QueryHistoryCountResponse), the number of entries the filters match. Empty, or a count of 0, when ZO_QUERY_HISTORY_ENABLED is false.", security( ("Authorization"= []) ), @@ -155,7 +227,8 @@ pub async fn record( QueryHistoryListParams, ), responses( - (status = 200, description = "Success", content_type = "application/json", body = Vec), + (status = 200, description = "The entries, or with count_only=true the count", content_type = "application/json", body = QueryHistoryListResponse), + (status = 400, description = "Unknown source or status, or an offset above 9223372036854775807", content_type = "application/json", body = ()), (status = 500, description = "Failure", content_type = "application/json", body = ()), ), extensions( @@ -177,6 +250,49 @@ pub async fn list( .await } +/// ClearQueryHistory + +#[utoipa::path( + delete, + path = "/{org_id}/query_history", + context_path = "/api", + tag = "Query History", + operation_id = "ClearQueryHistory", + summary = "Clear the caller's query history", + description = "Deletes the caller's query history entries for the given sources, or all sources. Starred entries are kept unless include_starred is true.", + security( + ("Authorization"= []) + ), + params( + ("org_id" = String, Path, description = "Organization name"), + QueryHistoryClearParams, + ), + responses( + (status = 200, description = "Success", content_type = "application/json", body = QueryHistoryClearResponse), + (status = 400, description = "Unknown source", content_type = "application/json", body = ()), + (status = 500, description = "Failure", content_type = "application/json", body = ()), + ), + extensions( + ("x-o2-ratelimit" = json!({"module": "Query History", "operation": "delete"})), + ("x-o2-mcp" = json!({"enabled": false})) + ) +)] +pub async fn clear( + Path(org_id): Path, + Headers(user_email): Headers, + Query(params): Query, +) -> Response { + let sources = match parse_sources(params.source.as_deref()) { + Ok(sources) => sources, + Err(e) => return MetaHttpResponse::bad_request(e), + }; + let include_starred = params.include_starred.unwrap_or(false); + match query_history::clear(&org_id, &user_email.user_id, &sources, include_starred).await { + Ok(deleted) => MetaHttpResponse::json(QueryHistoryClearResponse { deleted }), + Err(e) => MetaHttpResponse::internal_error(e), + } +} + /// StarQueryHistory #[utoipa::path( @@ -186,7 +302,7 @@ pub async fn list( tag = "Query History", operation_id = "StarQueryHistory", summary = "Star or unstar a query history entry", - description = "Starred entries are kept until deleted. A user can star at most 200 entries per organization.", + description = "Starred entries are kept until deleted. A user can star at most 200 entries per organization, shared across every source (logs, traces, metrics, dbm, profiles).", security( ("Authorization"= []) ), @@ -213,10 +329,10 @@ pub async fn star( Json(req): Json, ) -> Response { match query_history::set_starred(&org_id, &user_email.user_id, &id, req.starred).await { - Ok(SetStarred::Updated(entry)) => MetaHttpResponse::json(QueryHistoryEntry::from(entry)), + Ok(SetStarred::Updated(entry)) => MetaHttpResponse::json(QueryHistoryEntry::from(*entry)), Ok(SetStarred::NotFound) => MetaHttpResponse::not_found("query history entry not found"), Ok(SetStarred::StarredCapReached) => MetaHttpResponse::bad_request(format!( - "at most {MAX_STARRED_PER_USER} query history entries can be starred; unstar one first" + "at most {MAX_STARRED_PER_USER} query history entries can be starred across all sources; unstar one first" )), Err(e) => MetaHttpResponse::internal_error(e), } @@ -260,6 +376,81 @@ pub async fn delete( } } +fn list_limit(limit: Option) -> u64 { + limit.unwrap_or(DEFAULT_LIMIT).clamp(1, MAX_LIMIT) +} + +/// The databases bind OFFSET as a signed 64-bit integer. +fn list_offset(offset: Option) -> Result { + let offset = offset.unwrap_or(0); + if offset > i64::MAX as u64 { + return Err(format!("offset exceeds {}", i64::MAX)); + } + Ok(offset) +} + +/// Splits a comma list of sources; empty means all, an unknown name is an error. +fn parse_sources(sources: Option<&str>) -> Result, String> { + let mut parsed = Vec::new(); + for source in sources.unwrap_or("").split(',').map(str::trim) { + if source.is_empty() { + continue; + } + if !SOURCES.contains(&source) { + return Err(format!("unknown source: {source}")); + } + parsed.push(source.to_string()); + } + Ok(parsed) +} + +fn parse_status(status: Option<&str>) -> Result, String> { + match status.map(str::trim).filter(|s| !s.is_empty()) { + None => Ok(None), + Some(s) if s == STATUS_OK || s == STATUS_ERROR => Ok(Some(s.to_string())), + Some(s) => Err(format!("unknown status: {s}")), + } +} + +/// The source and status a record stores, or why the request is refused. +fn validate_record(req: &QueryHistoryRequest) -> Result<(String, String), String> { + let source = req + .source + .as_deref() + .map(str::trim) + .filter(|s| !s.is_empty()) + .unwrap_or(DEFAULT_SOURCE); + if !SOURCES.contains(&source) { + return Err(format!("unknown source: {source}")); + } + let status = parse_status(req.status.as_deref())?.unwrap_or_else(|| STATUS_OK.to_string()); + if req.query.trim().is_empty() { + return Err("query is empty".to_string()); + } + if req.query.len() > MAX_QUERY_BYTES { + return Err("query exceeds 16 KB".to_string()); + } + if req.context.to_string().len() > MAX_CONTEXT_BYTES { + return Err("context exceeds 64 KB".to_string()); + } + if req.streams.len() > MAX_STREAMS { + return Err(format!("at most {MAX_STREAMS} streams")); + } + let invalid: Vec = req + .streams + .iter() + .filter(|s| s.is_empty() || s.contains(',')) + .map(|s| format!("{s:?}")) + .collect(); + if !invalid.is_empty() { + return Err(format!( + "stream names must be non-empty and contain no comma: {}", + invalid.join(", ") + )); + } + Ok((source.to_string(), status)) +} + async fn record_entry( enabled: bool, org_id: &str, @@ -269,24 +460,22 @@ async fn record_entry( if !enabled { return StatusCode::NO_CONTENT.into_response(); } - if req.query.trim().is_empty() { - return MetaHttpResponse::bad_request("query is empty"); - } - if req.query.len() > MAX_QUERY_BYTES { - return MetaHttpResponse::bad_request("query exceeds 16 KB"); - } - if req.context.to_string().len() > MAX_CONTEXT_BYTES { - return MetaHttpResponse::bad_request("context exceeds 64 KB"); - } - match query_history::record( + let (source, status) = match validate_record(&req) { + Ok(valid) => valid, + Err(e) => return MetaHttpResponse::bad_request(e), + }; + let run = NewRun { org_id, user_email, - &req.query, - req.context, - config::utils::time::now_micros(), - ) - .await - { + source: &source, + query: &req.query, + query_key: req.query_key.as_deref(), + streams: &req.streams, + status: &status, + context: req.context, + now_us: config::utils::time::now_micros(), + }; + match query_history::record(run).await { Ok(entry) => MetaHttpResponse::json(QueryHistoryEntry::from(entry)), Err(e) => MetaHttpResponse::internal_error(e), } @@ -298,45 +487,81 @@ async fn list_entries( user_email: &str, params: QueryHistoryListParams, ) -> Response { + let count_only = params.count_only.unwrap_or(false); if !enabled { - return MetaHttpResponse::json(Vec::::new()); + return MetaHttpResponse::json(if count_only { + QueryHistoryListResponse::Count(QueryHistoryCountResponse { count: 0 }) + } else { + QueryHistoryListResponse::Entries(Vec::new()) + }); + } + let (filter, offset) = + match list_filter(¶ms).and_then(|f| Ok((f, list_offset(params.offset)?))) { + Ok(valid) => valid, + Err(e) => return MetaHttpResponse::bad_request(e), + }; + if count_only { + return match query_history::count(org_id, user_email, &filter).await { + Ok(count) => { + MetaHttpResponse::json(QueryHistoryListResponse::Count(QueryHistoryCountResponse { + count, + })) + } + Err(e) => MetaHttpResponse::internal_error(e), + }; } match query_history::list( org_id, user_email, - params.starred, - params.q.as_deref(), + &filter, list_limit(params.limit), - params.offset.unwrap_or(0), + offset, ) .await { - Ok(entries) => MetaHttpResponse::json( - entries - .into_iter() - .map(QueryHistoryEntry::from) - .collect::>(), - ), + Ok(entries) => MetaHttpResponse::json(QueryHistoryListResponse::Entries( + entries.into_iter().map(QueryHistoryEntry::from).collect(), + )), Err(e) => MetaHttpResponse::internal_error(e), } } +fn list_filter(params: &QueryHistoryListParams) -> Result { + // A v1.1.x client sends no source and can only load Metrics entries. + let sources = parse_sources(params.source.as_deref())?; + let sources = if sources.is_empty() { + vec![DEFAULT_SOURCE.to_string()] + } else { + sources + }; + Ok(ListFilter { + sources, + status: parse_status(params.status.as_deref())?, + stream: params.stream.clone(), + q: params.q.clone(), + starred: params.starred, + ran_from: params.ran_from, + ran_to: params.ran_to, + }) +} + #[cfg(test)] mod tests { - use infra::table::entity::query_history::Entity; - use sea_orm::{ConnectionTrait, Schema}; use serde_json::{Value, json}; use super::*; + const ALL: &str = "source=logs,traces,metrics,dbm,profiles"; + async fn setup() { - infra::db::create_table().await.unwrap(); - let conn = infra::db::get_orm_client_rw().await; - let backend = conn.get_database_backend(); - let mut stmt = Schema::new(backend).create_table_from_entity(Entity); - conn.execute(backend.build(stmt.if_not_exists())) - .await - .unwrap(); + static TABLE: tokio::sync::OnceCell<()> = tokio::sync::OnceCell::const_new(); + TABLE + .get_or_init(|| async { + infra::db::create_table().await.unwrap(); + let conn = infra::db::get_orm_client_rw().await; + query_history::create_table_for_tests(conn).await.unwrap(); + }) + .await; } fn user() -> String { @@ -349,6 +574,17 @@ mod tests { }) } + fn request(query: &str, context: Value) -> QueryHistoryRequest { + QueryHistoryRequest { + source: None, + query: query.to_string(), + query_key: None, + streams: Vec::new(), + status: None, + context, + } + } + async fn parts(resp: Response) -> (StatusCode, Value) { let status = resp.status(); let bytes = axum::body::to_bytes(resp.into_body(), usize::MAX) @@ -360,23 +596,26 @@ mod tests { ) } - async fn post(org: &str, user: &str, query: &str, context: Value) -> (StatusCode, Value) { - let req = QueryHistoryRequest { - query: query.to_string(), - context, - }; + async fn post_req(org: &str, user: &str, req: QueryHistoryRequest) -> (StatusCode, Value) { parts(record(Path(org.to_string()), headers(user), Json(req)).await).await } + async fn post(org: &str, user: &str, query: &str, context: Value) -> (StatusCode, Value) { + post_req(org, user, request(query, context)).await + } + async fn get(org: &str, user: &str, params: &str) -> Vec { - let params: QueryHistoryListParams = parse_params(params); - let (status, body) = - parts(list(Path(org.to_string()), headers(user), Query(params)).await).await; - assert_eq!(status, StatusCode::OK); + let (status, body) = get_raw(org, user, params).await; + assert_eq!(status, StatusCode::OK, "{body}"); body.as_array().unwrap().clone() } - fn parse_params(params: &str) -> QueryHistoryListParams { + async fn get_raw(org: &str, user: &str, params: &str) -> (StatusCode, Value) { + let params: QueryHistoryListParams = parse_params(params); + parts(list(Path(org.to_string()), headers(user), Query(params)).await).await + } + + fn parse_params(params: &str) -> T { let uri: axum::http::Uri = format!("/?{params}").parse().unwrap(); Query::try_from_uri(&uri).unwrap().0 } @@ -392,6 +631,17 @@ mod tests { parts(delete(path, headers(user)).await).await.0 } + async fn clear_all(org: &str, user: &str, params: &str) -> (StatusCode, Value) { + let params: QueryHistoryClearParams = parse_params(params); + parts(clear(Path(org.to_string()), headers(user), Query(params)).await).await + } + + fn queries(rows: Vec) -> Vec { + rows.iter() + .map(|e| e["query"].as_str().unwrap().to_string()) + .collect() + } + #[tokio::test] async fn test_another_users_entry_is_not_found() { setup().await; @@ -438,73 +688,317 @@ mod tests { } #[tokio::test] - async fn test_rerun_updates_the_latest_entry_context() { + async fn test_an_old_client_without_source_or_key_merges_by_query_into_metrics() { setup().await; let a = user(); post("default", &a, "up", json!({"chart": "line"})).await; - post("default", &a, "up", json!({"chart": "bar"})).await; + let (_, entry) = post("default", &a, "up", json!({"chart": "bar"})).await; + assert_eq!(entry["source"], "metrics"); + assert_eq!(entry["status"], "ok"); + assert_eq!(entry["run_count"], 2); + assert_eq!(entry["streams"], json!([])); + assert_eq!(entry["query_key"].as_str().unwrap().len(), 64); let listed = get("default", &a, "").await; assert_eq!(listed.len(), 1); assert_eq!(listed[0]["context"]["chart"], "bar"); + assert!(listed[0]["updated_at"].as_i64() >= listed[0]["created_at"].as_i64()); } #[tokio::test] - async fn test_oversized_query_or_context_is_rejected() { + async fn test_a_v1_1_row_without_updated_at_answers_its_created_at() { + use sea_orm::ConnectionTrait; setup().await; let a = user(); - let long_query = "x".repeat(MAX_QUERY_BYTES + 1); + let id = config::ider::uuid(); + infra::db::get_orm_client_rw() + .await + .execute_unprepared(&format!( + "INSERT INTO query_history (id, org_id, user_email, query, context, created_at, \ + updated_at, source) VALUES ('{id}', 'default', '{a}', 'up', '{{}}', 500, 0, \ + 'metrics')" + )) + .await + .unwrap(); + + let listed = get("default", &a, "").await; assert_eq!( - post("default", &a, &long_query, json!({})).await.0, - StatusCode::BAD_REQUEST + ( + listed[0]["created_at"].as_i64(), + listed[0]["updated_at"].as_i64() + ), + (Some(500), Some(500)) ); - let big = json!({"d": "x".repeat(MAX_CONTEXT_BYTES)}); - assert_eq!( - post("default", &a, "up", big).await.0, - StatusCode::BAD_REQUEST - ); - let max_query = "x".repeat(MAX_QUERY_BYTES); - assert_eq!( - post("default", &a, &max_query, json!({})).await.0, - StatusCode::OK - ); - assert_eq!(get("default", &a, "").await.len(), 1); + let (status, starred) = patch("default", &a, &id, true).await; + assert_eq!(status, StatusCode::OK); + assert_eq!(starred["updated_at"], 500); + assert_eq!(starred["created_at"], 500); } #[tokio::test] - async fn test_blank_query_is_rejected() { + async fn test_repeats_of_one_search_stay_one_entry() { setup().await; let a = user(); - for query in ["", " ", "\n\t "] { + for _ in 0..3 { + let req = QueryHistoryRequest { + source: Some("logs".to_string()), + query_key: Some("k8s|sql|up|".to_string()), + streams: vec!["k8s".to_string()], + ..request("up", json!({})) + }; + assert_eq!(post_req("default", &a, req).await.0, StatusCode::OK); + } + let rows = get("default", &a, "source=logs").await; + assert_eq!(rows.len(), 1); + assert_eq!(rows[0]["run_count"], 3); + assert_eq!(rows[0]["streams"], json!(["k8s"])); + } + + #[tokio::test] + async fn test_query_key_is_a_hash_per_user_and_falls_back_to_the_query() { + setup().await; + let (a, b) = (user(), user()); + let keyed = |key: Option<&str>| QueryHistoryRequest { + source: Some("logs".to_string()), + query_key: key.map(str::to_string), + ..request("up", json!({})) + }; + let (_, first) = post_req("default", &a, keyed(Some("k8s|sql|up|"))).await; + let key = first["query_key"].as_str().unwrap(); + assert_eq!(key.len(), 64); + assert!(key.chars().all(|c| c.is_ascii_hexdigit()), "{key}"); + assert_ne!(key, "k8s|sql|up|"); + + let (_, other_user) = post_req("default", &b, keyed(Some("k8s|sql|up|"))).await; + assert_eq!(other_user["query_key"], first["query_key"]); + assert_ne!(other_user["id"], first["id"], "user_email is in the key"); + assert_eq!(other_user["run_count"], 1); + + let (_, no_key) = post_req("default", &a, keyed(None)).await; + let (_, query_as_key) = post_req("default", &a, keyed(Some("up"))).await; + assert_eq!(query_as_key["id"], no_key["id"], "no key hashes the query"); + assert_eq!(query_as_key["run_count"], 2); + assert_eq!(get("default", &a, "source=logs").await.len(), 2); + } + + #[test] + fn test_invalid_stream_names_are_listed_in_the_error() { + let req = QueryHistoryRequest { + streams: vec!["ok".to_string(), "a,b".to_string(), String::new()], + ..request("up", json!({})) + }; + assert_eq!( + validate_record(&req).unwrap_err(), + r#"stream names must be non-empty and contain no comma: "a,b", """# + ); + } + + #[tokio::test] + async fn test_record_validation() { + setup().await; + let a = user(); + let refused = [ + QueryHistoryRequest { + source: Some("dashboards".to_string()), + ..request("up", json!({})) + }, + QueryHistoryRequest { + status: Some("timeout".to_string()), + ..request("up", json!({})) + }, + QueryHistoryRequest { + streams: (0..=MAX_STREAMS).map(|i| format!("s{i}")).collect(), + ..request("up", json!({})) + }, + QueryHistoryRequest { + streams: vec!["a,b".to_string()], + ..request("up", json!({})) + }, + request(&"x".repeat(MAX_QUERY_BYTES + 1), json!({})), + request("up", json!({"d": "x".repeat(MAX_CONTEXT_BYTES)})), + request("", json!({})), + request(" \n\t ", json!({})), + ]; + for req in refused { assert_eq!( - post("default", &a, query, json!({})).await.0, - StatusCode::BAD_REQUEST, - "{query:?}" + post_req("default", &a, req).await.0, + StatusCode::BAD_REQUEST ); } assert!(get("default", &a, "").await.is_empty()); + + let accepted = QueryHistoryRequest { + source: Some("traces".to_string()), + status: Some("error".to_string()), + streams: (0..MAX_STREAMS).map(|i| format!("s{i}")).collect(), + ..request(&"x".repeat(MAX_QUERY_BYTES), json!({})) + }; + let (status, entry) = post_req("default", &a, accepted).await; + assert_eq!(status, StatusCode::OK, "{entry}"); + assert_eq!(entry["status"], "error"); + assert_eq!(entry["source"], "traces"); } #[tokio::test] async fn test_list_applies_filters() { setup().await; let a = user(); - for query in ["up", "rate(http_total[5m])", "sum(http_total)"] { - post("default", &a, query, json!({})).await; + for (source, query, stream, status) in [ + ("metrics", "up", None, "ok"), + ("logs", "level = 'error'", Some("k8s_logs"), "error"), + ("traces", "service = 'cart'", Some("default"), "ok"), + ("metrics", "sum(http_total)", None, "ok"), + ] { + let req = QueryHistoryRequest { + source: Some(source.to_string()), + streams: stream.map(|s| vec![s.to_string()]).unwrap_or_default(), + status: Some(status.to_string()), + ..request(query, json!({})) + }; + post_req("default", &a, req).await; } - let queries = |rows: Vec| { - rows.iter() - .map(|e| e["query"].as_str().unwrap().to_string()) - .collect::>() - }; assert_eq!( - queries(get("default", &a, "q=http_total&limit=1").await), + queries(get("default", &a, "source=logs,traces").await), + vec!["service = 'cart'", "level = 'error'"] + ); + assert_eq!( + queries(get("default", &a, &format!("{ALL}&status=error")).await), + vec!["level = 'error'"] + ); + assert_eq!( + queries(get("default", &a, &format!("{ALL}&stream=k8s_logs")).await), + vec!["level = 'error'"] + ); + assert_eq!( + queries(get("default", &a, &format!("{ALL}&q=K8S")).await), + vec!["level = 'error'"] + ); + assert_eq!( + queries(get("default", &a, "").await), + vec!["sum(http_total)", "up"], + "a client that sends no source is a v1.1.x Metrics page" + ); + assert_eq!( + queries(get("default", &a, "source=metrics&limit=1").await), vec!["sum(http_total)"] ); assert_eq!( - queries(get("default", &a, "q=http_total&offset=1").await), - vec!["rate(http_total[5m])"] + queries(get("default", &a, "source=metrics&offset=1").await), + vec!["up"] + ); + assert!( + get("default", &a, &format!("{ALL}&ran_from=1&ran_to=2")) + .await + .is_empty() + ); + assert_eq!( + get("default", &a, &format!("{ALL}&ran_from=1")).await.len(), + 4 + ); + assert!( + get("default", &a, &format!("{ALL}&starred=true")) + .await + .is_empty() + ); + assert_eq!( + get_raw("default", &a, "source=dashboards").await.0, + StatusCode::BAD_REQUEST + ); + assert_eq!( + get_raw("default", &a, "status=partial").await.0, + StatusCode::BAD_REQUEST + ); + } + + #[tokio::test] + async fn test_count_only_answers_the_count_for_the_filters() { + setup().await; + let a = user(); + for (source, query) in [ + ("logs", "a"), + ("logs", "b"), + ("traces", "c"), + ("metrics", "d"), + ] { + let req = QueryHistoryRequest { + source: Some(source.to_string()), + ..request(query, json!({})) + }; + post_req("default", &a, req).await; + } + let starred = get("default", &a, "source=logs").await[0]["id"] + .as_str() + .unwrap() + .to_string(); + patch("default", &a, &starred, true).await; + let count = |params: &str| { + let a = a.clone(); + let params = format!("count_only=true&{params}"); + async move { get_raw("default", &a, ¶ms).await } + }; + assert_eq!( + count("source=logs").await, + (StatusCode::OK, json!({"count": 2})) + ); + assert_eq!( + count("source=logs&starred=false").await.1, + json!({"count": 1}) + ); + assert_eq!( + count(&format!("{ALL}&starred=true")).await.1, + json!({"count": 1}) + ); + assert_eq!(count(ALL).await.1, json!({"count": 4})); + assert_eq!(count("source=logs&limit=1").await.1, json!({"count": 2})); + assert_eq!(count("source=nope").await.0, StatusCode::BAD_REQUEST); + + let disabled = list_entries( + false, + "default", + &a, + parse_params("count_only=true&source=logs"), + ) + .await; + assert_eq!(parts(disabled).await, (StatusCode::OK, json!({"count": 0}))); + } + + #[tokio::test] + async fn test_clear_my_history() { + setup().await; + let (a, b) = (user(), user()); + for (source, query) in [ + ("logs", "a"), + ("logs", "b"), + ("traces", "c"), + ("metrics", "d"), + ] { + let req = QueryHistoryRequest { + source: Some(source.to_string()), + ..request(query, json!({})) + }; + post_req("default", &a, req).await; + } + post("default", &b, "up", json!({})).await; + let starred = get("default", &a, "source=logs").await[0]["id"] + .as_str() + .unwrap() + .to_string(); + patch("default", &a, &starred, true).await; + + let (status, body) = clear_all("default", &a, "source=logs").await; + assert_eq!(status, StatusCode::OK); + assert_eq!(body, json!({"deleted": 1})); + assert_eq!(clear_all("default", &a, "").await.1, json!({"deleted": 2})); + assert_eq!(queries(get("default", &a, ALL).await), vec!["b"]); + assert_eq!( + clear_all("default", &a, "include_starred=true").await.1, + json!({"deleted": 1}) + ); + assert!(get("default", &a, ALL).await.is_empty()); + assert_eq!(get("default", &b, ALL).await.len(), 1); + assert_eq!( + clear_all("default", &a, "source=nope").await.0, + StatusCode::BAD_REQUEST ); - assert!(get("default", &a, "starred=true").await.is_empty()); } #[tokio::test] @@ -512,10 +1006,7 @@ mod tests { setup().await; let a = user(); post("default", &a, "up", json!({})).await; - let req = QueryHistoryRequest { - query: "rate(x[5m])".to_string(), - context: json!({}), - }; + let req = request("rate(x[5m])", json!({})); let (status, body) = parts(record_entry(false, "default", &a, req).await).await; assert_eq!(status, StatusCode::NO_CONTENT); assert_eq!(body, Value::Null, "204 carries no body"); @@ -535,4 +1026,37 @@ mod tests { assert_eq!(list_limit(Some(500)), 200); assert_eq!(list_limit(Some(0)), 1); } + + #[tokio::test] + async fn test_an_offset_past_i64_max_is_a_bad_request() { + setup().await; + let a = user(); + post("default", &a, "up", json!({})).await; + let max = i64::MAX as u64; + let (status, _) = get_raw("default", &a, &format!("offset={}", max + 1)).await; + assert_eq!(status, StatusCode::BAD_REQUEST); + let (status, _) = get_raw( + "default", + &a, + &format!("offset={}&count_only=true", u64::MAX), + ) + .await; + assert_eq!(status, StatusCode::BAD_REQUEST); + assert!( + get("default", &a, &format!("offset={max}")) + .await + .is_empty() + ); + assert_eq!(list_offset(None), Ok(0)); + } + + #[test] + fn test_sources_parse_as_a_comma_list_of_known_pages() { + assert_eq!(parse_sources(None), Ok(vec![])); + assert_eq!( + parse_sources(Some("logs, traces,")), + Ok(vec!["logs".to_string(), "traces".to_string()]) + ); + assert!(parse_sources(Some("logs,alerts")).is_err()); + } } diff --git a/src/api/management/src/request/status/mod.rs b/src/api/management/src/request/status/mod.rs index c96ae67ab5..5431120d60 100644 --- a/src/api/management/src/request/status/mod.rs +++ b/src/api/management/src/request/status/mod.rs @@ -249,6 +249,10 @@ struct ConfigResponse<'a> { synthetics_recorder_extension_url: String, /// The only DBM flag (`ZO_DB_MONITORING_ENABLED`): one OSS switch for every signal. database_monitoring_enabled: bool, + /// Off hides recent searches in the UI; the server then records nothing. + query_history_enabled: bool, + /// Days unstarred recent searches are kept; 0 or less keeps them until the per-page cap. + query_history_retention_days: i64, enable_cross_linking: bool, show_fts_field_values: bool, search_inspector_enabled: bool, @@ -668,6 +672,8 @@ pub async fn zo_config( synthetics_browser_max_steps, synthetics_recorder_extension_url: synthetics_recorder_extension_url.to_string(), database_monitoring_enabled: cfg.db_monitoring.enabled, + query_history_enabled: cfg.limit.query_history_enabled, + query_history_retention_days: cfg.limit.query_history_retention_days, enable_cross_linking: cfg.common.enable_cross_linking, show_fts_field_values: cfg.common.show_fts_field_values, search_inspector_enabled, @@ -1286,11 +1292,11 @@ pub async fn redirect( Ok(v) => v.map(|(new_user, pending_invites)| { format!("&new_user_login={new_user}&pending_invites={pending_invites}") }), - Err(_) => { + Err(e) => { return Response::builder() .status(StatusCode::UNAUTHORIZED) .header(header::CONTENT_TYPE, "application/json") - .body(Body::from("\"Email Domain not allowed\"")) + .body(Body::from(e.to_string())) .unwrap(); } }; @@ -2563,6 +2569,20 @@ mod tests { assert_eq!(json["public_dashboards_enabled"], true); } + #[tokio::test] + async fn test_zo_config_serves_query_history_settings() { + let cfg = get_config(); + let json = zo_config_json().await; + assert_eq!( + json["query_history_enabled"], + cfg.limit.query_history_enabled + ); + assert_eq!( + json["query_history_retention_days"], + cfg.limit.query_history_retention_days + ); + } + #[tokio::test] async fn test_zo_config_reports_slack_per_org() { let reason = if cfg!(feature = "enterprise") { diff --git a/src/common/src/meta/organization.rs b/src/common/src/meta/organization.rs index 6413cbcfde..04c11f4b97 100644 --- a/src/common/src/meta/organization.rs +++ b/src/common/src/meta/organization.rs @@ -14,6 +14,8 @@ // along with this program. If not, see . use config::{meta::user::UserRole, stats::MemorySize}; +#[cfg(feature = "cloud")] +use o2_enterprise::enterprise::domain_management::meta::DomainManagementConfig; use serde::{Deserialize, Serialize}; use utoipa::ToSchema; @@ -539,8 +541,16 @@ pub struct OrganizationSettingPayload { pub claim_parser_function: Option, #[serde(skip_serializing_if = "Option::is_none")] pub cross_links: Option>, + #[cfg(feature = "cloud")] #[serde(skip_serializing_if = "Option::is_none")] pub domain_org_mappings: Option>, + #[cfg(feature = "cloud")] + #[serde(default)] + pub domain_management_config: Option, + #[cfg(feature = "cloud")] + #[serde(default)] + pub enforce_allowed_orgs_only: Option, + #[cfg(feature = "cloud")] #[serde(skip_serializing_if = "Option::is_none")] pub budget_config: Option>, } @@ -592,6 +602,12 @@ pub struct OrganizationSetting { pub domain_org_mappings: Vec, #[cfg(feature = "cloud")] #[serde(default)] + pub enforce_allowed_orgs_only: bool, + #[cfg(feature = "cloud")] + #[serde(default)] + pub domain_management_config: DomainManagementConfig, + #[cfg(feature = "cloud")] + #[serde(default)] pub budget_config: Vec, } @@ -631,6 +647,10 @@ impl Default for OrganizationSetting { #[cfg(feature = "cloud")] domain_org_mappings: Vec::new(), #[cfg(feature = "cloud")] + domain_management_config: Default::default(), + #[cfg(feature = "cloud")] + enforce_allowed_orgs_only: false, + #[cfg(feature = "cloud")] budget_config: Vec::new(), } } @@ -1474,7 +1494,13 @@ mod tests { #[cfg(feature = "enterprise")] claim_parser_function: None, cross_links: None, + #[cfg(feature = "cloud")] domain_org_mappings: None, + #[cfg(feature = "cloud")] + domain_management_config: None, + #[cfg(feature = "cloud")] + enforce_allowed_orgs_only: None, + #[cfg(feature = "cloud")] budget_config: None, }; let json = serde_json::to_value(&payload).unwrap(); @@ -1558,6 +1584,10 @@ mod tests { #[cfg(feature = "cloud")] domain_org_mappings: vec![], #[cfg(feature = "cloud")] + domain_management_config: Default::default(), + #[cfg(feature = "cloud")] + enforce_allowed_orgs_only: false, + #[cfg(feature = "cloud")] budget_config: vec![], }; let json = serde_json::to_value(&setting).unwrap(); @@ -1592,6 +1622,10 @@ mod tests { #[cfg(feature = "cloud")] domain_org_mappings: vec![], #[cfg(feature = "cloud")] + domain_management_config: Default::default(), + #[cfg(feature = "cloud")] + enforce_allowed_orgs_only: false, + #[cfg(feature = "cloud")] budget_config: vec![], }; let json = serde_json::to_value(&setting).unwrap(); diff --git a/src/config/src/config.rs b/src/config/src/config.rs index 5a8a766c06..7c3f9ad8df 100644 --- a/src/config/src/config.rs +++ b/src/config/src/config.rs @@ -126,7 +126,10 @@ pub type RwBTreeMap = tokio::sync::RwLock>; // 126: add held_action to slack_link_requests. // 127: add group_label_set to alert_states. // 128: widen alert_dedup_state.fingerprint to varchar(256). -pub const DB_SCHEMA_VERSION: u64 = 128; +// 129: add source, query_key, streams, run_count, updated_at and status to query_history. +// 130: add suppress_repeats to alerts. +// 131: org domain mapping migrations +pub const DB_SCHEMA_VERSION: u64 = 131; pub const DB_SCHEMA_KEY: &str = "/db_schema_version/"; // global version variables @@ -526,6 +529,11 @@ pub const SYNTHETICS_RELOAD_CLASSES: &[(&str, SyntheticsReloadClass)] = &[ "ZO_SYNTHETICS_ORPHAN_DETECTION_ENABLED", SyntheticsReloadClass::Hot, ), + ("ZO_SYNTHETICS_LEASE_BATCH_SIZE", SyntheticsReloadClass::Hot), + ( + "ZO_SYNTHETICS_MAX_INFLIGHT_PER_TICK", + SyntheticsReloadClass::Hot, + ), // Were `HotViaLimits` while these lived in `o2_enterprise` and had to be // pushed across a seam into this crate. They are declared here now, so the // seam and its class are both gone and they are plainly `Hot`. @@ -580,6 +588,8 @@ pub(crate) fn synthetics_restart_required_changes( devices: _, scheduler_jitter_enabled: _, orphan_detection_enabled: _, + lease_batch_size: _, + max_inflight_per_tick: _, } = new; let mut changed = Vec::new(); @@ -1126,14 +1136,14 @@ impl PublicAiChat { } } -/// Background self CPU/memory profile ingest into `_meta.self_profiles`. -/// Gated at runtime by `enabled`; sampling code is compile-gated on `profiling`. +/// Background self CPU/memory profile ingest into the `default` profiles stream. +/// Gated at runtime by `enabled`; CPU needs `cpu-profiling`, memory needs `profiling`. #[derive(Debug, Serialize, EnvConfig, Default)] pub struct SelfProfiles { #[env_config( name = "ZO_SELF_PROFILES_ENABLED", default = false, - help = "Enable background self CPU/memory profile sampling into _meta.self_profiles" + help = "Enable background self CPU/memory profile sampling into the default profiles stream" )] pub enabled: bool, #[env_config( @@ -1493,6 +1503,23 @@ pub struct Synthetics { help = "Kill switch for orphan detection, which reports enabled checks no scheduler is claiming." )] pub orphan_detection_enabled: bool, + /// Jobs leased per pool in one dispatcher lease call. Must be > 0; a + /// non-positive value falls back to the default. Read per tick. + #[env_config( + name = "ZO_SYNTHETICS_LEASE_BATCH_SIZE", + default = 10, + help = "Jobs the dispatcher leases per pool per lease call. Must be greater than 0." + )] + pub lease_batch_size: i64, + /// Upper bound on jobs the dispatcher leases across all pools in one tick. + /// While a pool's batch comes back full the dispatcher keeps leasing, up to + /// this cap. Must be > 0; a non-positive value falls back to the default. + #[env_config( + name = "ZO_SYNTHETICS_MAX_INFLIGHT_PER_TICK", + default = 100, + help = "Cap on jobs the dispatcher leases across all pools in one 2s tick. Must be greater than 0." + )] + pub max_inflight_per_tick: i64, } /// Feature 5 — SLO measurement (`alerts_2.md` §6b). @@ -2911,7 +2938,7 @@ pub struct Limit { #[env_config( name = "ZO_QUERY_HISTORY_RETENTION_DAYS", default = 14, - help = "How long unstarred query history entries are kept, in days. Starred entries are kept until deleted. 0 or less keeps every entry forever; it does not stop collection, ZO_QUERY_HISTORY_ENABLED=false does." + help = "How long unstarred query history entries are kept, in days. Starred entries are kept until deleted. 0 or less removes the age limit; each user still keeps at most 500 unstarred entries per source (logs, traces, metrics, dbm, profiles), the oldest of that source pruned first. It does not stop collection, ZO_QUERY_HISTORY_ENABLED=false does." )] pub query_history_retention_days: i64, #[env_config(name = "ZO_ALERT_SCHEDULE_TIMEOUT", default = 90)] // seconds @@ -5270,6 +5297,21 @@ fn check_synthetics_config(cfg: &mut Config) { cfg.synthetics.max_net_timeout_ms = crate::meta::synthetics::DEFAULT_MAX_NET_TIMEOUT_MS; } + if cfg.synthetics.lease_batch_size <= 0 { + log::error!( + "ZO_SYNTHETICS_LEASE_BATCH_SIZE ({}) must be greater than 0, using 10", + cfg.synthetics.lease_batch_size + ); + cfg.synthetics.lease_batch_size = 10; + } + if cfg.synthetics.max_inflight_per_tick <= 0 { + log::error!( + "ZO_SYNTHETICS_MAX_INFLIGHT_PER_TICK ({}) must be greater than 0, using 100", + cfg.synthetics.max_inflight_per_tick + ); + cfg.synthetics.max_inflight_per_tick = 100; + } + // Every synthetics var now has exactly one name, in every build — which // means an OSS user can set a private-agent var and get silence back. One // line per var is the difference between "configured and ignored" and @@ -5450,6 +5492,8 @@ mod tests { "ZO_SYNTHETICS_DEVICES", "ZO_SYNTHETICS_SCHEDULER_JITTER_ENABLED", "ZO_SYNTHETICS_ORPHAN_DETECTION_ENABLED", + "ZO_SYNTHETICS_LEASE_BATCH_SIZE", + "ZO_SYNTHETICS_MAX_INFLIGHT_PER_TICK", ]; fn synthetics_names_in_class(class: SyntheticsReloadClass) -> Vec<&'static str> { @@ -5469,8 +5513,8 @@ mod tests { fn synthetics_reload_classification_is_pinned() { assert_eq!( SYNTHETICS_RELOAD_CLASSES.len(), - 16, - "Synthetics has 16 keys; every one needs a reload class" + 18, + "Synthetics has 18 keys; every one needs a reload class" ); let mut classified: Vec<&str> = SYNTHETICS_RELOAD_CLASSES @@ -5502,7 +5546,9 @@ mod tests { "ZO_SYNTHETICS_JOB_LEASE_SECS", "ZO_SYNTHETICS_LAMBDA_BROWSER", "ZO_SYNTHETICS_LAMBDA_NET", + "ZO_SYNTHETICS_LEASE_BATCH_SIZE", "ZO_SYNTHETICS_MAX_CHECK_BUDGET_SECS", + "ZO_SYNTHETICS_MAX_INFLIGHT_PER_TICK", "ZO_SYNTHETICS_MAX_NET_TIMEOUT_MS", "ZO_SYNTHETICS_ORPHAN_DETECTION_ENABLED", "ZO_SYNTHETICS_RECORDER_EXTENSION_URL", @@ -5565,6 +5611,8 @@ mod tests { cfg.devices = "desktop:800:600".to_string(); cfg.scheduler_jitter_enabled = !cfg.scheduler_jitter_enabled; cfg.orphan_detection_enabled = !cfg.orphan_detection_enabled; + cfg.lease_batch_size += 1; + cfg.max_inflight_per_tick += 1; } #[test] diff --git a/src/config/src/meta/alerts/alert.rs b/src/config/src/meta/alerts/alert.rs index 5e247c2ea2..44bef5ab8d 100644 --- a/src/config/src/meta/alerts/alert.rs +++ b/src/config/src/meta/alerts/alert.rs @@ -165,6 +165,10 @@ pub struct Alert { /// today. Named after Prometheus' `keep_firing_for`, with the same meaning. #[serde(default)] pub keep_firing_for: i64, + + /// Notify once per state change; enterprise only, other builds ignore it. + #[serde(default)] + pub suppress_repeats: bool, } /// Accept a runbook link, or say why not. @@ -266,11 +270,17 @@ impl Default for Alert { notify_on_recovery: false, recovery_destinations: vec![], keep_firing_for: 0, + suppress_repeats: false, } } } impl Alert { + /// Whether repeat notifications are withheld; other builds than enterprise ignore the switch. + pub fn suppresses_repeats(&self) -> bool { + cfg!(feature = "enterprise") && self.suppress_repeats + } + /// Pending rule mode needs per-series state; the default count gate means any series. pub fn normalize_promql_rule_mode(&mut self) { if self.query_condition.query_type != super::QueryType::PromQL @@ -1233,4 +1243,22 @@ mod tests { let json = serde_json::to_value(&alert).unwrap(); assert!(json.get("runbook_url").is_none()); } + + #[cfg(not(feature = "enterprise"))] + #[test] + fn repeat_suppression_is_ignored_outside_enterprise() { + let alert = Alert { + suppress_repeats: true, + ..Default::default() + }; + assert!(!alert.suppresses_repeats()); + } + + #[test] + fn an_alert_saved_before_repeat_suppression_reads_with_it_off() { + let mut stored = serde_json::to_value(Alert::default()).unwrap(); + stored.as_object_mut().unwrap().remove("suppress_repeats"); + let alert: Alert = serde_json::from_value(stored).unwrap(); + assert!(!alert.suppress_repeats); + } } diff --git a/src/config/src/meta/alerts/dispatch.rs b/src/config/src/meta/alerts/dispatch.rs index e4194bb629..e7ad7f4e5d 100644 --- a/src/config/src/meta/alerts/dispatch.rs +++ b/src/config/src/meta/alerts/dispatch.rs @@ -112,6 +112,8 @@ pub struct DispatchPlan { pub inconsistent: Vec, /// Groups that would deliver but a downtime mutes; also counted in `suppressed`. pub muted: Vec, + /// Groups withheld because they repeat their last delivery; also counted in `suppressed`. + pub repeats: usize, } /// Decide which groups this evaluation notifies (MN-1, MN-8). @@ -135,6 +137,7 @@ pub struct DispatchPlan { /// * `max_sends` is the MN-8 knob: `0` = unlimited; otherwise it counts **qualifying** sends /// (applied after `delivery_decision`), worst-first. /// * A deliverable group `muted(key, decision, last_notified_level)` holds goes to `muted`, first. +#[allow(clippy::too_many_arguments)] pub fn plan_dispatch( classification: &GroupClassification, states: &HashMap, @@ -142,6 +145,7 @@ pub fn plan_dispatch( tc: &TriggerCondition, now: i64, max_sends: usize, + suppress_repeats: bool, muted: impl Fn(&str, &DeliveryDecision, Option) -> bool, ) -> DispatchPlan { let mut items = Vec::new(); @@ -150,6 +154,7 @@ pub fn plan_dispatch( let mut dropped_by_knob = Vec::new(); let mut inconsistent = Vec::new(); let mut muted_keys = Vec::new(); + let mut repeats = 0usize; // A key appearing twice in one classification is an invariant violation // (validation rejects the known source, MN-12). Counting first means both @@ -218,6 +223,17 @@ pub fn plan_dispatch( muted_keys.push(key.clone()); continue; } + if suppress_repeats + && repeats_last_notification( + level, + state.episode_id.is_some(), + state.last_notified_level, + ) + { + suppressed += 1; + repeats += 1; + continue; + } // The knob counts QUALIFYING sends, so a silenced group never // spends budget a deliverable one could have used. if max_sends > 0 && items.len() >= max_sends { @@ -237,7 +253,8 @@ pub fn plan_dispatch( DeliveryDecision::SuppressedBySilence | DeliveryDecision::SuppressedByWarningPolicy | DeliveryDecision::SuppressedByDowntime(_) - | DeliveryDecision::SuppressedByAck(_) => suppressed += 1, + | DeliveryDecision::SuppressedByAck(_) + | DeliveryDecision::SuppressedAsRepeat => suppressed += 1, DeliveryDecision::SuppressedByPending => pending += 1, DeliveryDecision::NotFiring => {} } @@ -250,9 +267,19 @@ pub fn plan_dispatch( dropped_by_knob, inconsistent, muted: muted_keys, + repeats, } } +/// The open recovery episode was already notified at `level`, so this firing tells nothing new. +pub fn repeats_last_notification( + level: AlertLevel, + episode_open: bool, + last_notified: Option, +) -> bool { + episode_open && last_notified == Some(level) +} + /// The write that records one group's **successful** delivery (MN-6). /// /// Returns the full replacement row, or `None` when the carried **level-episode @@ -744,6 +771,7 @@ mod tests { &tc(None), 1_000 * SEC, 0, + false, |_, _, _| false, ); @@ -769,9 +797,16 @@ mod tests { let c = classified(&[("crit", 150.0), ("ok1", 1.0), ("ok2", 2.0)], 500); let states = states_for(&c, now, None, None); - let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| { - false - }); + let plan = plan_dispatch( + &c, + &states, + &rows_for(&c), + &tc(None), + now, + 0, + false, + |_, _, _| false, + ); assert_eq!(plan.items.len(), 1); assert_eq!( @@ -791,9 +826,16 @@ mod tests { let c = classified(&[("a", 150.0)], 500); let states = states_for(&c, now, None, None); - let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| { - false - }); + let plan = plan_dispatch( + &c, + &states, + &rows_for(&c), + &tc(None), + now, + 0, + false, + |_, _, _| false, + ); assert!( plan.items[0].escalation, @@ -816,6 +858,7 @@ mod tests { &tc(None), 1_000 * SEC, 0, + false, |_, _, _| false, ); let b = plan @@ -841,6 +884,7 @@ mod tests { &tc(None), 1_000 * SEC, 0, + false, |_, _, _| false, ); let keys: Vec<_> = plan.items.iter().map(|i| i.group_key.clone()).collect(); @@ -863,9 +907,16 @@ mod tests { let c = classified(&[("a", 150.0)], 500); let states = states_for(&c, now, Some(AlertLevel::Critical), Some(now + 10 * MIN)); - let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| { - false - }); + let plan = plan_dispatch( + &c, + &states, + &rows_for(&c), + &tc(None), + now, + 0, + false, + |_, _, _| false, + ); assert!(plan.items.is_empty()); assert_eq!(plan.suppressed, 1); @@ -879,9 +930,16 @@ mod tests { let c = classified(&[("a", 150.0)], 500); let states = states_for(&c, now, Some(AlertLevel::Critical), Some(now - 1)); - let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| { - false - }); + let plan = plan_dispatch( + &c, + &states, + &rows_for(&c), + &tc(None), + now, + 0, + false, + |_, _, _| false, + ); assert_eq!(plan.items.len(), 1); assert!( @@ -898,9 +956,16 @@ mod tests { let c = classified(&[("a", 150.0)], 500); let states = states_for(&c, now, Some(AlertLevel::Warning), Some(now + 10 * MIN)); - let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| { - false - }); + let plan = plan_dispatch( + &c, + &states, + &rows_for(&c), + &tc(None), + now, + 0, + false, + |_, _, _| false, + ); assert_eq!(plan.items.len(), 1); assert!(plan.items[0].escalation); @@ -933,9 +998,16 @@ mod tests { assert!(row.is_firing(), "NotifyFailed is still a firing state"); let states = HashMap::from([(key.clone(), row)]); - let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| { - false - }); + let plan = plan_dispatch( + &c, + &states, + &rows_for(&c), + &tc(None), + now, + 0, + false, + |_, _, _| false, + ); assert_eq!( plan.items.len(), @@ -958,6 +1030,7 @@ mod tests { &tc(Some(false)), 1_000 * SEC, 0, + false, |_, _, _| false, ); @@ -999,7 +1072,9 @@ mod tests { ]), ); - let plan = plan_dispatch(&c, &states, &rows, &tc(None), now, 0, |_, _, _| false); + let plan = plan_dispatch(&c, &states, &rows, &tc(None), now, 0, false, |_, _, _| { + false + }); assert_eq!(plan.items.len(), 2, "only retained groups dispatch"); assert!( @@ -1030,9 +1105,16 @@ mod tests { state(ROLLUP_GROUP_KEY, AlertLevel::Critical, now, None, None), ); - let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| { - false - }); + let plan = plan_dispatch( + &c, + &states, + &rows_for(&c), + &tc(None), + now, + 0, + false, + |_, _, _| false, + ); assert_eq!(plan.items.len(), 1, "one group fired, so one page"); assert_eq!(plan.items[0].group_key, key_of("a")); @@ -1060,9 +1142,16 @@ mod tests { state(&gone, AlertLevel::Critical, now - 5 * MIN, None, None), ); - let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| { - false - }); + let plan = plan_dispatch( + &c, + &states, + &rows_for(&c), + &tc(None), + now, + 0, + false, + |_, _, _| false, + ); assert!(!plan.items.iter().any(|i| i.group_key == gone)); } @@ -1084,6 +1173,7 @@ mod tests { &tc(None), now, 0, + false, |_, _, _| false, ); @@ -1111,6 +1201,7 @@ mod tests { &tc(None), now, 0, + false, |_, _, _| false, ); @@ -1139,9 +1230,16 @@ mod tests { assert_eq!(c.groups.len(), 3, "fixture: `a` is observed twice"); let states = states_for(&c, now, None, None); - let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| { - false - }); + let plan = plan_dispatch( + &c, + &states, + &rows_for(&c), + &tc(None), + now, + 0, + false, + |_, _, _| false, + ); assert!( !plan.items.iter().any(|i| i.group_key == key_of("a")), @@ -1170,9 +1268,16 @@ mod tests { assert_eq!(c.groups.len(), 3, "fixture: `a` observed at two levels"); let states = states_for(&c, now, None, None); - let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| { - false - }); + let plan = plan_dispatch( + &c, + &states, + &rows_for(&c), + &tc(None), + now, + 0, + false, + |_, _, _| false, + ); for item in &plan.items { assert_ne!( @@ -1202,9 +1307,16 @@ mod tests { ); let states = states_for(&c, now, None, None); - let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 2, |_, _, _| { - false - }); + let plan = plan_dispatch( + &c, + &states, + &rows_for(&c), + &tc(None), + now, + 2, + false, + |_, _, _| false, + ); assert_eq!(plan.items.len(), 2, "`b` and `c` both page"); assert!(plan.dropped_by_knob.is_empty()); @@ -1219,7 +1331,9 @@ mod tests { let states = states_for(&c, now, None, None); let rows = rows_for(&c); - let plan = plan_dispatch(&c, &states, &rows, &tc(None), now, 0, |_, _, _| false); + let plan = plan_dispatch(&c, &states, &rows, &tc(None), now, 0, false, |_, _, _| { + false + }); assert_eq!( &plan.items[0].row, @@ -1243,9 +1357,16 @@ mod tests { row.level_at = Some(now); // freshness advanced by a later same-level run let states = HashMap::from([(key.clone(), row)]); - let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| { - false - }); + let plan = plan_dispatch( + &c, + &states, + &rows_for(&c), + &tc(None), + now, + 0, + false, + |_, _, _| false, + ); assert_eq!( plan.items[0].episode, @@ -1270,6 +1391,7 @@ mod tests { &tc(None), 1_000 * SEC, 0, + false, |_, _, _| false, ); assert_eq!(plan.items.len(), 3); @@ -1290,6 +1412,7 @@ mod tests { &tc(None), 1_000 * SEC, 2, + false, |_, _, _| false, ); @@ -1324,9 +1447,16 @@ mod tests { let c = build(); let states = states_for(&c, now, None, None); - let first = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 2, |_, _, _| { - false - }); + let first = plan_dispatch( + &c, + &states, + &rows_for(&c), + &tc(None), + now, + 2, + false, + |_, _, _| false, + ); let second = plan_dispatch( &build(), &states, @@ -1334,6 +1464,7 @@ mod tests { &tc(None), now, 2, + false, |_, _, _| false, ); @@ -1367,9 +1498,16 @@ mod tests { ), ]); - let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 1, |_, _, _| { - false - }); + let plan = plan_dispatch( + &c, + &states, + &rows_for(&c), + &tc(None), + now, + 1, + false, + |_, _, _| false, + ); assert_eq!(plan.items.len(), 1); assert_eq!(plan.items[0].group_key, key_b); @@ -1387,9 +1525,16 @@ mod tests { let (key_a, key_b) = (key_of("a"), key_of("b")); let states = states_for(&c, now, None, None); for _ in 0..2 { - let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 1, |k, _, _| { - k == key_a - }); + let plan = plan_dispatch( + &c, + &states, + &rows_for(&c), + &tc(None), + now, + 1, + false, + |k, _, _| k == key_a, + ); assert_eq!(plan.items.len(), 1); assert_eq!(plan.items[0].group_key, key_b); assert_eq!(plan.suppressed, 1); @@ -1408,7 +1553,7 @@ mod tests { let acked = |k: &str, d: &DeliveryDecision, last: Option| { (k == key_a || k == key_b) && d.ack_withholds(last) }; - let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, acked); + let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, false, acked); assert_eq!( plan.muted, vec![key_a], @@ -1428,9 +1573,16 @@ mod tests { let c = classified(&[("a", 150.0), ("b", 75.0)], 500); let (key_a, key_b) = (key_of("a"), key_of("b")); let states = states_for(&c, now, None, None); - let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |k, _, _| { - k == key_a - }); + let plan = plan_dispatch( + &c, + &states, + &rows_for(&c), + &tc(None), + now, + 0, + false, + |k, _, _| k == key_a, + ); assert_eq!(plan.items.len(), 1); assert_eq!(plan.items[0].group_key, key_b); assert_eq!(plan.muted, vec![key_a]); @@ -2185,4 +2337,109 @@ mod tests { assert_eq!(back, episode); assert_eq!(back.notified_at_enqueue, (None, None)); } + + // ═════════════════════════════════════════════════════════════════════ + // suppress_repeats — notify once per state change + // ═════════════════════════════════════════════════════════════════════ + + #[test] + fn only_the_notified_level_inside_an_open_episode_is_a_repeat() { + use AlertLevel::{Critical, Warning}; + // Still firing at the delivered level, or back at it inside a keep_firing_for hold. + assert!(repeats_last_notification(Critical, true, Some(Critical))); + // Escalated or de-escalated since the last delivery. + assert!(!repeats_last_notification(Critical, true, Some(Warning))); + assert!(!repeats_last_notification(Warning, true, Some(Critical))); + // The episode closed on a recovery: a new incident. + assert!(!repeats_last_notification(Critical, false, Some(Critical))); + // Nothing was ever delivered. + assert!(!repeats_last_notification(Critical, false, None)); + } + + #[test] + fn a_repeat_is_withheld_like_a_downtime_or_an_ack() { + assert!(DeliveryDecision::SuppressedAsRepeat.withheld()); + assert!(DeliveryDecision::SuppressedByAck("u".into()).withheld()); + assert!(!DeliveryDecision::SuppressedBySilence.withheld()); + assert!(!DeliveryDecision::Deliver.withheld()); + } + + /// Every group of `c` delivered at Critical, inside an open recovery episode. + fn notified_states(c: &GroupClassification) -> HashMap { + let mut states = states_for(c, MIN, Some(AlertLevel::Critical), None); + for row in states.values_mut() { + row.episode_id = Some("ep-1".to_string()); + } + states + } + + #[test] + fn plan_dispatch_withholds_a_repeat_only_when_the_switch_is_on() { + let c = classified(&[("a", 150.0)], 500); + let states = notified_states(&c); + let plan = |on| { + plan_dispatch( + &c, + &states, + &rows_for(&c), + &tc(None), + 10 * MIN, + 0, + on, + |_, _, _| false, + ) + }; + + assert_eq!( + plan(false).items.len(), + 1, + "silence 0 and the switch off page again" + ); + let on = plan(true); + assert!(on.items.is_empty()); + assert_eq!((on.suppressed, on.repeats), (1, 1)); + } + + #[test] + fn a_repeat_does_not_spend_the_send_budget() { + // `a` repeats; `b` fires for the first time and must get the single slot. + let c = classified(&[("a", 150.0), ("b", 150.0)], 500); + let mut states = states_for(&c, MIN, None, None); + let a = states.get_mut(&key_of("a")).unwrap(); + a.last_notified_level = Some(AlertLevel::Critical); + a.episode_id = Some("ep-1".to_string()); + + let plan = plan_dispatch( + &c, + &states, + &rows_for(&c), + &tc(None), + 10 * MIN, + 1, + true, + |_, _, _| false, + ); + assert_eq!(plan.items.len(), 1); + assert_eq!(plan.items[0].group_key, key_of("b")); + assert!(plan.dropped_by_knob.is_empty()); + assert_eq!(plan.repeats, 1); + } + + #[test] + fn a_muted_group_is_reported_muted_not_repeated() { + let c = classified(&[("a", 150.0)], 500); + let states = notified_states(&c); + let plan = plan_dispatch( + &c, + &states, + &rows_for(&c), + &tc(None), + 10 * MIN, + 0, + true, + |_, _, _| true, + ); + assert_eq!(plan.muted, vec![key_of("a")]); + assert_eq!(plan.repeats, 0); + } } diff --git a/src/config/src/meta/alerts/level.rs b/src/config/src/meta/alerts/level.rs index a77d64fa97..0b981e99a4 100644 --- a/src/config/src/meta/alerts/level.rs +++ b/src/config/src/meta/alerts/level.rs @@ -366,6 +366,8 @@ pub enum DeliveryDecision { SuppressedByDowntime(String), /// The current firing is acknowledged by this user; repeats are withheld. SuppressedByAck(String), + /// `suppress_repeats` is on and nothing changed since the last delivered notification. + SuppressedAsRepeat, } impl DeliveryDecision { @@ -385,6 +387,14 @@ impl DeliveryDecision { } } + /// Withheld by a downtime, an ack or as a repeat, rather than by the level decision. + pub fn withheld(&self) -> bool { + matches!( + self, + Self::SuppressedByDowntime(_) | Self::SuppressedByAck(_) | Self::SuppressedAsRepeat + ) + } + pub fn acked_by(&self) -> Option<&str> { match self { Self::SuppressedByAck(by) => Some(by), diff --git a/src/config/src/meta/dashboards/v8/mod.rs b/src/config/src/meta/dashboards/v8/mod.rs index 57384779c8..8abb67ea34 100644 --- a/src/config/src/meta/dashboards/v8/mod.rs +++ b/src/config/src/meta/dashboards/v8/mod.rs @@ -480,6 +480,10 @@ pub struct FilterCondition { pub value: Option, pub logical_operator: String, pub filter_type: String, + /// Set when `value` is an explicit `''` from SQL, so the builder keeps `x <> ''` + /// instead of treating the empty value as "not entered yet". + #[serde(default, skip_serializing_if = "Option::is_none")] + pub empty_literal: Option, } #[derive(Debug, Clone, PartialEq, Hash, Serialize, Deserialize, ToSchema)] @@ -810,6 +814,22 @@ pub struct PanelConfig { #[serde(skip_serializing_if = "Option::is_none")] flame_graph_value_field: Option, #[serde(skip_serializing_if = "Option::is_none")] + boxplot_whisker: Option, + #[serde(skip_serializing_if = "Option::is_none")] + logs_message_field: Option, + #[serde(skip_serializing_if = "Option::is_none")] + logs_level_field: Option, + #[serde(skip_serializing_if = "Option::is_none")] + logs_time_field: Option, + #[serde(skip_serializing_if = "Option::is_none")] + logs_wrap: Option, + #[serde(skip_serializing_if = "Option::is_none")] + logs_prettify_json: Option, + #[serde(skip_serializing_if = "Option::is_none")] + logs_show_time: Option, + #[serde(skip_serializing_if = "Option::is_none")] + logs_order: Option, + #[serde(skip_serializing_if = "Option::is_none")] hostmap_entity_label: Option, #[serde(skip_serializing_if = "Option::is_none")] hostmap_group_by: Option>, @@ -1556,6 +1576,39 @@ mod tests { assert_eq!(again, d); } + #[test] + fn test_filter_condition_empty_literal_roundtrip() { + let kept = serde_json::json!({ + "type": "condition", + "values": [], + "column": { "field": "service_name", "streamAlias": null }, + "operator": "<>", + "value": "", + "logicalOperator": "AND", + "filterType": "condition", + "emptyLiteral": true + }); + let filter: PanelFilter = serde_json::from_value(kept).unwrap(); + match &filter { + PanelFilter::Condition(c) => assert_eq!(c.empty_literal, Some(true)), + PanelFilter::Group(_) => panic!("expected a condition"), + } + assert_eq!(serde_json::to_value(&filter).unwrap()["emptyLiteral"], true); + + let plain = serde_json::json!({ + "type": "condition", + "values": [], + "column": { "field": "service_name", "streamAlias": null }, + "operator": "=", + "value": "", + "logicalOperator": "AND", + "filterType": "condition" + }); + let filter: PanelFilter = serde_json::from_value(plain).unwrap(); + let back = serde_json::to_value(&filter).unwrap(); + assert!(!back.as_object().unwrap().contains_key("emptyLiteral")); + } + #[test] fn test_annotation_sources_change_the_hash() { let plain: super::super::Dashboard = make_dashboard().into(); @@ -3022,7 +3075,15 @@ mod tests { }); let rest4 = serde_json::json!({ "flame_graph_stack_field": "stack", - "flame_graph_value_field": "samples" + "flame_graph_value_field": "samples", + "boxplot_whisker": "iqr", + "logs_message_field": "message", + "logs_level_field": "level", + "logs_time_field": "_timestamp", + "logs_wrap": true, + "logs_prettify_json": false, + "logs_show_time": true, + "logs_order": "oldest" }); for part in [rest, rest2, rest3, rest4] { fields diff --git a/src/config/src/meta/fleet.rs b/src/config/src/meta/fleet.rs index 2b96feff45..1909e1670b 100644 --- a/src/config/src/meta/fleet.rs +++ b/src/config/src/meta/fleet.rs @@ -69,7 +69,7 @@ pub struct Collector { pub version: Option, pub outdated: bool, pub status: CollectorStatus, - /// The connection's peer address, honouring `X-Forwarded-For`. + /// The connection's client address, as `ZO_HTTP_REAL_IP_SOURCE` resolves it. pub last_connected_from: Option, pub fleet: CollectorFleet, pub token_id: Option, diff --git a/src/config/src/meta/public_dashboards.rs b/src/config/src/meta/public_dashboards.rs index 1ab0d08b46..ee428e750c 100644 --- a/src/config/src/meta/public_dashboards.rs +++ b/src/config/src/meta/public_dashboards.rs @@ -413,9 +413,12 @@ pub struct KeyedRange { pub refresh_secs: i64, } -/// A dashboard variable as the public viewer sees it: display label and frozen value only. +/// A dashboard variable as the public viewer sees it: name, display label and frozen value only. #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] pub struct PublicVariable { + /// The name panel titles reference as `$name`; empty in a result stored before it was added. + #[serde(default)] + pub name: String, pub label: String, pub value: serde_json::Value, /// Set for a tab-scoped variable: the tab this value belongs to. @@ -472,6 +475,7 @@ impl PublicVariable { .into_iter() .map(|(key, tab_id, panel_id)| { let entry = Self { + name: v.name.clone(), label: label.clone(), value: values.get(&key).cloned().unwrap_or(serde_json::Value::Null), tab_id, @@ -750,6 +754,34 @@ mod tests { assert_eq!(cfg.refresh_secs, 300); } + #[test] + fn public_variables_carry_the_name_panel_titles_reference() { + let dash = Dashboard { + v8: Some( + serde_json::from_value(serde_json::json!({ + "variables": { "list": [ + { "type": "custom", "name": "level", "label": "Level" }, + { "type": "custom", "name": "svc", "scope": "tabs", "tabs": ["t1"] } + ] } + })) + .unwrap(), + ), + ..Default::default() + }; + let values = BTreeMap::from([("level".to_string(), serde_json::json!("error"))]); + let list = PublicVariable::list(&dash, &values); + let names: Vec<(&str, &str)> = list + .iter() + .map(|v| (v.name.as_str(), v.label.as_str())) + .collect(); + assert_eq!(names, [("level", "Level"), ("svc", "svc")]); + assert_eq!(serde_json::to_value(&list[0]).unwrap()["name"], "level"); + let stored: PublicVariable = + serde_json::from_value(serde_json::json!({ "label": "Level", "value": "error" })) + .unwrap(); + assert_eq!(stored.name, ""); + } + #[test] fn identity_names_the_org() { assert_eq!( diff --git a/src/config/src/meta/self_reporting/mod.rs b/src/config/src/meta/self_reporting/mod.rs index 187de9df48..d202fdb155 100644 --- a/src/config/src/meta/self_reporting/mod.rs +++ b/src/config/src/meta/self_reporting/mod.rs @@ -405,6 +405,7 @@ mod tests { ack_source: None, taken_over_from: None, delivery_attempted: None, + repeat_suppressed: None, trigger_source: None, triggered_by: None, recipient_count: None, @@ -965,6 +966,7 @@ mod tests { ack_source: None, taken_over_from: None, delivery_attempted: None, + repeat_suppressed: None, trigger_source: None, triggered_by: None, recipient_count: None, @@ -1057,6 +1059,7 @@ mod tests { ack_source: None, taken_over_from: None, delivery_attempted: None, + repeat_suppressed: None, trigger_source: None, triggered_by: None, recipient_count: None, @@ -1133,6 +1136,7 @@ mod tests { ack_source: None, taken_over_from: None, delivery_attempted: None, + repeat_suppressed: None, trigger_source: None, triggered_by: None, recipient_count: None, diff --git a/src/config/src/meta/self_reporting/usage.rs b/src/config/src/meta/self_reporting/usage.rs index 703e85cc2f..8724dbf3e9 100644 --- a/src/config/src/meta/self_reporting/usage.rs +++ b/src/config/src/meta/self_reporting/usage.rs @@ -354,6 +354,9 @@ pub struct TriggerData { /// Whether a firing tried a destination, workflow or incident notification; None on old rows. #[serde(default, skip_serializing_if = "Option::is_none")] pub delivery_attempted: Option, + /// Set when `suppress_repeats` withheld a notification that repeated the last delivered state. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub repeat_suppressed: Option, /// "scheduled" or "manual"; None on older rows, which read as scheduled. #[serde(default, skip_serializing_if = "Option::is_none")] pub trigger_source: Option, @@ -413,6 +416,7 @@ impl Default for TriggerData { ack_source: None, taken_over_from: None, delivery_attempted: None, + repeat_suppressed: None, trigger_source: None, triggered_by: None, recipient_count: None, @@ -473,6 +477,7 @@ impl TriggerData { ack_source: Some(AckSource::Api), taken_over_from: Some(String::new()), delivery_attempted: Some(true), + repeat_suppressed: None, trigger_source: Some(String::new()), triggered_by: Some(String::new()), recipient_count: Some(0), @@ -1915,6 +1920,7 @@ mod tests { ack_source: None, taken_over_from: None, delivery_attempted: None, + repeat_suppressed: None, trigger_source: None, triggered_by: None, recipient_count: None, diff --git a/src/config/src/meta/status_pages.rs b/src/config/src/meta/status_pages.rs index 944f9128c8..d426f41e74 100644 --- a/src/config/src/meta/status_pages.rs +++ b/src/config/src/meta/status_pages.rs @@ -46,6 +46,10 @@ pub const CHECK_STATUS_UNKNOWN: i32 = 0; pub const CHECK_STATUS_UP: i32 = 1; pub const CHECK_STATUS_WARNING: i32 = 2; pub const CHECK_STATUS_DOWN: i32 = 3; +/// The probe could not produce a verdict (dispatch dead-letter, expired lease, +/// infra fault). Says nothing about the target: it never opens an incident and +/// never counts toward recovery. Written by the dispatcher and the reaper. +pub const CHECK_STATUS_ERROR: i32 = 4; const MICROS_PER_MIN: i64 = 60 * 1_000_000; const MICROS_PER_DAY: i64 = 86_400 * 1_000_000; @@ -120,6 +124,10 @@ pub struct EngineCheck { pub first_fail_at: Option, /// When the streak started failing, for time-based confirmation. pub failing_since: Option, + /// True only when this tick sees a run result it has not seen before. + /// Recovery advances on distinct results, never on rebuild ticks, so the + /// same UP observed over ten ticks counts once. + pub fresh_result: bool, } /// An auto-incident notice currently open (or just resolved, for the merge @@ -535,12 +543,33 @@ pub fn incident_actions( snoozed: &HashSet, th: &Thresholds, now: i64, +) -> Vec { + incident_actions_with( + checks, + open_by_check, + recently_resolved, + snoozed, + |_| *th, + now, + ) +} + +/// [`incident_actions`] with per-check thresholds (each check is judged by the +/// confirmation settings of the page(s) it is mapped to). +pub fn incident_actions_with( + checks: &[EngineCheck], + open_by_check: &HashMap, + recently_resolved: &HashMap, + snoozed: &HashSet, + th_for: impl Fn(&str) -> Thresholds, + now: i64, ) -> Vec { let mut actions = Vec::new(); for check in checks { + let th = th_for(&check.id); match open_by_check.get(&check.id) { - Some(open) => handle_open(check, open, th, now, &mut actions), - None => handle_closed(check, recently_resolved, snoozed, th, now, &mut actions), + Some(open) => handle_open(check, open, &th, now, &mut actions), + None => handle_closed(check, recently_resolved, snoozed, &th, now, &mut actions), } } actions @@ -568,7 +597,16 @@ fn handle_open( now: i64, actions: &mut Vec, ) { - if check.last_check_status == CHECK_STATUS_UP { + // UP and WARNING both mean the target answered: they advance recovery, but + // only on a result this loop has not counted yet. + let healthy = matches!( + check.last_check_status, + CHECK_STATUS_UP | CHECK_STATUS_WARNING + ); + if healthy { + if !check.fresh_result { + return; + } let streak = open.recovery_streak + 1; if streak >= th.confirm_recovery { actions.push(EngineAction::Resolve { @@ -581,8 +619,14 @@ fn handle_open( streak, }); } - } else if check.last_check_status == CHECK_STATUS_DOWN && open.recovery_streak > 0 { - // Failed again mid-recovery: the streak restarts from zero. + } else if matches!( + check.last_check_status, + CHECK_STATUS_DOWN | CHECK_STATUS_ERROR + ) && open.recovery_streak > 0 + { + // Failed again mid-recovery: the streak restarts from zero. ERROR does + // the same: an unknown verdict cannot prove recovery, so UP, ERROR, UP + // is not two consecutive good results. The incident stays open. actions.push(EngineAction::RecoveryProgress { notice_id: open.notice_id.clone(), streak: 0, @@ -761,9 +805,119 @@ mod tests { consecutive_failures: streak, first_fail_at: None, failing_since: None, + fresh_result: true, } } + fn open_inc(streak: i32) -> HashMap { + [( + "c1".to_string(), + OpenAutoIncident { + notice_id: "n1".into(), + check_id: "c1".into(), + recovery_streak: streak, + }, + )] + .into() + } + + /// Replays results against an open incident, applying RecoveryProgress + /// back into the streak; returns whether it resolved. + fn replay(statuses: &[(i32, bool)], th: Thresholds) -> bool { + let mut streak = 0; + for &(status, fresh) in statuses { + let mut c = check("c1", status, 0); + c.fresh_result = fresh; + let acts = incident_actions( + &[c], + &open_inc(streak), + &HashMap::new(), + &HashSet::new(), + &th, + 0, + ); + for a in acts { + match a { + EngineAction::Resolve { .. } => return true, + EngineAction::RecoveryProgress { streak: s, .. } => streak = s, + _ => panic!("unexpected {a:?}"), + } + } + } + false + } + + fn th(failures: i32, recovery: i32) -> Thresholds { + Thresholds { + confirm_failures: failures, + confirm_recovery: recovery, + confirm_after_micros: None, + } + } + + #[test] + fn up_error_up_does_not_resolve_with_recovery_two() { + let s = CHECK_STATUS_UP; + let e = CHECK_STATUS_ERROR; + assert!(!replay(&[(s, true), (e, true), (s, true)], th(3, 2))); + assert!(replay( + &[(s, true), (e, true), (s, true), (s, true)], + th(3, 2) + )); + } + + #[test] + fn error_never_opens_an_incident() { + let c = check("c1", CHECK_STATUS_ERROR, 50); + let acts = incident_actions( + &[c], + &HashMap::new(), + &HashMap::new(), + &HashSet::new(), + &th(1, 1), + 0, + ); + assert!(acts.is_empty()); + } + + #[test] + fn error_keeps_an_open_incident_open() { + let e = CHECK_STATUS_ERROR; + assert!(!replay(&[(e, true); 10], th(3, 1))); + } + + #[test] + fn warning_advances_recovery() { + let w = CHECK_STATUS_WARNING; + assert!(replay(&[(w, true), (w, true)], th(3, 2))); + } + + #[test] + fn the_same_result_over_many_ticks_counts_once() { + let mut ticks = vec![(CHECK_STATUS_UP, true)]; + ticks.extend(std::iter::repeat_n((CHECK_STATUS_UP, false), 20)); + assert!(!replay(&ticks, th(3, 2))); + ticks.push((CHECK_STATUS_UP, true)); + assert!(replay(&ticks, th(3, 2))); + } + + #[test] + fn seven_failures_four_recoveries() { + let t = th(7, 4); + let none = HashMap::new(); + let quiet = HashSet::new(); + let six = [check("c1", CHECK_STATUS_DOWN, 6)]; + assert!(incident_actions(&six, &none, &HashMap::new(), &quiet, &t, 0).is_empty()); + let seven = [check("c1", CHECK_STATUS_DOWN, 7)]; + assert_eq!( + incident_actions(&seven, &none, &HashMap::new(), &quiet, &t, 0).len(), + 1 + ); + let up = CHECK_STATUS_UP; + assert!(!replay(&[(up, true); 3], t)); + assert!(replay(&[(up, true); 4], t)); + } + #[test] fn union_counts_overlaps_once() { // Two checks confirm the same outage: 10:00-10:30 and 10:10-10:40. diff --git a/src/config/src/meta/user.rs b/src/config/src/meta/user.rs index 6454a1dc80..5869dfb25e 100644 --- a/src/config/src/meta/user.rs +++ b/src/config/src/meta/user.rs @@ -153,7 +153,10 @@ impl FromStr for UserRole { "user" | "allowed_user" => Ok(UserRole::User), "service_account" => Ok(UserRole::ServiceAccount), "sre_agent" => Ok(UserRole::SreAgent), + #[cfg(not(feature = "enterprise"))] _ => Ok(UserRole::Admin), + #[cfg(feature = "enterprise")] + _ => Ok(UserRole::User), } } } @@ -413,9 +416,18 @@ mod tests { #[test] fn test_invalid_string_defaults_to_admin() { + #[cfg(feature = "enterprise")] + assert_eq!(UserRole::from_str("invalid").unwrap(), UserRole::User); + #[cfg(not(feature = "enterprise"))] assert_eq!(UserRole::from_str("invalid").unwrap(), UserRole::Admin); + #[cfg(feature = "enterprise")] + assert_eq!(UserRole::from_str("").unwrap(), UserRole::User); + #[cfg(not(feature = "enterprise"))] assert_eq!(UserRole::from_str("").unwrap(), UserRole::Admin); + #[cfg(not(feature = "enterprise"))] assert_eq!(UserRole::from_str("Admin").unwrap(), UserRole::Admin); // case sensitive + #[cfg(feature = "enterprise")] + assert_eq!(UserRole::from_str("Admin").unwrap(), UserRole::User); // case sensitive } } diff --git a/src/config/src/utils/str.rs b/src/config/src/utils/str.rs index 60f5cca90a..46d5142756 100644 --- a/src/config/src/utils/str.rs +++ b/src/config/src/utils/str.rs @@ -34,11 +34,19 @@ pub static EMAIL_REGEX: LazyLock = LazyLock::new(|| { .unwrap() }); +pub static DOMAIN_REGEX: LazyLock = + LazyLock::new(|| Regex::new(r"^([a-zA-Z0-9]+([\-\.][a-zA-Z0-9]+)*\.[a-zA-Z]{2,63})$").unwrap()); + /// Returns true if `email` is syntactically valid per [`EMAIL_REGEX`]. pub fn is_valid_email(email: &str) -> bool { EMAIL_REGEX.is_match(email) } +/// Returns true if `domain` is syntactically valid per [`DOMAIN_REGEX`]. +pub fn is_valid_domain(domain: &str) -> bool { + DOMAIN_REGEX.is_match(domain) +} + /// Characters an OpenFGA object id cannot represent. Detection /// ([`is_ofga_unsupported`]) and sanitization ([`into_ofga_supported_format`]) /// share this pattern so they always agree on what is unsafe. diff --git a/src/core/Cargo.toml b/src/core/Cargo.toml index 2bd451861a..816cd6f0bd 100644 --- a/src/core/Cargo.toml +++ b/src/core/Cargo.toml @@ -48,9 +48,10 @@ vectorscan = [ ] # Lets dependants' tests install a fake stream access checker; never enabled outside dev-dependencies. test-utils = [] +cpu-profiling = ["dep:pprof"] profiling = [ + "cpu-profiling", "dep:jemalloc_pprof", - "dep:pprof", "dep:tikv-jemalloc-ctl", ] diff --git a/src/core/src/alerts/alert.rs b/src/core/src/alerts/alert.rs index ac197dd8df..796a150157 100644 --- a/src/core/src/alerts/alert.rs +++ b/src/core/src/alerts/alert.rs @@ -327,6 +327,8 @@ pub enum AlertError { CronHasNoFutureOccurrence { cron: String }, #[error("Realtime alerts cannot notify on recovery or keep firing")] RecoveryOnRealtimeAlert, + #[error("Realtime alerts cannot suppress repeat notifications")] + RepeatSuppressionOnRealtimeAlert, #[error("recovery destinations were set without notify_on_recovery, so nothing would use them")] RecoveryDestinationsWithoutRecovery, #[error( @@ -895,6 +897,10 @@ async fn prepare_alert( if alert.is_real_time && (alert.notify_on_recovery || alert.keep_firing_for != 0) { return Err(AlertError::RecoveryOnRealtimeAlert); } + // A realtime alert keeps no state row, so it has no last delivered state to compare against. + if alert.is_real_time && alert.suppress_repeats { + return Err(AlertError::RepeatSuppressionOnRealtimeAlert); + } // Multi-level thresholds (alerts_2.md Feature 1). Rejected at write time so // an unreachable warning level can never reach the evaluator. @@ -8853,6 +8859,33 @@ mod tests { } } + #[tokio::test] + async fn prepare_alert_refuses_repeat_suppression_on_a_realtime_alert() { + use infra::schema::{STREAM_SCHEMAS_LATEST, SchemaCache}; + + let org_id = "test_org_alert_save_repeat_realtime"; + let key = format!("{org_id}/{}/app", StreamType::Logs); + STREAM_SCHEMAS_LATEST.write().await.insert( + key.clone(), + SchemaCache::new(arrow_schema::Schema::new(vec![arrow_schema::Field::new( + "id", + DataType::Utf8, + true, + )])), + ); + let mut alert = Alert::default(); + alert.trigger_condition.frequency = 60; + alert.is_real_time = true; + alert.suppress_repeats = true; + let res = prepare_alert(org_id, "app", "realtime", &mut alert, true, false).await; + STREAM_SCHEMAS_LATEST.write().await.remove(&key); + + assert!( + matches!(res, Err(AlertError::RepeatSuppressionOnRealtimeAlert)), + "{res:?}" + ); + } + #[tokio::test] async fn prepare_alert_rejects_a_cron_with_no_future_occurrence() { let err = prepare_schedule(|a| { diff --git a/src/core/src/alerts/prom_rule.rs b/src/core/src/alerts/prom_rule.rs index 166957f33d..5aa028977b 100644 --- a/src/core/src/alerts/prom_rule.rs +++ b/src/core/src/alerts/prom_rule.rs @@ -446,6 +446,7 @@ mod tests { &TriggerCondition::default(), 600, 0, + false, |_, _, _| false, ); assert!(dispatch.inconsistent.is_empty()); diff --git a/src/core/src/alerts/scheduler/handlers.rs b/src/core/src/alerts/scheduler/handlers.rs index 43d68b3bff..5b278ad2d7 100644 --- a/src/core/src/alerts/scheduler/handlers.rs +++ b/src/core/src/alerts/scheduler/handlers.rs @@ -515,6 +515,8 @@ struct GroupDispatchOutcome { downtime_id: Option, /// Who acked the first group an ack kept silent. acked_by: Option, + /// Some group was withheld because it repeated its last delivery. + repeated: bool, } impl GroupDispatchOutcome { @@ -629,6 +631,7 @@ async fn dispatch_per_group( state_failed: true, downtime_id: None, acked_by: None, + repeated: false, }); }; @@ -647,6 +650,7 @@ async fn dispatch_per_group( state_failed: true, downtime_id: None, acked_by: None, + repeated: false, }); } }; @@ -684,6 +688,7 @@ async fn dispatch_per_group( &alert.trigger_condition, now_micros(), cfg.limit.alert_max_group_notifications_per_eval, + alert.suppresses_repeats(), |key, decision, last_notified| { muted_groups.contains_key(key) || (acked_groups.contains_key(key) && decision.ack_withholds(last_notified)) @@ -749,7 +754,7 @@ async fn dispatch_per_group( continue; } // Chosen before the send: a resolve only closes a ticket opened under the key it quotes. - let episode_key = alert.notify_on_recovery.then(|| { + let episode_key = (alert.notify_on_recovery || alert.suppresses_repeats()).then(|| { states .get(&item.group_key) .and_then(|state| state.episode_id.clone()) @@ -771,7 +776,7 @@ async fn dispatch_per_group( // last so a label value containing `{...}` cannot expand. Some(&item.labels), &[], - episode_key.clone(), + episode_key.clone().filter(|_| alert.notify_on_recovery), ) .await; @@ -835,9 +840,11 @@ async fn dispatch_per_group( log::info!( "[SCHEDULER trace_id {trace_id}] alert {alert_id}: per-group dispatch delivered={delivered} \ - pending={} failed={failed} suppressed={} downtime={downtime_suppressed} candidates={}", + pending={} failed={failed} suppressed={} repeats={} downtime={downtime_suppressed} \ + candidates={}", plan.pending, plan.suppressed, + plan.repeats, plan.items.len() ); Some(GroupDispatchOutcome { @@ -849,6 +856,7 @@ async fn dispatch_per_group( state_failed: false, downtime_id, acked_by, + repeated: plan.repeats > 0, }) } @@ -3053,6 +3061,45 @@ pub(crate) async fn correlate_muted_incident( } } +/// A repeat still correlates, or a continuous firing lets its incident go stale and auto-resolve. +#[cfg(feature = "enterprise")] +async fn correlate_repeated_incident( + trace_id: &str, + alert: &config::meta::alerts::alert::Alert, + rows: &[config::utils::json::Map], + triggered_at: i64, + eval_level: Option, + actual_value: Option, +) { + let Some(first_row) = rows.first() else { + return; + }; + let incidents_enabled = o2_enterprise::enterprise::common::config::get_config() + .incidents + .enabled; + if !correlates_repeated_firing(alert.creates_incident, incidents_enabled) { + return; + } + // Correlation itself stays silent on a repeat and only notifies a new or escalated incident. + if let Err(e) = crate::alerts::incidents::correlate_alert_to_incident( + alert, + first_row, + rows, + triggered_at, + eval_level, + actual_value, + None, + ) + .await + { + log::error!( + "[SCHEDULER trace_id {trace_id}] repeat incident correlation failed for {}/{}: {e}", + alert.org_id, + alert.name + ); + } +} + /// Whether a matched single-series run is still inside its pending period. fn holds_pending( alert: &config::meta::alerts::alert::Alert, @@ -3134,6 +3181,12 @@ fn correlates_muted_firing( creates_incident && incidents_enabled && downtime.incident_mode.is_muted() } +/// Whether a firing withheld as a repeat still feeds its incident. +#[cfg(feature = "enterprise")] +fn correlates_repeated_firing(creates_incident: bool, incidents_enabled: bool) -> bool { + creates_incident && incidents_enabled +} + /// Who acked a single-row alert's current firing, when that ack withholds this run's send. async fn single_row_ack( alert: &config::meta::alerts::alert::Alert, @@ -3191,6 +3244,32 @@ fn records_delivery_state(alert_level_delivery: bool, delivery: &DeliveryDecisio alert_level_delivery && delivery.resets_silence() } +/// Whether a single-row firing at `level` repeats what its open recovery episode already sent. +fn single_row_repeats( + alert: &config::meta::alerts::alert::Alert, + level: Option, + rollup: Option<&config::meta::alerts::state::AlertState>, + recorded_level: Option, + alert_level_delivery: bool, +) -> bool { + let (Some(level), Some(rollup)) = (level.filter(|l| l.is_firing()), rollup) else { + return false; + }; + // Only the multi-level path records the delivered level; a single-level alert fires at one. + let last_notified = if alert_level_delivery { + recorded_level.and_then(config::meta::alerts::level::AlertLevel::from_i32) + } else { + Some(level) + }; + alert.suppresses_repeats() + && !alert.query_condition.multi_alert_enabled() + && config::meta::alerts::dispatch::repeats_last_notification( + level, + rollup.episode_id.is_some(), + last_notified, + ) +} + /// A delivered firing ends the Muted chip; a pending run persists before the downtime decision. #[cfg(feature = "enterprise")] fn clears_recorded_mute(outcome: &RunOutcome) -> bool { @@ -3594,7 +3673,7 @@ async fn handle_alert_triggers( .unwrap(); let last_states = if !alert.query_condition.multi_alert_enabled() - && alert.pending_period_sec > 0 + && (alert.pending_period_sec > 0 || alert.suppresses_repeats()) { load_tracked_group_states(&alert.get_unique_key()).await.inspect_err(|e|{ log::error!("[SCHEDULER trace_id {scheduler_trace_id}] alert {}/{} error in getting alert state: {e}",trigger.org,trigger.module_key); @@ -3927,10 +4006,27 @@ async fn handle_alert_triggers( } else { None }; + let delivery = match (downtime.as_ref(), acked_by) { + (Some(downtime), _) => DeliveryDecision::SuppressedByDowntime(downtime.id.clone()), + (None, Some(by)) => DeliveryDecision::SuppressedByAck(by), + (None, None) + if level_delivery.should_deliver() + && single_row_repeats( + &alert, + eval_level, + last_states.get(config::meta::alerts::state::ROLLUP_GROUP_KEY), + trigger_data.last_notified_level, + alert_level_delivery, + ) => + { + DeliveryDecision::SuppressedAsRepeat + } + (None, None) => level_delivery, + }; // D6: a suppressed run starts no silence window, so the first run after the window delivers. if starts_silence_window( trigger_results.data.is_some(), - downtime.is_some() || acked_by.is_some(), + delivery.withheld(), alert.trigger_condition.silence, evaluates_through_silence, ) { @@ -3952,14 +4048,7 @@ async fn handle_alert_triggers( // send notification // Nowhere to send means no delivery to record: no silence window or episode. let has_target = has_delivery_target(&alert); - let delivery = match (downtime.as_ref(), acked_by) { - (Some(downtime), _) => DeliveryDecision::SuppressedByDowntime(downtime.id.clone()), - (None, Some(by)) => DeliveryDecision::SuppressedByAck(by), - (None, None) => level_delivery, - }; - if !delivery.should_deliver() - && (alert_level_delivery || downtime.is_some() || delivery.acked_by().is_some()) - { + if !delivery.should_deliver() && (alert_level_delivery || delivery.withheld()) { log::info!( "[SCHEDULER trace_id {scheduler_trace_id}] delivery suppressed ({delivery:?}) for {}/{}", new_trigger.org, @@ -4016,6 +4105,20 @@ async fn handle_alert_triggers( ) .await; } + #[cfg(feature = "enterprise")] + if delivery == DeliveryDecision::SuppressedAsRepeat + && let Some(rows) = trigger_results.data.as_deref() + { + correlate_repeated_incident( + &scheduler_trace_id, + &alert, + rows, + triggered_at, + eval_level, + trigger_results.actual_value, + ) + .await; + } // What this evaluation DELIVERED, not what it decided to: the episode opens on a landed send. let mut episode_delivered = false; @@ -4213,6 +4316,9 @@ async fn handle_alert_triggers( } dispatch.record_downtime(&mut trigger_data_stream); + if dispatch.repeated { + trigger_data_stream.repeat_suppressed = Some(true); + } // The rollup row was committed before anything was sent. if let Some(outcome) = dispatch.rollup_rewrite() && let Some(alert_id) = alert.id.as_ref() @@ -4486,6 +4592,8 @@ async fn handle_alert_triggers( "alerts", by.to_string(), ); + } else if delivery == DeliveryDecision::SuppressedAsRepeat { + trigger_data_stream.repeat_suppressed = Some(true); } } else if trigger_results.frozen { // Frozen is not Normal: nothing was measured (§7.6). `Skipped` is @@ -7384,6 +7492,7 @@ mod tests { state_failed: false, downtime_id: None, acked_by: None, + repeated: false, } } @@ -8944,6 +9053,14 @@ mod tests { ); } + #[cfg(feature = "enterprise")] + #[test] + fn a_repeat_still_feeds_its_incident_only_when_incidents_apply() { + assert!(correlates_repeated_firing(true, true)); + assert!(!correlates_repeated_firing(false, true)); + assert!(!correlates_repeated_firing(true, false)); + } + #[cfg(feature = "enterprise")] #[test] fn only_an_unsuppressed_matched_run_clears_the_recorded_mute() { diff --git a/src/core/src/fleet.rs b/src/core/src/fleet.rs index 65cada0036..ec93d431f1 100644 --- a/src/core/src/fleet.rs +++ b/src/core/src/fleet.rs @@ -58,9 +58,19 @@ pub async fn get_pipelines( } /// The handler has already checked the caller's org-wide fleet access, so this runs as the server. -pub async fn list_audit(org_id: &str, query: AuditQuery) -> Result { - o2_enterprise::enterprise::fleet::audit::list_audit(org_id, query, now_micros(), search_audit) - .await +pub async fn list_audit( + org_id: &str, + query: AuditQuery, + fleets: Option<&HashSet>, +) -> Result { + o2_enterprise::enterprise::fleet::audit::list_audit( + org_id, + query, + fleets, + now_micros(), + search_audit, + ) + .await } /// The check `/prometheus/api/v1/query` makes per metric stream; a refused query is `None`. diff --git a/src/core/src/http.rs b/src/core/src/http.rs index 0db066b3c1..06cc541502 100644 --- a/src/core/src/http.rs +++ b/src/core/src/http.rs @@ -99,6 +99,7 @@ impl From for Response { | AlertError::TzOffsetOutOfRange | AlertError::CronHasNoFutureOccurrence { .. } | AlertError::RecoveryOnRealtimeAlert + | AlertError::RepeatSuppressionOnRealtimeAlert | AlertError::RecoveryDestinationsWithoutRecovery | AlertError::RecoveryDestinationIsPlatform { .. } | AlertError::RecoveryNeedsJsonPayload { .. } diff --git a/src/core/src/lib.rs b/src/core/src/lib.rs index 39a82c671e..02ada8c213 100644 --- a/src/core/src/lib.rs +++ b/src/core/src/lib.rs @@ -67,7 +67,7 @@ pub mod public_dashboards; pub mod ratelimit; pub mod rum_pa; use search_service as search; -#[cfg(feature = "profiling")] +#[cfg(feature = "cpu-profiling")] pub mod self_profiles; pub mod self_reporting; pub mod service; @@ -88,6 +88,10 @@ pub mod users; #[cfg(feature = "enterprise")] pub mod workflows; +#[cfg(feature = "cloud")] +pub mod org_domain_ownership; +#[cfg(feature = "cloud")] +pub mod org_domain_requests; /// One authorized AI operation, or `()` in builds without the quota module, so /// the RCA signature keeps one shape instead of being cfg-gated at every caller. #[cfg(feature = "cloud")] diff --git a/src/core/src/logs/bulk.rs b/src/core/src/logs/bulk.rs index f8f282e5df..645a12f0e1 100644 --- a/src/core/src/logs/bulk.rs +++ b/src/core/src/logs/bulk.rs @@ -288,6 +288,12 @@ pub async fn ingest( written_any = true; } for status in v.status { + // the stream status keeps the items but not the write's errors flag + bulk_res.errors |= status + .items + .iter() + .flat_map(|item| item.values()) + .any(|item| item.error.is_some()); bulk_res.items.extend(status.items); } } @@ -621,8 +627,7 @@ mod tests { } else { assert!(rejected.is_empty()); } - // not set: Fluent Bit retries a whole chunk on it, duplicating the other streams' rows - assert!(!resp.errors); + assert_eq!(resp.errors, bounded); assert!( infra::schema::exists("test_org_id", StreamType::Logs, "test_bulk_key_len_ok").await ); diff --git a/src/core/src/logs/mod.rs b/src/core/src/logs/mod.rs index ab38955c0d..02c9792e41 100644 --- a/src/core/src/logs/mod.rs +++ b/src/core/src/logs/mod.rs @@ -396,6 +396,7 @@ fn reject_stream_rows( .map(|(_, record)| record.get("_id").and_then(Value::as_str).map(String::from)) .chain(columnar.into_iter().flat_map(|mut c| c.take_bulk_ids())); for doc_id in doc_ids { + bulk_res.errors = true; bulk::add_record_status( stream_name.to_string(), doc_id, @@ -798,7 +799,7 @@ mod tests { let IngestionStatus::Bulk(bulk) = bulk else { unreachable!() }; - assert!(!bulk.errors); + assert!(bulk.errors); let ids: Vec<&str> = bulk .items .iter() diff --git a/src/core/src/metrics/ingest.rs b/src/core/src/metrics/ingest.rs index cb44c233f8..f3bf2fd2d5 100644 --- a/src/core/src/metrics/ingest.rs +++ b/src/core/src/metrics/ingest.rs @@ -105,10 +105,10 @@ impl StreamKeyChecks { } /// Removes the records bound for streams that cannot be stored, as (stream, reason, records). - pub(super) fn drop_unstorable( + pub(super) fn drop_unstorable( &mut self, org_id: &str, - records_by_stream: &mut RecordsByStream, + records_by_stream: &mut HashMap>, ) -> Vec<(String, String, usize)> { let mut dropped = Vec::new(); records_by_stream.retain(|stream_name, records| { diff --git a/src/core/src/metrics/prom.rs b/src/core/src/metrics/prom.rs index c5d4294a25..59a701ea68 100644 --- a/src/core/src/metrics/prom.rs +++ b/src/core/src/metrics/prom.rs @@ -143,10 +143,15 @@ pub async fn remote_write( // check if stream is deleting from cache let mut stream_delete_status: HashMap = HashMap::new(); let mut skipped_records: u32 = 0; + // remote write has no partial success: a 4xx drops the whole batch, a 5xx retries it forever + let mut stream_key_checks = ingest::StreamKeyChecks::default(); // parse metadata for item in request.metadata { let metric_name = format_stream_name(item.metric_family_name.to_string()); + if stream_key_checks.violation(org_id, &metric_name).is_some() { + continue; + } let schema = infra::schema::get(org_id, &metric_name, StreamType::Metrics) .await .unwrap_or_else(|_| Schema::empty()); @@ -218,6 +223,8 @@ pub async fn remote_write( } } + unique_metrics.retain(|name| stream_key_checks.violation(org_id, name).is_none()); + let mut preload_pipeline_time = 0u128; let mut preload_uds_time = 0u128; let mut preload_schema_time = 0u128; @@ -379,6 +386,10 @@ pub async fn remote_write( continue; } + if stream_key_checks.violation(org_id, &metric_name).is_some() { + continue; + } + // Note: All configurations (pipeline, UDS, schema, partition, alerts) are now pre-loaded // before the loop to avoid repeated async queries @@ -552,6 +563,8 @@ pub async fn remote_write( .map(|(record, timestamp)| (record, timestamp, None)), ); } + // histogram bucket, exemplar and pipeline output streams can be too long when the metric fits + stream_key_checks.drop_unstorable(org_id, &mut json_data_by_stream); let step_start = std::time::Instant::now(); for (stream_name, mut json_data) in json_data_by_stream { @@ -2309,6 +2322,87 @@ mod tests { assert!(matchers.iter().all(|m| m.is_match("http_requests"))); } + // cloud builds reject the unknown test org at the trial check + #[cfg(not(feature = "cloud"))] + #[tokio::test] + async fn remote_write_drops_only_the_streams_whose_key_is_too_long() { + use infra::file_list::STREAM_KEY_MAX_LEN; + use prost::Message; + use proto::prometheus_rpc; + + let org_id = "test_org_id"; + let now = Utc::now().timestamp_millis(); + let series = |name: &str| prometheus_rpc::TimeSeries { + labels: vec![prometheus_rpc::Label { + name: NAME_LABEL.to_string(), + value: name.to_string(), + }], + samples: vec![prometheus_rpc::Sample { + value: 1.0, + timestamp: now, + }], + ..Default::default() + }; + let long_name = "m".repeat(STREAM_KEY_MAX_LEN); + // `_sum` fills the key exactly; `_count`, `_bucket` and `_exemplars` overflow it + let histogram_name = + "h".repeat(STREAM_KEY_MAX_LEN - format!("{org_id}/metrics/_sum").len()); + let histogram = prometheus_rpc::TimeSeries { + samples: vec![], + exemplars: vec![prometheus_rpc::Exemplar { + labels: vec![], + value: 1.0, + timestamp: now, + }], + histograms: vec![prometheus_rpc::Histogram { + sum: 1.0, + count: Some(prometheus_rpc::histogram::Count::CountInt(1)), + zero_count: Some(prometheus_rpc::histogram::ZeroCount::ZeroCountInt(1)), + timestamp: now, + ..Default::default() + }], + ..series(&histogram_name) + }; + let request = prometheus_rpc::WriteRequest { + timeseries: vec![ + series("test_prom_key_len_ok"), + series(&long_name), + histogram, + ], + metadata: vec![prometheus_rpc::MetricMetadata { + r#type: prometheus_rpc::metric_metadata::MetricType::Gauge as i32, + metric_family_name: long_name.clone(), + ..Default::default() + }], + }; + let body = snap::raw::Encoder::new() + .compress_vec(&request.encode_to_vec()) + .unwrap(); + remote_write( + org_id, + Bytes::from(body), + IngestUser::User("root@example.com".to_string()), + ) + .await + .unwrap(); + + let bounded = + infra::file_list::stream_key_violation(org_id, StreamType::Metrics, &long_name) + .is_some(); + let expected = [ + ("test_prom_key_len_ok".to_string(), true), + (format!("{histogram_name}_sum"), true), + (long_name, !bounded), + (format!("{histogram_name}_count"), !bounded), + (format!("{histogram_name}_bucket"), !bounded), + (exemplars_stream_name(&histogram_name), !bounded), + ]; + for (name, created) in expected { + let exists = infra::schema::exists(org_id, StreamType::Metrics, &name).await; + assert_eq!(exists, created, "{name}"); + } + } + #[tokio::test] async fn remote_write_rejects_a_snappy_header_declaring_more_than_the_limit() { let mut n = get_config().limit.req_payload_limit as u64 + 1; diff --git a/src/core/src/org_cleanup/coverage.rs b/src/core/src/org_cleanup/coverage.rs index d51dd436dc..c0f6a89d71 100644 --- a/src/core/src/org_cleanup/coverage.rs +++ b/src/core/src/org_cleanup/coverage.rs @@ -230,6 +230,8 @@ pub const COVERAGE: &[Coverage] = &[ table("workflow_errors", DB), table("workflow_run_data", DB), table("workflows", DB), + table("org_domain_ownership", DB), + table("org_domain_requests", DB), covered(Kind::KvPrefix, "/cipher_keys/{org}/", DB), covered(Kind::KvPrefix, "/compact/downsampling/{org}/", DB), covered(Kind::KvPrefix, "/compact/files/{org}/", DB), diff --git a/src/core/src/org_cleanup/mod.rs b/src/core/src/org_cleanup/mod.rs index 2636dd1d8b..345a799722 100644 --- a/src/core/src/org_cleanup/mod.rs +++ b/src/core/src/org_cleanup/mod.rs @@ -32,6 +32,9 @@ use infra::{ }; use sea_orm::TransactionTrait; +#[cfg(feature = "cloud")] +use crate::{org_domain_ownership, org_domain_requests}; + const LOCK_KEY: &str = "/org_cleanup/worker_lock"; const MAX_ATTEMPTS: i32 = 10; @@ -1034,17 +1037,11 @@ async fn delete_org_offsets_and_caches(org_id: &str) -> Result<(), anyhow::Error Ok(()) } -// The mappings live in the `_meta` org's settings, so the org's own settings delete misses them. #[cfg(feature = "cloud")] async fn delete_org_domain_mappings(org_id: &str) -> Result<(), anyhow::Error> { - // The shared writer serializes this with the settings API and other orgs' cleanups. let org_id = org_id.to_string(); - crate::db::organization::update_org_setting(config::META_ORG_ID, move |settings| { - let before = settings.domain_org_mappings.len(); - settings.domain_org_mappings.retain(|m| m.org_id != org_id); - settings.domain_org_mappings.len() != before - }) - .await?; + org_domain_requests::delete_by_org(&org_id).await?; + org_domain_ownership::delete_by_org(&org_id).await?; Ok(()) } diff --git a/src/core/src/org_domain_ownership.rs b/src/core/src/org_domain_ownership.rs new file mode 100644 index 0000000000..05b91f2822 --- /dev/null +++ b/src/core/src/org_domain_ownership.rs @@ -0,0 +1,288 @@ +// Copyright 2026 OpenObserve Inc. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +use std::sync::{Arc, LazyLock}; + +use hashbrown::HashMap; +use infra::{ + coordinator::get_coordinator, + db::Event, + table::{ + entity::{org_domain_ownership::Model, status_page_custom_domains}, + org_domain_ownership::{OwnershipRecord, OwnershipState}, + }, +}; + +static CACHE: LazyLock> = + LazyLock::new(|| tokio::sync::RwLock::new(HashMap::new())); + +pub const ODO_PREFIX: &str = "/org_domain_ownership/"; +const DOMAIN_EXPIRY_BUFFER_SEC: i64 = 60 * 30; + +pub async fn init() { + let items = match infra::table::org_domain_ownership::list_active_domain_org_map().await { + Ok(v) => v, + Err(e) => { + log::error!("error in listing active domain mapping for init caching : {e}"); + return; + } + }; + let mut lock = CACHE.write().await; + for (domain, org) in items { + lock.insert(domain, org); + } +} + +pub async fn get_cached_org_for_domain(domain: &str) -> Option { + let lock = CACHE.read().await; + let org = lock.get(domain); + org.map(ToOwned::to_owned) +} + +pub async fn remove_domain_from_cache(domain: &str) { + let mut lock = CACHE.write().await; + lock.remove(domain); +} + +pub async fn add_domain_to_cache(domain: &str, org: &str) { + let mut lock = CACHE.write().await; + lock.insert(domain.to_string(), org.to_string()); +} + +pub async fn get_verified_org_for_domain(domain: &str) -> Result, anyhow::Error> { + let lock = CACHE.read().await; + let org = lock.get(domain); + if let Some(org) = org { + return Ok(Some(org.to_owned())); + } + drop(lock); + infra::table::org_domain_ownership::get_verified_org_for_domain(domain).await +} + +pub async fn save_org_domain_mapping(org_id: &str, domain: &str) -> Result<(), anyhow::Error> { + let record = OwnershipRecord { + org_id: org_id.to_owned(), + domain: domain.to_owned(), + state: OwnershipState::Pending, + }; + infra::table::org_domain_ownership::save_org_domain_mapping(record, false).await +} + +pub async fn delete_linked_domain(record: Model) -> Result<(), anyhow::Error> { + infra::table::org_domain_ownership::delete_linked_domain(&record.org_id, &record.domain) + .await?; + // only sync cache for domain which is verified + // non verified domain is not going to be in cache + if record.verification_state == OwnershipState::Verified as i32 { + let mut lock = CACHE.write().await; + let cached = lock.get(&record.domain); + if cached.is_some_and(|v| v == &record.org_id) { + lock.remove(&record.domain); + } + drop(lock); + emit_sync_event(&record.org_id, &record.domain).await; + } + Ok(()) +} + +pub async fn get_domain_org_record( + org_id: &str, + domain: &str, +) -> Result, anyhow::Error> { + let res = infra::table::org_domain_ownership::get_domain_org_record(org_id, domain).await?; + Ok(res) +} + +pub async fn verify(mut record: Model) -> Result { + if record.meta_mapped { + return Ok(record); + } + let res = o2_enterprise::enterprise::status_pages::domain_verifier::check_domain_ownership( + status_page_custom_domains::Model { + domain: record.domain.clone(), + verification_token: record.verification_token.clone(), + last_checked_at: record.last_checked_at, + updated_at: record.updated_at, + verification_state: record.verification_state, + verification_failure_reason: record.verification_failure_reason, + verified_at: record.verified_at, + ..Default::default() + }, + Some(DOMAIN_EXPIRY_BUFFER_SEC), + ) + .await?; + record.last_checked_at = res.last_checked_at; + record.updated_at = res.updated_at; + record.verification_state = res.verification_state; + record.verification_failure_reason = res.verification_failure_reason; + record.verified_at = res.verified_at; + + infra::table::org_domain_ownership::save_domain_org_record(record.clone()).await?; + + Ok(record) +} + +pub async fn verify_domain_now(org_id: &str, domain: &str) -> Result { + let record = infra::table::org_domain_ownership::get_domain_org_record(org_id, domain).await?; + let Some(record) = record else { + return Err(anyhow::anyhow!( + "no mapping for domain {domain} found for org {org_id}" + )); + }; + if record.meta_mapped { + return Ok(record); + } + let updated_record = verify(record).await?; + if updated_record.verification_state == OwnershipState::Verified as i32 { + let mut lock = CACHE.write().await; + lock.insert(domain.to_owned(), org_id.to_owned()); + } + emit_sync_event(org_id, domain).await; + Ok(updated_record) +} + +pub async fn update_state(org_id: &str, domain: &str, new_state: i32) -> Result<(), anyhow::Error> { + let mut lock = CACHE.write().await; + let existing = lock.get(domain); + if new_state == OwnershipState::Verified as i32 { + lock.insert(domain.to_owned(), org_id.to_owned()); + } else if existing.is_some_and(|v| org_id == v) { + lock.remove(domain); + } + drop(lock); + emit_sync_event(org_id, domain).await; + Ok(()) +} + +pub async fn get_domains_for_org(org_id: &str) -> Result, anyhow::Error> { + let records = infra::table::org_domain_ownership::get_domains_for_org(org_id).await?; + Ok(records) +} + +pub async fn link_from_meta(org_id: &str, domain: &str) -> Result<(), anyhow::Error> { + let domains = get_domains_for_org(org_id).await?; + let mapped = domains + .into_iter() + .filter(|v| v.domain == domain) + .next() + .is_some(); + if mapped { + infra::table::org_domain_ownership::mark_meta_mapped(org_id, domain).await?; + } else { + let record = OwnershipRecord { + org_id: org_id.to_owned(), + domain: domain.to_owned(), + state: OwnershipState::Verified, + }; + infra::table::org_domain_ownership::save_org_domain_mapping(record, true).await?; + } + emit_sync_event(org_id, domain).await; + Ok(()) +} + +pub async fn delete_by_org(org_id: &str) -> Result<(), anyhow::Error> { + let domains = infra::table::org_domain_ownership::get_domains_for_org(org_id).await?; + for domain in domains { + delete_linked_domain(domain).await?; + } + Ok(()) +} + +async fn emit_sync_event(org_id: &str, domain: &str) { + let cluster_coordinator = get_coordinator().await; + if let Err(e) = cluster_coordinator + .put( + &format!("{ODO_PREFIX}{org_id}/{domain}"), + "".into(), + true, + None, + ) + .await + { + log::error!( + "error sending cluster sync message for org domain ownership sync for org {org_id} : {e}" + ); + } +} + +pub async fn watch() -> Result<(), anyhow::Error> { + let cluster_coordinator = ::infra::db::get_coordinator().await; + let mut events = cluster_coordinator.watch(ODO_PREFIX).await?; + let events = Arc::get_mut(&mut events).unwrap(); + log::info!("Start watching org_domain_ownership"); + + loop { + let ev = match events.recv().await { + Some(ev) => ev, + None => { + log::error!("watch_org_domain_ownership: event channel closed"); + return Ok(()); + } + }; + + if let Event::Put(ev) = ev { + let Some(key) = ev.key.strip_prefix(ODO_PREFIX) else { + log::error!("unexpected key for org domain prefix watch : {}", ev.key); + continue; + }; + let Some((org, domain)) = key.split_once("/") else { + log::error!( + "invalid key received for org domain ownership sync : {}", + ev.key + ); + continue; + }; + log::info!("received sync event for org domain ownership for org {org}"); + let record = match infra::table::org_domain_ownership::get_domain_org_record( + org, domain, + ) + .await + { + Ok(v) => v, + Err(e) => { + log::error!( + "error in retrieving org domain ownership record from db for {org} domain {domain} : {e}" + ); + continue; + } + }; + let cached = get_cached_org_for_domain(domain).await; + let mut lock = CACHE.write().await; + match record { + None => { + if cached.is_some_and(|v| v == org) { + log::info!( + "removing org {org} domain {domain} from memory cache as record not found in db" + ); + lock.remove(domain); + } + } + Some(v) => { + if v.verification_state == OwnershipState::Verified as i32 { + log::info!( + "added org {org} domain {domain} from memory cache as verified in db" + ); + lock.insert(domain.to_owned(), org.to_owned()); + } else if cached.is_some_and(|v| v == org) { + log::info!( + "removed org {org} domain {domain} from memory cache as not verified in db" + ); + lock.remove(domain); + } + } + } + } + } +} diff --git a/src/core/src/org_domain_requests.rs b/src/core/src/org_domain_requests.rs new file mode 100644 index 0000000000..c8b3d7d337 --- /dev/null +++ b/src/core/src/org_domain_requests.rs @@ -0,0 +1,72 @@ +// Copyright 2026 OpenObserve Inc. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +use config::utils::time::now_micros; +use infra::table::{ + entity::org_domain_requests::Model, + org_domain_requests::{self, Status}, +}; + +pub async fn save_org_domain_request( + org_id: &str, + domain: &str, + req_by: &str, +) -> Result<(), anyhow::Error> { + let now = now_micros(); + let req = Model { + id: config::ider::generate(), + org_id: org_id.to_string(), + domain: domain.trim().to_lowercase(), + created_at: now, + updated_at: now, + requested_by: req_by.to_string(), + answered_by: None, + status: Status::Pending as i32, + reason: None, + }; + org_domain_requests::save_org_domain_request(req).await?; + Ok(()) +} + +pub async fn list_all_req() -> Result, anyhow::Error> { + Ok(org_domain_requests::list_all().await?) +} + +pub async fn list_by_org(org_id: &str) -> Result, anyhow::Error> { + Ok(org_domain_requests::list_by_org(org_id).await?) +} +pub async fn get_req_by_id(id: &str) -> Result, anyhow::Error> { + Ok(org_domain_requests::get_by_id(id).await?) +} + +pub async fn get_req_by_org_domain( + org_id: &str, + domain: &str, +) -> Result, anyhow::Error> { + Ok(org_domain_requests::get_by_org_domain(org_id, domain).await?) +} + +pub async fn update_status( + id: &str, + user: &str, + new_status: Status, + reason: String, +) -> Result<(), anyhow::Error> { + Ok(org_domain_requests::update_status(id, user, new_status, reason).await?) +} + +pub async fn delete_by_org(org_id: &str) -> Result<(), anyhow::Error> { + org_domain_requests::delete_by_org(org_id).await +} diff --git a/src/core/src/organization.rs b/src/core/src/organization.rs index 5762f683fd..831b06acb8 100644 --- a/src/core/src/organization.rs +++ b/src/core/src/organization.rs @@ -25,6 +25,8 @@ use config::{ }, utils::{json, rand::generate_random_string, time}, }; +#[cfg(feature = "cloud")] +use db::organization::get_org_setting; use db::{self, org_users, user::is_root_user}; use infra::table::{self, org_users::UserOrgExpandedRecord}; #[cfg(feature = "enterprise")] @@ -46,6 +48,8 @@ use { #[cfg(feature = "cloud")] use super::self_reporting::cloud_events::{CloudEvent, EventType, enqueue_cloud_event}; +#[cfg(feature = "cloud")] +use crate::org_domain_ownership::get_cached_org_for_domain; use crate::{ auth::{delete_org_tuples, save_org_tuples}, common::{ @@ -1103,6 +1107,21 @@ pub async fn generate_invitation( return Err(anyhow::anyhow!("Email Domain not allowed for {invitee}")); } + let Some((_, domain)) = invitee.split_once('@') else { + return Err(anyhow::anyhow!("invalid email id {invitee}")); + }; + + if let Some(linked_org) = get_cached_org_for_domain(domain).await { + let settings = get_org_setting(&linked_org).await?; + for mapping in settings.domain_org_mappings { + if mapping.domain == domain && mapping.org_id == org_id { + return Err(anyhow::anyhow!( + "invitee '{invitee}' domain is already mapped to org {org_id} so cannot send invite" + )); + } + } + } + if get_billing_by_org_id(org_id).await?.is_none() { // If the org we are inviting to is paid, its fine to send invitations // irrespective of what other orgs invitees are part of. diff --git a/src/core/src/public_dashboards/defaults.rs b/src/core/src/public_dashboards/defaults.rs index 73466de1c1..a52dd85e17 100644 --- a/src/core/src/public_dashboards/defaults.rs +++ b/src/core/src/public_dashboards/defaults.rs @@ -113,6 +113,12 @@ impl<'a> Defaults<'a> { self.values } + /// Whether any value waits for a values query. + #[cfg(feature = "enterprise")] + pub fn has_pending(&self) -> bool { + !self.pending.is_empty() + } + /// The values queries `from` reaches through their stream, field and filter references. #[cfg_attr(not(feature = "enterprise"), allow(dead_code))] pub fn reachable_queries( diff --git a/src/core/src/public_dashboards/mask.rs b/src/core/src/public_dashboards/mask.rs index 734a0eeae0..8f02ecfe7f 100644 --- a/src/core/src/public_dashboards/mask.rs +++ b/src/core/src/public_dashboards/mask.rs @@ -236,6 +236,7 @@ mod tests { #[test] fn gate02_a_variable_from_a_lost_values_stream_is_nulled() { let var = |label: &str, value| PublicVariable { + name: label.into(), label: label.into(), value, tab_id: None, diff --git a/src/core/src/public_dashboards/materialize.rs b/src/core/src/public_dashboards/materialize.rs index 575ed1bd7e..af1c283f7c 100644 --- a/src/core/src/public_dashboards/materialize.rs +++ b/src/core/src/public_dashboards/materialize.rs @@ -1457,6 +1457,7 @@ pub(super) mod tests { Run { panels, variables: vec![PublicVariable { + name: "svc".into(), label: "svc".into(), value: json!("checkout"), tab_id: None, diff --git a/src/core/src/public_dashboards/mod.rs b/src/core/src/public_dashboards/mod.rs index 5583e0fab5..fbf79a2c61 100644 --- a/src/core/src/public_dashboards/mod.rs +++ b/src/core/src/public_dashboards/mod.rs @@ -352,9 +352,7 @@ pub struct LinkValues { impl LinkValues { pub fn new(dash: &Dashboard, frozen: Option<&str>) -> Self { let values = Defaults::new(dash, parse_frozen_vars(frozen)).into_values(); - Self { - vars: QueryVars::new(dash, values, None), - } + Self::from_values(dash, values) } /// The panel's query text as its build substitutes it; the window only sizes `$__interval`. @@ -376,6 +374,12 @@ impl LinkValues { let vars = self.vars.for_panel(tab_id, &panel.id); plan_promql(panel, &vars, now - 3_600_000_000, now) } + + fn from_values(dash: &Dashboard, values: BTreeMap) -> Self { + Self { + vars: QueryVars::new(dash, values, None), + } + } } /// One authorizer health answer per build or deadline check, asked only when a check denies. @@ -808,7 +812,14 @@ pub async fn update_link( let now = now_micros(); validate_dashboard(&dash, &cfg.frozen_variables, now).map_err(Invalid)?; #[cfg(feature = "enterprise")] - authorize_variable_edit(&link, &dash, &cfg.frozen_variables, user_id).await?; + authorize_variable_edit( + &link, + &dash, + &cfg.frozen_variables, + range_changed(&link, &cfg.time_range).then_some(cfg.time_range.ranges.as_slice()), + user_id, + ) + .await?; let ranges = serde_json::to_string(&cfg.time_range.ranges)?; let frozen = serde_json::to_string(&cfg.frozen_variables)?; edit(&link.org_id, &link.id, |l| { @@ -1508,15 +1519,29 @@ where paused } -/// Enterprise: the editor must read every stream a changed frozen value makes a panel read. +/// Whether an edit changes the set of ranges a link serves or its default range. +#[cfg(feature = "enterprise")] +fn range_changed(link: &PublicDashboard, policy: &TimeRangePolicy) -> bool { + let keys = |ranges: &[TimeRange]| ranges.iter().map(TimeRange::key).collect::>(); + let stored = time_ranges(link); + keys(&stored.ranges) != keys(&policy.ranges) || stored.default.key() != policy.default.key() +} + +/// Enterprise: the editor must read what a changed frozen value or range makes the link read. #[cfg(feature = "enterprise")] async fn authorize_variable_edit( link: &PublicDashboard, dash: &Dashboard, frozen: &BTreeMap, + changed_ranges: Option<&[TimeRange]>, user_id: &str, ) -> Result<(), anyhow::Error> { - let sources = variable_edit_sources(link, dash, frozen); + let sources = match changed_ranges { + Some(ranges) => { + range_edit_sources(&ENGINE, link, dash, frozen, ranges, user_id, now_micros()).await? + } + None => variable_edit_sources(link, dash, frozen), + }; if sources.is_empty() { return Ok(()); } @@ -1560,57 +1585,190 @@ fn variable_edit_sources( let values = LinkValues::new(dash, new_json.as_deref()); for tab in &v8.tabs { for panel in &tab.panels { - let texts = template_texts(panel); - let texts: Vec<&str> = texts.iter().map(String::as_str).collect(); - let reads_changed = values - .vars - .referenced(&tab.tab_id, &panel.id, &texts) - .iter() - .any(|(_, key)| reached.contains(key)); - if !reads_changed { - continue; - } - let promql = panel.query_type == "promql"; - let planned = promql - .then(|| values.plan_promql(&tab.tab_id, panel).ok()) - .flatten(); - for (qi, q) in panel.queries.iter().enumerate() { - let lang = if promql { "promql" } else { "sql" }; - // A sent PromQL query is checked as the build sends it, a formula as its expansion. - let text = match planned.as_ref().and_then(|p| p[qi].clone()) { - Some(expr) => expr, - None => { - let Some(text) = query_text(q) else { - continue; - }; - values.substitute(&tab.tab_id, &panel.id, text, lang) - } - }; - sources.push(if promql { - QuerySource::PromQl { - org_id: link.org_id.clone(), - query: text, - } - } else { - QuerySource::Sql { - org_id: link.org_id.clone(), - sql: text, - default_type: q.fields.stream_type, - } - }); - let vrl = q.vrl_function_query.as_deref().map(str::trim); - if let Some(vrl) = vrl.filter(|f| !promql && !f.is_empty()) { - sources.push(QuerySource::Vrl { - org_id: link.org_id.clone(), - source: vrl.to_string(), - }); - } + if panel_keys(&values, tab, panel).any(|key| reached.contains(&key)) { + sources.extend(panel_sources(&link.org_id, &values, tab, panel)); } } } + dedup(sources) +} + +/// Every query the link runs in each range, with its values queries run as the editor. +#[cfg(feature = "enterprise")] +async fn range_edit_sources( + search: &dyn PanelSearch, + link: &PublicDashboard, + dash: &Dashboard, + frozen: &BTreeMap, + ranges: &[TimeRange], + user_id: &str, + now: i64, +) -> Result, anyhow::Error> { + use crate::authz::QuerySource; + + let Some(v8) = dash.v8.as_ref() else { + return Ok(vec![]); + }; + let editor = PublicDashboard { + published_by: user_id.to_string(), + ..link.clone() + }; + let probe = AuthorizerProbe::default(); + let family = TraceFamily::new(&link.org_id); + let mut sources = Vec::new(); + // Values depend on the window only when a values query picks them. + let windows: Vec<&TimeRange> = if Defaults::new(dash, frozen.clone()).has_pending() { + ranges.iter().collect() + } else { + ranges.iter().take(1).collect() + }; + for range in windows { + let range_key = range.key(); + let (start, end) = materialize::window(range, now); + let ctx = Ctx { + search, + probe: &probe, + link: &editor, + dash, + identity: public_dashboard_identity(&link.org_id), + range_key: &range_key, + start, + end, + now, + family: &family, + }; + let filled = fill_defaults(&ctx, frozen.clone()).await; + sources.extend(filled.deps.values().flat_map(|dep| { + dep.sources + .iter() + .map(|(stream_type, name)| QuerySource::Stream { + org_id: link.org_id.clone(), + stream_type: *stream_type, + name: name.clone(), + }) + })); + if let Some(key) = filled.denied.first() { + return Err(Forbidden(format!( + "Unauthorized Access: no read permission for the values of variable {key}" + )) + .into()); + } + // A later build may resolve the value, so an unchecked panel must not be saved. + if let Some(key) = filled.failed.first() { + return Err(Invalid(format!( + "The values of variable {key} could not be loaded to check stream access for this range. Try again." + )) + .into()); + } + let values = LinkValues::from_values(dash, filled.values); + for tab in &v8.tabs { + for panel in &tab.panels { + sources.extend(panel_sources(&link.org_id, &values, tab, panel)); + } + } + } + Ok(dedup(sources)) +} + +/// Keys of the scoped values the panel's queries read. +#[cfg(feature = "enterprise")] +fn panel_keys<'a>( + values: &'a LinkValues, + tab: &'a Tab, + panel: &'a Panel, +) -> impl Iterator + 'a { + let texts = template_texts(panel); + let texts: Vec<&str> = texts.iter().map(String::as_str).collect(); + values + .vars + .referenced(&tab.tab_id, &panel.id, &texts) + .into_iter() + .map(|(_, key)| key) +} + +/// The queries and VRL functions a panel's build sends; a custom chart never runs. +#[cfg(feature = "enterprise")] +fn panel_sources( + org_id: &str, + values: &LinkValues, + tab: &Tab, + panel: &Panel, +) -> Vec { + use crate::authz::QuerySource; + + let mut sources = Vec::new(); + if panel.typ == "custom_chart" || withholds_vrl(org_id, panel) { + return sources; + } + let promql = panel.query_type == "promql"; + let planned = promql + .then(|| values.plan_promql(&tab.tab_id, panel).ok()) + .flatten(); + for (qi, q) in panel.queries.iter().enumerate() { + let lang = if promql { "promql" } else { "sql" }; + // A sent PromQL query is checked as the build sends it, a formula as its expansion. + let text = match planned.as_ref().map(|p| p[qi].clone()) { + Some(Some(expr)) => expr, + // A hidden or empty query is never sent; formulas carry their inputs. + Some(None) => continue, + None => { + let Some(text) = query_text(q) else { + continue; + }; + values.substitute(&tab.tab_id, &panel.id, text, lang) + } + }; + sources.push(if promql { + QuerySource::PromQl { + org_id: org_id.to_string(), + query: text, + } + } else { + QuerySource::Sql { + org_id: org_id.to_string(), + sql: text, + default_type: q.fields.stream_type, + } + }); + let vrl = q.vrl_function_query.as_deref().map(str::trim); + if let Some(vrl) = vrl.filter(|f| !promql && !f.is_empty()) { + sources.push(QuerySource::Vrl { + org_id: org_id.to_string(), + source: vrl.to_string(), + }); + } + } sources } +/// Whether the build withholds the panel for a VRL whose reads it cannot prove, as `run_sql_panel` +/// does. +#[cfg(feature = "enterprise")] +fn withholds_vrl(org_id: &str, panel: &Panel) -> bool { + panel.query_type != "promql" + && panel.queries.iter().any(|q| { + q.vrl_function_query + .as_deref() + .map(str::trim) + .filter(|f| !f.is_empty()) + .is_some_and(|vrl| { + transform::enrichment_tables_read(vrl, org_id).is_err() + || skips_vrl_in_search(vrl) + }) + }) +} + +#[cfg(feature = "enterprise")] +fn dedup(sources: Vec) -> Vec { + let mut unique = Vec::with_capacity(sources.len()); + for source in sources { + if !unique.contains(&source) { + unique.push(source); + } + } + unique +} + /// The outermost explicit LIMIT of a SQL query; None without one or when it does not parse. fn explicit_limit(sql: &str) -> Option { use sqlparser::{ @@ -4208,6 +4366,113 @@ mod tests { ); } + #[cfg(feature = "enterprise")] + #[test] + fn only_a_new_range_set_or_default_is_a_range_change() { + let (mut link, ..) = ctx_parts(); + link.time_ranges = Some( + json!([{ "type": "relative", "secs": 86400 }, { "type": "relative", "secs": 3600 }]) + .to_string(), + ); + link.default_range_key = Some("r86400".into()); + let policy = |ranges: serde_json::Value, default: usize| -> TimeRangePolicy { + let ranges: Vec = serde_json::from_value(ranges).unwrap(); + TimeRangePolicy { + default: ranges[default], + ranges, + } + }; + let sorted = + json!([{ "type": "relative", "secs": 3600 }, { "type": "relative", "secs": 86400 }]); + assert!( + !range_changed(&link, &policy(sorted.clone(), 1)), + "a reorder" + ); + assert!(range_changed(&link, &policy(sorted, 0)), "a new default"); + assert!(range_changed( + &link, + &policy(json!([{ "type": "relative", "secs": 2592000 }]), 0) + )); + } + + #[cfg(feature = "enterprise")] + #[tokio::test] + async fn a_range_edit_checks_every_panel_with_the_values_its_build_picks() { + let (link, ..) = ctx_parts(); + let dash = Dashboard { + version: 8, + v8: Some( + serde_json::from_value(json!({ + "tabs": [{ "tabId": "t1", "name": "", "panels": [ + panel("b", Some("SELECT * FROM fixed WHERE env = '$env'")), + panel("d", Some("SELECT * FROM \"$pick\"")), + panel("e", Some("SELECT * FROM \"$sub\"")), + panel("f", Some("SELECT * FROM \"$sub\"")), + typed_panel("g", "custom_chart", "sql", "SELECT * FROM never_run", None), + ]}], + "variables": { "list": [ + { "type": "custom", "name": "env", "label": "E", "value": "prod" }, + { "type": "query_values", "name": "pick", "label": "P", + "query_data": { "stream_type": "logs", "stream": "names", "field": "n" } }, + { "type": "query_values", "name": "sub", "label": "U", + "query_data": { "stream_type": "logs", "stream": "$pick", "field": "n" } } + ]} + })) + .unwrap(), + ), + ..Default::default() + }; + let frozen: BTreeMap = + serde_json::from_value(json!({ "env": "prod" })).unwrap(); + let ranges: Vec = + serde_json::from_value(json!([{ "type": "relative", "secs": 3600 }])).unwrap(); + let search = FakeSearch { + hits: 1, + ..Default::default() + }; + let sources = range_edit_sources(&search, &link, &dash, &frozen, &ranges, "editor@x", 0) + .await + .unwrap(); + let mut texts: Vec = sources + .into_iter() + .map(|s| match s { + crate::authz::QuerySource::Sql { sql, .. } => sql, + crate::authz::QuerySource::Stream { name, .. } => format!("stream {name}"), + other => panic!("{other:?}"), + }) + .collect(); + texts.sort(); + assert_eq!( + texts, + [ + "SELECT * FROM \"v0\"", + "SELECT * FROM fixed WHERE env = 'prod'", + "stream names", + "stream v0", + ], + "the dependent values query reads the stream its parent picks" + ); + let denied = FakeSearch { + hits: 1, + refuse: Some((NotAvailableReason::Unauthorized, "FROM \"names\"")), + ..Default::default() + }; + let err = range_edit_sources(&denied, &link, &dash, &frozen, &ranges, "editor@x", 0) + .await + .unwrap_err(); + assert!(err.downcast_ref::().is_some(), "{err}"); + // A failed lookup leaves the dependent panels unchecked, so the edit cannot be saved. + let failed = FakeSearch { + hits: 1, + fail_sql: Some("FROM \"names\""), + ..Default::default() + }; + let err = range_edit_sources(&failed, &link, &dash, &frozen, &ranges, "editor@x", 0) + .await + .unwrap_err(); + assert!(err.downcast_ref::().is_some(), "{err}"); + } + #[cfg(all(feature = "enterprise", feature = "test-utils"))] #[tokio::test] async fn gate04_a_link_variable_edit_selecting_an_unreadable_stream_is_refused() { @@ -4226,7 +4491,7 @@ mod tests { let to = |value: &str| -> BTreeMap { BTreeMap::from([("src.p.c".to_string(), json!(value))]) }; - let err = authorize_variable_edit(&link, &dash, &to("secret"), &editor) + let err = authorize_variable_edit(&link, &dash, &to("secret"), None, &editor) .await .unwrap_err(); assert!(err.downcast_ref::().is_some()); @@ -4235,7 +4500,7 @@ mod tests { "Unauthorized Access: no read permission on logs/secret" ); assert!( - authorize_variable_edit(&link, &dash, &to("safe"), &editor) + authorize_variable_edit(&link, &dash, &to("safe"), None, &editor) .await .is_ok() ); diff --git a/src/core/src/self_profiles/job.rs b/src/core/src/self_profiles/job.rs index e393098aaa..589125cb63 100644 --- a/src/core/src/self_profiles/job.rs +++ b/src/core/src/self_profiles/job.rs @@ -48,9 +48,10 @@ pub async fn run() { let cpu_secs = cfg.self_profiles.cpu_secs; let lock = Arc::new(Mutex::new(())); log::info!( - "[SELF-PROFILES] starting background loop: interval={}s cpu={}s url={}", + "[SELF-PROFILES] starting background loop: interval={}s cpu={}s memory={} url={}", cfg.self_profiles.interval_secs, cpu_secs, + cfg!(feature = "profiling"), if cfg.self_profiles.url.is_empty() { "local" } else { @@ -73,32 +74,38 @@ pub async fn run() { } async fn run_cycle(cpu_secs: u64) -> Result<(), String> { - let host = config::get_config().common.instance_name.clone(); - let extra = [("host.name", host.as_str())]; + let cfg = config::get_config(); + let extra = [ + ("host.name", cfg.common.instance_name.as_str()), + ("cluster.name", cfg.common.cluster_name.as_str()), + ]; - match sample::dump_cpu_profile(cpu_secs, CPU_FREQUENCY_HZ).await { - Ok(bytes) => match convert::pprof_bytes_to_otlp(&bytes, &extra) { - Ok(req) => { - if let Err(e) = sink::ingest(req).await { - log::error!("[SELF-PROFILES] CPU ingest failed: {e}"); - } - } - Err(e) => log::error!("[SELF-PROFILES] CPU convert failed: {e}"), - }, - Err(e) => log::error!("[SELF-PROFILES] CPU sample failed: {e}"), - } + let cpu = sample::dump_cpu_profile(cpu_secs, CPU_FREQUENCY_HZ).await; + ingest_sample("CPU", cpu, &extra).await; - match sample::dump_memory_pprof().await { - Ok(bytes) => match convert::pprof_bytes_to_otlp(&bytes, &extra) { - Ok(req) => { - if let Err(e) = sink::ingest(req).await { - log::error!("[SELF-PROFILES] memory ingest failed: {e}"); - } - } - Err(e) => log::error!("[SELF-PROFILES] memory convert failed: {e}"), - }, - Err(e) => log::error!("[SELF-PROFILES] memory sample failed: {e}"), + #[cfg(feature = "profiling")] + { + let memory = sample::dump_memory_pprof().await; + ingest_sample("memory", memory, &extra).await; } Ok(()) } + +async fn ingest_sample(kind: &str, sampled: Result, String>, extra: &[(&str, &str)]) { + let bytes = match sampled { + Ok(bytes) => bytes, + Err(e) => { + log::error!("[SELF-PROFILES] {kind} sample failed: {e}"); + return; + } + }; + match convert::pprof_bytes_to_otlp(&bytes, extra) { + Ok(req) => { + if let Err(e) = sink::ingest(req).await { + log::error!("[SELF-PROFILES] {kind} ingest failed: {e}"); + } + } + Err(e) => log::error!("[SELF-PROFILES] {kind} convert failed: {e}"), + } +} diff --git a/src/core/src/self_profiles/mod.rs b/src/core/src/self_profiles/mod.rs index 2d3e782cc4..090248efed 100644 --- a/src/core/src/self_profiles/mod.rs +++ b/src/core/src/self_profiles/mod.rs @@ -13,7 +13,7 @@ // You should have received a copy of the GNU Affero General Public License // along with this program. If not, see . -//! Self CPU/memory profiling into `_meta.self_profiles` (feature `profiling`). +//! Self CPU (feature `cpu-profiling`) and memory (feature `profiling`) profiling. mod convert; mod job; @@ -21,8 +21,9 @@ mod sample; mod sink; pub use job::run; -pub use sample::{dump_cpu_profile, dump_memory_flamegraph, dump_memory_pprof, jemalloc_stats}; +pub use sample::dump_cpu_profile; +#[cfg(feature = "profiling")] +pub use sample::{dump_memory_flamegraph, dump_memory_pprof, jemalloc_stats}; -pub(crate) const STREAM_NAME: &str = "self_profiles"; pub(crate) const SERVICE_NAME: &str = "openobserve"; pub(crate) const CPU_FREQUENCY_HZ: i32 = 100; diff --git a/src/core/src/self_profiles/sample.rs b/src/core/src/self_profiles/sample.rs index 76cdc86588..0d1c1af677 100644 --- a/src/core/src/self_profiles/sample.rs +++ b/src/core/src/self_profiles/sample.rs @@ -33,6 +33,7 @@ pub async fn dump_cpu_profile(duration_secs: u64, frequency: i32) -> Result Result, String> { let prof_ctl = jemalloc_pprof::PROF_CTL .as_ref() @@ -52,6 +53,7 @@ pub async fn dump_memory_pprof() -> Result, String> { } /// Dump jemalloc heap flamegraph SVG bytes (for `/debug/profile/memory`). +#[cfg(feature = "profiling")] pub async fn dump_memory_flamegraph() -> Result, String> { let prof_ctl = jemalloc_pprof::PROF_CTL .as_ref() @@ -71,6 +73,7 @@ pub async fn dump_memory_flamegraph() -> Result, String> { } /// Read current jemalloc stats (for `/debug/profile/stats`). +#[cfg(feature = "profiling")] pub fn jemalloc_stats() -> Result { tikv_jemalloc_ctl::epoch::mib() .map_err(|e| format!("Failed to get jemalloc epoch mib: {e}"))? diff --git a/src/core/src/self_profiles/sink.rs b/src/core/src/self_profiles/sink.rs index 92b6d81d3f..d31bd64b42 100644 --- a/src/core/src/self_profiles/sink.rs +++ b/src/core/src/self_profiles/sink.rs @@ -15,11 +15,11 @@ use std::time::Duration; -use config::{META_ORG_ID, get_config, meta::otlp::OtlpRequestType}; +use config::{DEFAULT_STREAM_NAME, META_ORG_ID, get_config, meta::otlp::OtlpRequestType}; use ingestion_common::{IngestUser, SystemJobType}; use opentelemetry_proto::tonic::collector::profiles::v1development::ExportProfilesServiceRequest; -use super::{STREAM_NAME, convert}; +use super::convert; const REMOTE_REQUEST_TIMEOUT: Duration = Duration::from_secs(10); const REMOTE_CONNECT_TIMEOUT: Duration = Duration::from_secs(5); @@ -40,7 +40,7 @@ async fn ingest_local(request: ExportProfilesServiceRequest) -> Result<(), Strin META_ORG_ID, request, OtlpRequestType::HttpProtobuf, - Some(STREAM_NAME), + Some(DEFAULT_STREAM_NAME), IngestUser::SystemJob(SystemJobType::SelfReporting), ) .await diff --git a/src/core/src/traces/service_graph/api.rs b/src/core/src/traces/service_graph/api.rs index ce39ce0b0d..ac4ab8a9c5 100644 --- a/src/core/src/traces/service_graph/api.rs +++ b/src/core/src/traces/service_graph/api.rs @@ -21,6 +21,17 @@ use axum::response::Response as HttpResponse; use common::meta::http::HttpResponse as MetaHttpResponse; use serde::Deserialize; +#[cfg(feature = "enterprise")] +const EDGE_HISTORY_WINDOW_MICROS: i64 = 24 * 60 * 60 * 1_000_000; +// Allows clock skew while keeping every bound far from i64 overflow in the step alignment. +#[cfg(any(feature = "enterprise", test))] +const MAX_FUTURE_MICROS: i64 = 24 * 60 * 60 * 1_000_000; +// A stream's own retention can outlast the global one, so the cap never drops below the default. +#[cfg(any(feature = "enterprise", test))] +const MIN_MAX_WINDOW_DAYS: i64 = 3650; +#[cfg(any(feature = "enterprise", test))] +const DAY_MICROS: i64 = 24 * 60 * 60 * 1_000_000; + /// Query parameters for service graph API #[derive(Debug, Deserialize)] pub struct ServiceGraphQuery { @@ -53,6 +64,7 @@ pub struct ServiceGraphQuery { ), responses( (status = 200, description = "Success", content_type = "application/json", body = Object), + (status = 400, description = "Invalid time window (negative start_time, end_time not after start_time, end_time more than a day ahead, or longer than the data retention, at least 3650 days) or a stream name the metrics query cannot quote", content_type = "application/json", body = String), (status = 403, description = "Forbidden - Enterprise feature", content_type = "application/json", body = String), (status = 500, description = "Internal Server Error", content_type = "application/json", body = ()), ), @@ -65,15 +77,16 @@ pub async fn get_current_topology( axum::extract::Path(org_id): axum::extract::Path, axum::extract::Query(query): axum::extract::Query, ) -> HttpResponse { - // Resolve the current window boundaries - let (start_time, end_time) = - if let (Some(start), Some(end)) = (query.start_time, query.end_time) { - (start, end) - } else { - let now = chrono::Utc::now().timestamp_micros(); - let window_micros = super::DEFAULT_QUERY_WINDOW_MINUTES * 60 * 1_000_000; - (now - window_micros, now) - }; + let (start_time, end_time) = match resolve_window( + query.start_time, + query.end_time, + chrono::Utc::now().timestamp_micros(), + super::DEFAULT_QUERY_WINDOW_MINUTES * 60 * 1_000_000, + max_window_days(config::get_config().compact.data_retention_days), + ) { + Ok(window) => window, + Err(e) => return MetaHttpResponse::bad_request(e), + }; let filter = v4_read_filter(&query); if let Err(e) = filter.validate() { @@ -152,7 +165,7 @@ pub struct EdgeHistoryQuery { ), responses( (status = 200, description = "Success", content_type = "application/json", body = Object), - (status = 400, description = "A service or stream name the metrics query cannot quote", content_type = "application/json", body = String), + (status = 400, description = "Invalid time window (negative start_time, end_time not after start_time, end_time more than a day ahead, or longer than the data retention, at least 3650 days), or a service or stream name the metrics query cannot quote", content_type = "application/json", body = String), (status = 403, description = "Forbidden - Enterprise feature"), (status = 500, description = "Internal Server Error", content_type = "application/json", body = ()), ), @@ -166,13 +179,16 @@ pub async fn get_edge_history( use super::v4::history::{EdgeFilter, fetch_edge_history}; - let (start_time, end_time) = - if let (Some(start), Some(end)) = (query.start_time, query.end_time) { - (start, end) - } else { - let now = chrono::Utc::now().timestamp_micros(); - (now - 24 * 60 * 60 * 1_000_000, now) - }; + let (start_time, end_time) = match resolve_window( + query.start_time, + query.end_time, + chrono::Utc::now().timestamp_micros(), + EDGE_HISTORY_WINDOW_MICROS, + max_window_days(config::get_config().compact.data_retention_days), + ) { + Ok(window) => window, + Err(e) => return MetaHttpResponse::bad_request(e), + }; let non_empty = |v: Option| v.filter(|s| !s.is_empty()); let filter = EdgeFilter { client: non_empty(query.client_service), @@ -228,10 +244,137 @@ pub async fn get_current_topology( MetaHttpResponse::forbidden("Not Supported") } -#[cfg(all(test, feature = "enterprise"))] +/// The longest window a request may ask for: the global data retention, at least 3650 days. +#[cfg(any(feature = "enterprise", test))] +fn max_window_days(retention_days: i64) -> i64 { + retention_days.max(MIN_MAX_WINDOW_DAYS) +} + +#[cfg(any(feature = "enterprise", test))] +fn resolve_window( + start: Option, + end: Option, + now: i64, + default_span_micros: i64, + max_days: i64, +) -> Result<(i64, i64), String> { + let (Some(start), Some(end)) = (start, end) else { + return Ok((now - default_span_micros, now)); + }; + if start < 0 { + return Err("start_time must not be negative".to_string()); + } + if end <= start { + return Err("end_time must be after start_time".to_string()); + } + if end > now.saturating_add(MAX_FUTURE_MICROS) { + return Err("end_time must be at most one day after the current time".to_string()); + } + if end - start > max_days.saturating_mul(DAY_MICROS) { + return Err(format!( + "the window from start_time to end_time must not exceed {max_days} days" + )); + } + Ok((start, end)) +} + +#[cfg(test)] mod tests { use super::*; + const NOW: i64 = 1_800_000_000_000_000; + const SPAN: i64 = 3_600_000_000; + const MAX_DAYS: i64 = 3650; + + #[test] + fn test_resolve_window_uses_an_explicit_window() { + assert_eq!( + resolve_window(Some(1_000), Some(2_000), NOW, SPAN, MAX_DAYS), + Ok((1_000, 2_000)) + ); + assert_eq!( + resolve_window(Some(0), Some(1), NOW, SPAN, MAX_DAYS), + Ok((0, 1)) + ); + } + + #[test] + fn test_resolve_window_rejects_a_negative_start() { + assert!(resolve_window(Some(-1), Some(2_000), NOW, SPAN, MAX_DAYS).is_err()); + assert!(resolve_window(Some(i64::MIN), Some(i64::MAX), NOW, SPAN, MAX_DAYS).is_err()); + } + + #[test] + fn test_resolve_window_rejects_an_end_not_after_the_start() { + assert!(resolve_window(Some(2_189_567), Some(-1_358), NOW, SPAN, MAX_DAYS).is_err()); + assert!(resolve_window(Some(2_000), Some(2_000), NOW, SPAN, MAX_DAYS).is_err()); + assert!(resolve_window(Some(2_000), Some(1_999), NOW, SPAN, MAX_DAYS).is_err()); + } + + #[test] + fn test_resolve_window_rejects_an_end_beyond_tomorrow_and_near_i64_max() { + assert_eq!( + resolve_window( + Some(NOW), + Some(NOW + MAX_FUTURE_MICROS), + NOW, + SPAN, + MAX_DAYS + ), + Ok((NOW, NOW + MAX_FUTURE_MICROS)) + ); + assert!( + resolve_window( + Some(NOW), + Some(NOW + MAX_FUTURE_MICROS + 1), + NOW, + SPAN, + MAX_DAYS + ) + .is_err() + ); + assert!(resolve_window(Some(i64::MAX - 1), Some(i64::MAX), NOW, SPAN, MAX_DAYS).is_err()); + assert!(resolve_window(Some(0), Some(i64::MAX), NOW, SPAN, MAX_DAYS).is_err()); + assert!(resolve_window(Some(0), Some(1), i64::MAX, SPAN, MAX_DAYS).is_ok()); + } + + #[test] + fn test_resolve_window_caps_the_span_at_the_given_days() { + let max_span = MAX_DAYS * DAY_MICROS; + assert_eq!( + resolve_window(Some(NOW - max_span), Some(NOW), NOW, SPAN, MAX_DAYS), + Ok((NOW - max_span, NOW)) + ); + let err = + resolve_window(Some(NOW - max_span - 1), Some(NOW), NOW, SPAN, MAX_DAYS).unwrap_err(); + assert!(err.contains("3650 days"), "{err}"); + let longer = 7300 * DAY_MICROS; + assert!(resolve_window(Some(NOW - longer), Some(NOW), NOW, SPAN, 7300).is_ok()); + assert!(resolve_window(Some(0), Some(NOW), NOW, SPAN, i64::MAX).is_ok()); + } + + #[test] + fn test_max_window_days_follows_a_longer_retention() { + assert_eq!(max_window_days(7300), 7300); + assert_eq!(max_window_days(3650), 3650); + assert_eq!( + max_window_days(30), + 3650, + "a stream may keep data longer than the default" + ); + assert_eq!(max_window_days(0), 3650, "0 means unbounded retention"); + assert_eq!(max_window_days(-1), 3650); + } + + #[test] + fn test_resolve_window_defaults_when_a_bound_is_missing() { + let default = Ok((NOW - SPAN, NOW)); + assert_eq!(resolve_window(None, None, NOW, SPAN, MAX_DAYS), default); + assert_eq!(resolve_window(Some(-5), None, NOW, SPAN, MAX_DAYS), default); + assert_eq!(resolve_window(None, Some(-5), NOW, SPAN, MAX_DAYS), default); + } + + #[cfg(feature = "enterprise")] #[test] fn test_v4_read_filter_maps_stream_and_env_only() { let query = |stream: Option<&str>, env: Option<&str>| ServiceGraphQuery { @@ -253,4 +396,69 @@ mod tests { assert_eq!(f.trace_stream.as_deref(), Some("default")); assert_eq!(f.agent_env.as_deref(), Some("prod")); } + + #[cfg(feature = "enterprise")] + #[tokio::test] + async fn test_handlers_reject_an_invalid_window_with_400() { + use axum::extract::{Path, Query}; + + let topology = get_current_topology( + Path("default".to_string()), + Query(ServiceGraphQuery { + start_time: Some(2_189_567), + end_time: Some(-1_358), + filter: None, + stream_name: None, + agent_id: None, + agent_name: None, + agent_env: None, + }), + ) + .await; + assert_eq!(topology.status(), axum::http::StatusCode::BAD_REQUEST); + + let history = get_edge_history( + Path("default".to_string()), + Query(EdgeHistoryQuery { + client_service: None, + server_service: None, + start_time: Some(-1), + end_time: Some(10), + stream_name: None, + }), + ) + .await; + assert_eq!(history.status(), axum::http::StatusCode::BAD_REQUEST); + + let near_max = get_edge_history( + Path("default".to_string()), + Query(EdgeHistoryQuery { + client_service: None, + server_service: None, + start_time: Some(i64::MAX - 1), + end_time: Some(i64::MAX), + stream_name: None, + }), + ) + .await; + assert_eq!(near_max.status(), axum::http::StatusCode::BAD_REQUEST); + + let near_max_topology = get_current_topology( + Path("default".to_string()), + Query(ServiceGraphQuery { + start_time: Some(i64::MAX - 1), + end_time: Some(i64::MAX), + filter: None, + stream_name: None, + agent_id: None, + agent_name: None, + agent_env: None, + }), + ) + .await; + assert_eq!( + near_max_topology.status(), + axum::http::StatusCode::BAD_REQUEST + ); + } } diff --git a/src/core/src/traces/service_graph/v4/history.rs b/src/core/src/traces/service_graph/v4/history.rs index e9b2cf04c0..a817003421 100644 --- a/src/core/src/traces/service_graph/v4/history.rs +++ b/src/core/src/traces/service_graph/v4/history.rs @@ -90,7 +90,7 @@ pub async fn fetch_edge_history( /// Coarser than a minute only when the range would exceed `MAX_POINTS`; always whole minutes. fn step_micros(start: i64, end: i64) -> i64 { - let span_secs = ((end - start) / 1_000_000).max(1) as u64; + let span_secs = (end.saturating_sub(start) / 1_000_000).max(1) as u64; let minutes = span_secs .div_ceil(MAX_POINTS as u64) .div_ceil(MIN_STEP_SECS as u64); @@ -99,7 +99,7 @@ fn step_micros(start: i64, end: i64) -> i64 { /// Evaluation times sit on step boundaries, where the rollup's samples are, inside the range. fn eval_bounds(start: i64, end: i64, step: i64) -> Option<(i64, i64)> { - let first = align_down(start + step - 1, step) + step; + let first = align_down(start.checked_add(step - 1)?, step).checked_add(step)?; let last = align_down(end, step); (first <= last).then_some((first, last)) } @@ -217,6 +217,14 @@ mod tests { assert_eq!(eval_bounds(0, MIN - 1, MIN), None); } + #[test] + fn test_step_and_bounds_survive_windows_near_i64_max() { + let step = step_micros(i64::MAX - 1, i64::MAX); + assert_eq!(step, MIN); + assert_eq!(eval_bounds(i64::MAX - 1, i64::MAX, step), None); + assert!(step_micros(i64::MIN, i64::MAX) > 0); + } + #[test] fn test_plan_queries() { let plan = plan("server=\"b\"", MIN); diff --git a/src/core/src/traces/service_graph/v4/read.rs b/src/core/src/traces/service_graph/v4/read.rs index 42edd7c629..6e487c85fe 100644 --- a/src/core/src/traces/service_graph/v4/read.rs +++ b/src/core/src/traces/service_graph/v4/read.rs @@ -257,7 +257,7 @@ where Fut: Future, anyhow::Error>>, { // Range selectors are left-open; +1ms keeps the `start` sample, not a baseline 1ms before it. - let range_ms = ((end - start) / 1_000).max(1_000) + 1; + let range_ms = (end.saturating_sub(start) / 1_000).max(1_000) + 1; let plan = |v4_only| plan_queries(filter, v4_only, range_ms, start, end).map_err(anyhow::Error::msg); let mut results = run_plan(&plan(true)?, &run).await; diff --git a/src/core/src/users.rs b/src/core/src/users.rs index ba1641cad1..31a6953ec9 100644 --- a/src/core/src/users.rs +++ b/src/core/src/users.rs @@ -1876,12 +1876,7 @@ mod tests { } async fn create_query_history_table() { - use sea_orm::{ConnectionTrait, Schema}; - let conn = get_orm_client_rw().await; - let backend = conn.get_database_backend(); - let mut stmt = Schema::new(backend) - .create_table_from_entity(infra_table::entity::query_history::Entity); - conn.execute(backend.build(stmt.if_not_exists())) + infra_table::query_history::create_table_for_tests(get_orm_client_rw().await) .await .unwrap(); } @@ -1906,14 +1901,24 @@ mod tests { }; create_new_user(user).await.unwrap(); for org in ["dummy", "other"] { - infra_table::query_history::record(org, email, "up", serde_json::json!({}), 1) - .await - .unwrap(); + infra_table::query_history::record(infra_table::query_history::NewRun { + org_id: org, + user_email: email, + source: "logs", + query: "up", + query_key: None, + streams: &[], + status: "ok", + context: serde_json::json!({}), + now_us: 1, + }) + .await + .unwrap(); } } async fn history_count(org: &str, email: &str) -> usize { - infra_table::query_history::list(org, email, None, None, 10, 0) + infra_table::query_history::list(org, email, &Default::default(), 10, 0) .await .unwrap() .len() diff --git a/src/db/src/authz.rs b/src/db/src/authz.rs index e15fa4d278..ce046fac05 100644 --- a/src/db/src/authz.rs +++ b/src/db/src/authz.rs @@ -90,11 +90,31 @@ pub async fn list_objects_for_user( permission: &str, object_type: &str, ) -> anyhow::Result>> { - let openfga_config = o2_openfga::config::get_config(); - if crate::user::is_root_user(user_id) - || !openfga_config.enabled - || !openfga_config.list_only_permitted - { + if !o2_openfga::config::get_config().list_only_permitted { + return Ok(None); + } + list_permitted_objects(org_id, user_id, permission, object_type).await +} + +#[cfg(not(feature = "enterprise"))] +pub async fn list_objects_for_user( + _org_id: &str, + _user_id: &str, + _permission: &str, + _object_type: &str, +) -> anyhow::Result>> { + Ok(None) +} + +/// [`list_objects_for_user`] whatever `list_only_permitted` says, for lists that carry secrets. +#[cfg(feature = "enterprise")] +pub async fn list_permitted_objects( + org_id: &str, + user_id: &str, + permission: &str, + object_type: &str, +) -> anyhow::Result>> { + if crate::user::is_root_user(user_id) || !o2_openfga::config::get_config().enabled { return Ok(None); } @@ -113,20 +133,12 @@ pub async fn list_objects_for_user( ) .await .map_err(|_| anyhow::anyhow!("Unauthorized Access"))?; - log::debug!("list_objects_for_user for user {user_id} from {org_id} org returns: {objects:#?}"); + log::debug!( + "list_permitted_objects for user {user_id} from {org_id} org returns: {objects:#?}" + ); Ok(Some(objects)) } -#[cfg(not(feature = "enterprise"))] -pub async fn list_objects_for_user( - _org_id: &str, - _user_id: &str, - _permission: &str, - _object_type: &str, -) -> anyhow::Result>> { - Ok(None) -} - /// OpenFGA object and parent type of `obj`; `None` when either type is unmapped. #[cfg(feature = "enterprise")] fn ofga_object(obj_type: &str, obj: &Authz) -> Option<(String, &'static str)> { diff --git a/src/infra/src/table/alerts/mod.rs b/src/infra/src/table/alerts/mod.rs index a31e3e0c6d..1a5e2f36a5 100644 --- a/src/infra/src/table/alerts/mod.rs +++ b/src/infra/src/table/alerts/mod.rs @@ -220,6 +220,7 @@ impl TryFrom for MetaAlert { alert.notify_on_recovery = value.notify_on_recovery.unwrap_or(false); alert.recovery_destinations = recovery_destinations; alert.keep_firing_for = value.keep_firing_for_seconds.unwrap_or(0); + alert.suppress_repeats = value.suppress_repeats.unwrap_or(false); Ok(alert) } @@ -958,6 +959,7 @@ fn update_mutable_fields( Some(serde_json::to_string(&alert.recovery_destinations)?) }; let keep_firing_for_seconds = alert.keep_firing_for; + let suppress_repeats = alert.suppress_repeats; alert_am.name = Set(name); alert_am.is_real_time = Set(is_real_time); @@ -1016,6 +1018,7 @@ fn update_mutable_fields( alert_am.notify_on_recovery = Set(Some(notify_on_recovery)); alert_am.recovery_destinations = Set(recovery_destinations); alert_am.keep_firing_for_seconds = Set(Some(keep_firing_for_seconds)); + alert_am.suppress_repeats = Set(Some(suppress_repeats)); Ok(()) } @@ -1096,6 +1099,7 @@ pub(super) mod tests { notify_on_recovery: None, recovery_destinations: None, keep_firing_for_seconds: None, + suppress_repeats: None, } } diff --git a/src/infra/src/table/entity/alerts.rs b/src/infra/src/table/entity/alerts.rs index 91a3545ac3..0b172fb41a 100644 --- a/src/infra/src/table/entity/alerts.rs +++ b/src/infra/src/table/entity/alerts.rs @@ -92,6 +92,8 @@ pub struct Model { /// How long the condition must stay clear before the episode recovers. /// NULL or 0 recovers on the first clear evaluation. pub keep_firing_for_seconds: Option, + /// NULL predates the feature, which is the same as `false`. + pub suppress_repeats: Option, } #[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)] @@ -178,6 +180,7 @@ mod tests { notify_on_recovery: None, recovery_destinations: None, keep_firing_for_seconds: None, + suppress_repeats: None, }; assert_eq!(m.id, "alert-1"); assert_eq!(m.name, "High Error Rate"); diff --git a/src/infra/src/table/entity/mod.rs b/src/infra/src/table/entity/mod.rs index 036f79d367..62c794dc0e 100644 --- a/src/infra/src/table/entity/mod.rs +++ b/src/infra/src/table/entity/mod.rs @@ -80,6 +80,8 @@ pub mod oncall_user_contacts; pub mod online_eval_jobs; pub mod org_ai_toolsets; pub mod org_cleanup_tasks; +pub mod org_domain_ownership; +pub mod org_domain_requests; pub mod org_ingestion_tokens; pub mod org_storage_providers; pub mod org_telephony; diff --git a/src/infra/src/table/entity/org_domain_ownership.rs b/src/infra/src/table/entity/org_domain_ownership.rs new file mode 100644 index 0000000000..963e3f8b2f --- /dev/null +++ b/src/infra/src/table/entity/org_domain_ownership.rs @@ -0,0 +1,26 @@ +//! `SeaORM` Entity for the domain_org_ownership table. + +use sea_orm::entity::prelude::*; + +#[derive(Clone, Debug, PartialEq, DeriveEntityModel, serde::Serialize, serde::Deserialize)] +#[sea_orm(table_name = "org_domain_ownership")] +pub struct Model { + #[sea_orm(primary_key, auto_increment = false)] + pub id: String, + pub org_id: String, + pub domain: String, + pub verification_token: String, + pub verification_state: i32, + pub verification_failure_reason: Option, + pub verified_at: Option, + pub released_at: Option, + pub last_checked_at: Option, + pub created_at: i64, + pub updated_at: i64, + pub meta_mapped: bool, +} + +#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)] +pub enum Relation {} + +impl ActiveModelBehavior for ActiveModel {} diff --git a/src/infra/src/table/entity/org_domain_requests.rs b/src/infra/src/table/entity/org_domain_requests.rs new file mode 100644 index 0000000000..beef6b7a88 --- /dev/null +++ b/src/infra/src/table/entity/org_domain_requests.rs @@ -0,0 +1,23 @@ +//! `SeaORM` Entity for the domain_org_requests table. + +use sea_orm::entity::prelude::*; + +#[derive(Clone, Debug, PartialEq, DeriveEntityModel, serde::Serialize, serde::Deserialize)] +#[sea_orm(table_name = "org_domain_requests")] +pub struct Model { + #[sea_orm(primary_key, auto_increment = false)] + pub id: String, + pub org_id: String, + pub domain: String, + pub created_at: i64, + pub updated_at: i64, + pub requested_by: String, + pub answered_by: Option, + pub status: i32, + pub reason: Option, +} + +#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)] +pub enum Relation {} + +impl ActiveModelBehavior for ActiveModel {} diff --git a/src/infra/src/table/entity/prelude.rs b/src/infra/src/table/entity/prelude.rs index 20d0c3e4da..b093be72ea 100644 --- a/src/infra/src/table/entity/prelude.rs +++ b/src/infra/src/table/entity/prelude.rs @@ -21,7 +21,9 @@ pub use super::{ llm_prompt_label_events::Entity as LlmPromptLabelEvents, llm_prompt_labels::Entity as LlmPromptLabels, llm_prompt_versions::Entity as LlmPromptVersions, llm_prompt_webhook_deliveries::Entity as LlmPromptWebhookDeliveries, - llm_prompts::Entity as LlmPrompts, org_ingestion_tokens::Entity as OrgIngestionTokens, + llm_prompts::Entity as LlmPrompts, org_domain_ownership::Entity as OrgDomainOwnership, + org_domain_requests::Entity as OrgDomainRequests, + org_ingestion_tokens::Entity as OrgIngestionTokens, org_storage_providers::Entity as OrgStorageProviders, org_users::Entity as OrgUsers, organizations::Entity as Organizations, pipeline_last_errors::Entity as PipelineLastErrors, re_pattern_stream_map::Entity as RePatternStreamMap, re_patterns::Entity as RePatterns, diff --git a/src/infra/src/table/entity/query_history.rs b/src/infra/src/table/entity/query_history.rs index 1aa738ed85..9e24535a06 100644 --- a/src/infra/src/table/entity/query_history.rs +++ b/src/infra/src/table/entity/query_history.rs @@ -26,7 +26,20 @@ pub struct Model { pub query: String, pub context: Json, pub starred: bool, + /// First run of this row; never changed after insert. pub created_at: i64, + #[sea_orm(column_type = "Text")] + pub source: String, + /// Hex sha256 of the merge key; NULL on rows written before the key existed. + pub query_key: Option, + /// Comma-wrapped stream list (`,a,b,`), empty when there are none. + #[sea_orm(column_type = "Text")] + pub streams: String, + pub run_count: i64, + /// Last run of this row. + pub updated_at: i64, + #[sea_orm(column_type = "Text")] + pub status: String, } #[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)] diff --git a/src/infra/src/table/entity/status_page_custom_domains.rs b/src/infra/src/table/entity/status_page_custom_domains.rs index d2da9838fb..fc37bae3fe 100644 --- a/src/infra/src/table/entity/status_page_custom_domains.rs +++ b/src/infra/src/table/entity/status_page_custom_domains.rs @@ -2,7 +2,9 @@ use sea_orm::entity::prelude::*; -#[derive(Clone, Debug, PartialEq, DeriveEntityModel, serde::Serialize, serde::Deserialize)] +#[derive( + Clone, Default, Debug, PartialEq, DeriveEntityModel, serde::Serialize, serde::Deserialize, +)] #[sea_orm(table_name = "status_page_custom_domains")] pub struct Model { #[sea_orm(primary_key, auto_increment = false)] diff --git a/src/infra/src/table/entity/synthetics_checks.rs b/src/infra/src/table/entity/synthetics_checks.rs index 358990742d..9d898286b4 100644 --- a/src/infra/src/table/entity/synthetics_checks.rs +++ b/src/infra/src/table/entity/synthetics_checks.rs @@ -31,7 +31,7 @@ pub struct Model { /// When the scheduler last fanned out this check (microseconds). pub last_triggered_at: i64, /// Denormalised status from the most recent completed check. - /// 0=Unknown, 1=Up, 2=Warning, 3=Down + /// 0=Unknown, 1=Up, 2=Warning, 3=Down, 4=Error (no verdict: dispatch/lease failure) pub last_check_status: i32, /// Runs that have failed back to back; reset to 0 by a pass. Compared /// against `settings.alert_if_fails` to decide whether to notify. diff --git a/src/infra/src/table/migration/m20261025_000005_add_held_action_to_slack_link_requests.rs b/src/infra/src/table/migration/m20261025_000005_add_held_action_to_slack_link_requests.rs index 08242eab64..f5a9a83bf2 100644 --- a/src/infra/src/table/migration/m20261025_000005_add_held_action_to_slack_link_requests.rs +++ b/src/infra/src/table/migration/m20261025_000005_add_held_action_to_slack_link_requests.rs @@ -46,12 +46,6 @@ impl MigrationTrait for Migration { } } -/// Adds the column to a freshly created table (test setups that skip the migrator). -pub(crate) async fn add_column(db: &C) -> Result<(), DbErr> { - let stmt = db.get_database_backend().build(&add_column_statement()); - db.execute(stmt).await.map(|_| ()) -} - fn add_column_statement() -> TableAlterStatement { Table::alter() .table(SlackLinkRequests::Table) diff --git a/src/infra/src/table/migration/m20261027_000001_alter_query_history.rs b/src/infra/src/table/migration/m20261027_000001_alter_query_history.rs new file mode 100644 index 0000000000..07f071eb54 --- /dev/null +++ b/src/infra/src/table/migration/m20261027_000001_alter_query_history.rs @@ -0,0 +1,644 @@ +// Copyright 2026 OpenObserve Inc. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +//! Adds source, merge key, streams, run count, last run and status to `query_history`. + +use sea_orm_migration::prelude::*; + +const TABLE: &str = "query_history"; +const OLD_ORG_USER_CREATED_AT_IDX: &str = "query_history_org_user_created_at_idx"; +const OLD_STARRED_CREATED_AT_IDX: &str = "query_history_starred_created_at_idx"; +const ORG_USER_UPDATED_AT_IDX: &str = "query_history_org_user_updated_at_idx"; +const ORG_USER_SOURCE_UPDATED_AT_IDX: &str = "query_history_org_user_source_updated_at_idx"; +const STARRED_UPDATED_AT_IDX: &str = "query_history_starred_updated_at_idx"; +const ORG_USER_SOURCE_KEY_UIDX: &str = "query_history_org_user_source_key_uidx"; + +#[derive(DeriveMigrationName)] +pub struct Migration; + +#[async_trait::async_trait] +impl MigrationTrait for Migration { + async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> { + // NOT NULL DEFAULT keeps v1.1.x nodes inserting during a rolling upgrade. + for mut def in [ + ColumnDef::new(QueryHistory::Source) + .text() + .not_null() + .default("metrics") + .to_owned(), + ColumnDef::new(QueryHistory::QueryKey) + .char_len(64) + .null() + .to_owned(), + ColumnDef::new(QueryHistory::Streams) + .text() + .not_null() + .default("") + .to_owned(), + ColumnDef::new(QueryHistory::RunCount) + .big_integer() + .not_null() + .default(1) + .to_owned(), + ColumnDef::new(QueryHistory::UpdatedAt) + .big_integer() + .not_null() + .default(0) + .to_owned(), + ColumnDef::new(QueryHistory::Status) + .text() + .not_null() + .default("ok") + .to_owned(), + ] { + add_column(manager, &mut def).await?; + } + // v1.1.x stored each row's last run in created_at; the guard makes a rerun a no-op. + manager + .get_connection() + .execute_unprepared( + "UPDATE query_history SET updated_at = created_at WHERE updated_at < created_at", + ) + .await?; + for name in [OLD_ORG_USER_CREATED_AT_IDX, OLD_STARRED_CREATED_AT_IDX] { + drop_index(manager, name).await?; + } + create_index( + manager, + ORG_USER_UPDATED_AT_IDX, + &[ + QueryHistory::OrgId, + QueryHistory::UserEmail, + QueryHistory::UpdatedAt, + ], + false, + ) + .await?; + create_index( + manager, + ORG_USER_SOURCE_UPDATED_AT_IDX, + &[ + QueryHistory::OrgId, + QueryHistory::UserEmail, + QueryHistory::Source, + QueryHistory::UpdatedAt, + ], + false, + ) + .await?; + create_index( + manager, + STARRED_UPDATED_AT_IDX, + &[QueryHistory::Starred, QueryHistory::UpdatedAt], + false, + ) + .await?; + // NULL keys on v1.1.x rows are distinct, so they never collide here. + create_index( + manager, + ORG_USER_SOURCE_KEY_UIDX, + &[ + QueryHistory::OrgId, + QueryHistory::UserEmail, + QueryHistory::Source, + QueryHistory::QueryKey, + ], + true, + ) + .await + } + + async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> { + for name in [ + ORG_USER_SOURCE_KEY_UIDX, + STARRED_UPDATED_AT_IDX, + ORG_USER_SOURCE_UPDATED_AT_IDX, + ORG_USER_UPDATED_AT_IDX, + ] { + drop_index(manager, name).await?; + } + // A repeated or partial rollback has already dropped updated_at. + if manager + .has_column(TABLE, &QueryHistory::UpdatedAt.to_string()) + .await? + { + manager + .get_connection() + .execute_unprepared( + "UPDATE query_history SET created_at = updated_at WHERE updated_at > created_at", + ) + .await?; + } + create_index( + manager, + OLD_ORG_USER_CREATED_AT_IDX, + &[ + QueryHistory::OrgId, + QueryHistory::UserEmail, + QueryHistory::CreatedAt, + ], + false, + ) + .await?; + create_index( + manager, + OLD_STARRED_CREATED_AT_IDX, + &[QueryHistory::Starred, QueryHistory::CreatedAt], + false, + ) + .await?; + for column in [ + QueryHistory::Status, + QueryHistory::UpdatedAt, + QueryHistory::RunCount, + QueryHistory::Streams, + QueryHistory::QueryKey, + QueryHistory::Source, + ] { + if manager.has_column(TABLE, &column.to_string()).await? { + manager + .alter_table( + Table::alter() + .table(QueryHistory::Table) + .drop_column(column) + .to_owned(), + ) + .await?; + } + } + Ok(()) + } +} + +#[derive(DeriveIden, Clone, Copy)] +enum QueryHistory { + Table, + OrgId, + UserEmail, + Starred, + CreatedAt, + Source, + QueryKey, + Streams, + RunCount, + UpdatedAt, + Status, +} + +async fn add_column(manager: &SchemaManager<'_>, def: &mut ColumnDef) -> Result<(), DbErr> { + if manager.has_column(TABLE, &def.get_column_name()).await? { + return Ok(()); + } + manager + .alter_table( + Table::alter() + .table(QueryHistory::Table) + .add_column(def.to_owned()) + .to_owned(), + ) + .await +} + +async fn drop_index(manager: &SchemaManager<'_>, name: &str) -> Result<(), DbErr> { + manager + .drop_index( + Index::drop() + .if_exists() + .name(name) + .table(QueryHistory::Table) + .to_owned(), + ) + .await +} + +async fn create_index( + manager: &SchemaManager<'_>, + name: &str, + columns: &[QueryHistory], + unique: bool, +) -> Result<(), DbErr> { + let mut index = Index::create(); + index.if_not_exists().name(name).table(QueryHistory::Table); + if unique { + index.unique(); + } + for column in columns { + index.col(*column); + } + manager.create_index(index.to_owned()).await +} + +#[cfg(test)] +mod tests { + use sea_orm::{ConnectionTrait, Database, DatabaseConnection, DbBackend, Statement}; + use sea_orm_migration::{MigrationName, MigrationTrait}; + + use super::*; + + type Row = ( + String, + String, + Option, + String, + i64, + i64, + i64, + String, + ); + + async fn v1_1_table_with_rows() -> DatabaseConnection { + let db = Database::connect("sqlite::memory:").await.unwrap(); + seed_v1_1_table(&db).await; + db + } + + async fn seed_v1_1_table(db: &DatabaseConnection) { + super::super::m20261004_000001_create_query_history::Migration + .up(&SchemaManager::new(db)) + .await + .unwrap(); + for (id, starred, created_at) in [("old", false, 100), ("star", true, 200)] { + db.execute_unprepared(&format!( + "INSERT INTO query_history (id, org_id, user_email, query, context, starred, \ + created_at) VALUES ('{id}', 'default', 'a@x.com', 'up', '{{\"step\":\"1m\"}}', \ + {starred}, {created_at})" + )) + .await + .unwrap(); + } + } + + /// A throwaway schema on the CI Postgres; `None` where none is configured. + async fn postgres_schema() -> Option<(DatabaseConnection, DatabaseConnection, String)> { + let dsn = std::env::var("ZO_META_POSTGRES_DSN").ok()?; + // The SQLite lane sets the variable empty rather than unsetting it. + if dsn.trim().is_empty() { + return None; + } + let schema = format!( + "qh_migration_test_{}", + config::ider::generate().to_ascii_lowercase() + ); + let admin = Database::connect(&dsn).await.unwrap(); + admin + .execute_unprepared(&format!(r#"CREATE SCHEMA "{schema}""#)) + .await + .unwrap(); + let mut options = sea_orm::ConnectOptions::new(dsn); + options + .max_connections(1) + .set_schema_search_path(schema.clone()); + Some((admin, Database::connect(options).await.unwrap(), schema)) + } + + async fn rows(db: &DatabaseConnection) -> Vec { + db.query_all(Statement::from_string( + DbBackend::Sqlite, + "SELECT id, source, query_key, streams, run_count, created_at, updated_at, status \ + FROM query_history ORDER BY id", + )) + .await + .unwrap() + .iter() + .map(|r| { + ( + r.try_get_by_index(0).unwrap(), + r.try_get_by_index(1).unwrap(), + r.try_get_by_index(2).unwrap(), + r.try_get_by_index(3).unwrap(), + r.try_get_by_index(4).unwrap(), + r.try_get_by_index(5).unwrap(), + r.try_get_by_index(6).unwrap(), + r.try_get_by_index(7).unwrap(), + ) + }) + .collect() + } + + async fn index_columns(db: &DatabaseConnection, index: &str) -> Vec> { + db.query_all(Statement::from_string( + DbBackend::Sqlite, + format!("SELECT name FROM pragma_index_info('{index}') ORDER BY seqno"), + )) + .await + .unwrap() + .iter() + .map(|r| r.try_get::>("", "name").unwrap()) + .collect() + } + + #[tokio::test] + async fn test_up_backfills_updated_at_from_created_at_on_a_populated_v1_1_table() { + let db = v1_1_table_with_rows().await; + let manager = SchemaManager::new(&db); + Migration.up(&manager).await.unwrap(); + + let to_row = |id: &str, created_at: i64, updated_at: i64| -> Row { + ( + id.to_string(), + "metrics".to_string(), + None, + String::new(), + 1, + created_at, + updated_at, + "ok".to_string(), + ) + }; + assert_eq!( + rows(&db).await, + vec![to_row("old", 100, 100), to_row("star", 200, 200)], + "every v1.1.x row gets updated_at = created_at" + ); + + db.execute_unprepared("UPDATE query_history SET updated_at = 900 WHERE id = 'old'") + .await + .unwrap(); + Migration.up(&manager).await.expect("idempotent"); + assert_eq!( + rows(&db).await, + vec![to_row("old", 100, 900), to_row("star", 200, 200)], + "a rerun leaves a row that ran after the upgrade alone" + ); + + let listed = crate::table::query_history::list_with( + &db, + "default", + "a@x.com", + &crate::table::query_history::ListFilter { + sources: vec!["metrics".to_string()], + ..Default::default() + }, + 10, + 0, + ) + .await + .unwrap(); + assert_eq!( + listed + .iter() + .map(|r| (r.id.as_str(), r.updated_at)) + .collect::>(), + vec![("old", 900), ("star", 200)] + ); + } + + #[tokio::test] + async fn test_up_swaps_indexes() { + let db = v1_1_table_with_rows().await; + let manager = SchemaManager::new(&db); + Migration.up(&manager).await.unwrap(); + + for old in [OLD_ORG_USER_CREATED_AT_IDX, OLD_STARRED_CREATED_AT_IDX] { + assert!(!manager.has_index(TABLE, old).await.unwrap(), "{old}"); + } + let names = |cols: &[Option<&str>]| -> Vec> { + cols.iter().map(|c| c.map(str::to_string)).collect() + }; + assert_eq!( + index_columns(&db, ORG_USER_UPDATED_AT_IDX).await, + names(&[Some("org_id"), Some("user_email"), Some("updated_at")]) + ); + assert_eq!( + index_columns(&db, ORG_USER_SOURCE_UPDATED_AT_IDX).await, + names(&[ + Some("org_id"), + Some("user_email"), + Some("source"), + Some("updated_at") + ]) + ); + assert_eq!( + index_columns(&db, STARRED_UPDATED_AT_IDX).await, + names(&[Some("starred"), Some("updated_at")]) + ); + assert_eq!( + index_columns(&db, ORG_USER_SOURCE_KEY_UIDX).await, + names(&[ + Some("org_id"), + Some("user_email"), + Some("source"), + Some("query_key") + ]) + ); + } + + #[tokio::test] + async fn test_up_leaves_exactly_the_agreed_index_set() { + let db = v1_1_table_with_rows().await; + Migration.up(&SchemaManager::new(&db)).await.unwrap(); + let indexes: Vec<(String, bool)> = db + .query_all(Statement::from_string( + DbBackend::Sqlite, + "SELECT name, \"unique\" FROM pragma_index_list('query_history') \ + WHERE origin = 'c' ORDER BY name", + )) + .await + .unwrap() + .iter() + .map(|r| { + ( + r.try_get_by_index(0).unwrap(), + r.try_get_by_index::(1).unwrap() == 1, + ) + }) + .collect(); + assert_eq!( + indexes, + vec![ + (ORG_USER_SOURCE_KEY_UIDX.to_string(), true), + (ORG_USER_SOURCE_UPDATED_AT_IDX.to_string(), false), + (ORG_USER_UPDATED_AT_IDX.to_string(), false), + (STARRED_UPDATED_AT_IDX.to_string(), false), + ] + ); + } + + #[tokio::test] + async fn test_unique_key_allows_null_keys_and_refuses_a_duplicate_key() { + let db = v1_1_table_with_rows().await; + Migration.up(&SchemaManager::new(&db)).await.unwrap(); + let insert = |id: &str, key: &str| { + format!( + "INSERT INTO query_history (id, org_id, user_email, query, context, created_at, \ + query_key) VALUES ('{id}', 'default', 'a@x.com', 'up', '{{}}', 1, {key})" + ) + }; + db.execute_unprepared(&insert("legacy", "NULL")) + .await + .expect("a v1.1.x node inserts with no key"); + db.execute_unprepared(&insert("k1", "'abc'")).await.unwrap(); + assert!(db.execute_unprepared(&insert("k2", "'abc'")).await.is_err()); + } + + #[tokio::test] + async fn test_down_restores_the_v1_1_table() { + let db = v1_1_table_with_rows().await; + let manager = SchemaManager::new(&db); + Migration.up(&manager).await.unwrap(); + Migration.down(&manager).await.unwrap(); + for column in [ + "source", + "query_key", + "streams", + "run_count", + "updated_at", + "status", + ] { + assert!( + !manager.has_column(TABLE, column).await.unwrap(), + "{column}" + ); + } + assert!( + manager + .has_index(TABLE, OLD_ORG_USER_CREATED_AT_IDX) + .await + .unwrap() + ); + assert!( + manager + .has_index(TABLE, OLD_STARRED_CREATED_AT_IDX) + .await + .unwrap() + ); + } + + #[tokio::test] + async fn test_down_keeps_the_last_run_of_rows_an_old_node_wrote() { + let db = v1_1_table_with_rows().await; + let manager = SchemaManager::new(&db); + Migration.up(&manager).await.unwrap(); + db.execute_unprepared( + "INSERT INTO query_history (id, org_id, user_email, query, context, created_at) \ + VALUES ('mid', 'default', 'a@x.com', 'up', '{}', 300)", + ) + .await + .unwrap(); + db.execute_unprepared("UPDATE query_history SET updated_at = 900 WHERE id = 'old'") + .await + .unwrap(); + Migration.down(&manager).await.unwrap(); + + let created: Vec<(String, i64)> = db + .query_all(Statement::from_string( + DbBackend::Sqlite, + "SELECT id, created_at FROM query_history ORDER BY id", + )) + .await + .unwrap() + .iter() + .map(|r| { + ( + r.try_get_by_index(0).unwrap(), + r.try_get_by_index(1).unwrap(), + ) + }) + .collect(); + assert_eq!( + created, + vec![ + ("mid".to_string(), 300), + ("old".to_string(), 900), + ("star".to_string(), 200) + ] + ); + } + + #[tokio::test] + async fn test_down_runs_twice() { + let db = v1_1_table_with_rows().await; + let manager = SchemaManager::new(&db); + Migration.up(&manager).await.unwrap(); + Migration.down(&manager).await.unwrap(); + Migration + .down(&manager) + .await + .expect("a second rollback is a no-op"); + } + + #[tokio::test] + async fn test_postgres_up_and_down_on_a_populated_v1_1_table_when_ci_provides_it() { + let Some((admin, db, schema)) = postgres_schema().await else { + return; + }; + seed_v1_1_table(&db).await; + let manager = SchemaManager::new(&db); + Migration.up(&manager).await.unwrap(); + Migration.up(&manager).await.expect("idempotent"); + + for index in [ + ORG_USER_UPDATED_AT_IDX, + ORG_USER_SOURCE_UPDATED_AT_IDX, + STARRED_UPDATED_AT_IDX, + ORG_USER_SOURCE_KEY_UIDX, + ] { + assert!(manager.has_index(TABLE, index).await.unwrap(), "{index}"); + } + let listed = crate::table::query_history::list_with( + &db, + "default", + "a@x.com", + &crate::table::query_history::ListFilter { + sources: vec!["metrics".to_string()], + ..Default::default() + }, + 10, + 0, + ) + .await + .unwrap(); + // Postgres pads the char(27) id. + assert_eq!( + listed + .iter() + .map(|r| (r.id.trim(), r.updated_at)) + .collect::>(), + vec![("star", 200), ("old", 100)] + ); + let insert = |id: &str| { + format!( + "INSERT INTO query_history (id, org_id, user_email, query, context, created_at, \ + query_key) VALUES ('{id}', 'default', 'a@x.com', 'up', '{{}}', 1, 'abc')" + ) + }; + db.execute_unprepared(&insert("k1")).await.unwrap(); + assert!(db.execute_unprepared(&insert("k2")).await.is_err()); + + Migration.down(&manager).await.unwrap(); + Migration + .down(&manager) + .await + .expect("a second rollback is a no-op"); + assert!(!manager.has_column(TABLE, "updated_at").await.unwrap()); + assert!( + manager + .has_index(TABLE, OLD_ORG_USER_CREATED_AT_IDX) + .await + .unwrap() + ); + + db.close().await.unwrap(); + admin + .execute_unprepared(&format!(r#"DROP SCHEMA "{schema}" CASCADE"#)) + .await + .unwrap(); + } + + #[test] + fn test_migration_name() { + assert_eq!(Migration.name(), "m20261027_000001_alter_query_history"); + } +} diff --git a/src/infra/src/table/migration/m20261030_000001_add_suppress_repeats_to_alerts.rs b/src/infra/src/table/migration/m20261030_000001_add_suppress_repeats_to_alerts.rs new file mode 100644 index 0000000000..a24f3a941f --- /dev/null +++ b/src/infra/src/table/migration/m20261030_000001_add_suppress_repeats_to_alerts.rs @@ -0,0 +1,92 @@ +// Copyright 2026 OpenObserve Inc. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +use sea_orm_migration::prelude::*; + +const TABLE: &str = "alerts"; +const COLUMN: &str = "suppress_repeats"; + +#[derive(DeriveMigrationName)] +pub struct Migration; + +#[async_trait::async_trait] +impl MigrationTrait for Migration { + async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> { + // SQLite emits a plain ADD COLUMN for add_column_if_not_exists, so guard explicitly. + if manager.has_column(TABLE, COLUMN).await? { + return Ok(()); + } + manager.alter_table(add_column_statement()).await + } + + async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> { + if !manager.has_column(TABLE, COLUMN).await? { + return Ok(()); + } + manager + .alter_table( + Table::alter() + .table(Alerts::Table) + .drop_column(Alerts::SuppressRepeats) + .to_owned(), + ) + .await + } +} + +#[derive(DeriveIden)] +enum Alerts { + Table, + SuppressRepeats, +} + +fn add_column_statement() -> TableAlterStatement { + Table::alter() + .table(Alerts::Table) + .add_column(ColumnDef::new(Alerts::SuppressRepeats).boolean().null()) + .to_owned() +} + +#[cfg(test)] +mod tests { + use sea_orm::{ConnectionTrait, Database}; + + use super::*; + + #[test] + fn postgres() { + assert_eq!( + &add_column_statement().to_string(PostgresQueryBuilder), + r#"ALTER TABLE "alerts" ADD COLUMN "suppress_repeats" bool NULL"# + ); + } + + #[tokio::test] + async fn test_up_reruns_without_adding_the_column_twice() { + let db = Database::connect("sqlite::memory:").await.unwrap(); + db.execute_unprepared("CREATE TABLE alerts (id varchar(27) PRIMARY KEY)") + .await + .unwrap(); + let manager = SchemaManager::new(&db); + + Migration.up(&manager).await.expect("first run"); + Migration.up(&manager).await.expect("second run"); + assert!(manager.has_column(TABLE, COLUMN).await.unwrap()); + + Migration.down(&manager).await.unwrap(); + Migration.down(&manager).await.unwrap(); + assert!(!manager.has_column(TABLE, COLUMN).await.unwrap()); + } +} diff --git a/src/infra/src/table/migration/m20261030_000001_create_org_domain_ownership_table.rs b/src/infra/src/table/migration/m20261030_000001_create_org_domain_ownership_table.rs new file mode 100644 index 0000000000..193b6c575e --- /dev/null +++ b/src/infra/src/table/migration/m20261030_000001_create_org_domain_ownership_table.rs @@ -0,0 +1,135 @@ +// Copyright 2026 OpenObserve Inc. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +use sea_orm_migration::prelude::*; + +#[derive(DeriveMigrationName)] +pub struct Migration; + +#[async_trait::async_trait] +impl MigrationTrait for Migration { + async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> { + manager.create_table(create_domains_stmt()).await?; + manager.create_index(domain_org_idx()).await?; + Ok(()) + } + + async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> { + manager + .drop_table(Table::drop().table(OrgDomainOwnership::Table).to_owned()) + .await?; + Ok(()) + } +} + +fn create_domains_stmt() -> TableCreateStatement { + Table::create() + .table(OrgDomainOwnership::Table) + .if_not_exists() + .col( + ColumnDef::new(OrgDomainOwnership::Id) + .string_len(27) + .not_null() + .primary_key(), + ) + .col( + ColumnDef::new(OrgDomainOwnership::OrgId) + .string_len(100) + .not_null(), + ) + .col( + ColumnDef::new(OrgDomainOwnership::Domain) + .string_len(253) + .not_null(), + ) + .col( + ColumnDef::new(OrgDomainOwnership::VerificationToken) + .string_len(128) + .not_null(), + ) + // 0 pending, 1 verified, 2 failed. + .col( + ColumnDef::new(OrgDomainOwnership::VerificationState) + .integer() + .not_null() + .default(0), + ) + // 0 record-missing, 1 value-mismatch, 2 dns-resolution-failed. + .col( + ColumnDef::new(OrgDomainOwnership::VerificationFailureReason) + .integer() + .null(), + ) + .col( + ColumnDef::new(OrgDomainOwnership::VerifiedAt) + .big_integer() + .null(), + ) + .col( + ColumnDef::new(OrgDomainOwnership::ReleasedAt) + .big_integer() + .null(), + ) + .col( + ColumnDef::new(OrgDomainOwnership::LastCheckedAt) + .big_integer() + .null(), + ) + .col( + ColumnDef::new(OrgDomainOwnership::CreatedAt) + .big_integer() + .not_null(), + ) + .col( + ColumnDef::new(OrgDomainOwnership::UpdatedAt) + .big_integer() + .not_null(), + ) + .col( + ColumnDef::new(OrgDomainOwnership::MetaMapped) + .boolean() + .not_null() + .default(false), + ) + .to_owned() +} + +fn domain_org_idx() -> IndexCreateStatement { + sea_query::Index::create() + .if_not_exists() + .name("org_domain_ownership_org_domain_idx") + .table(OrgDomainOwnership::Table) + .col(OrgDomainOwnership::OrgId) + .col(OrgDomainOwnership::Domain) + .col(OrgDomainOwnership::VerificationState) + .to_owned() +} + +#[derive(DeriveIden)] +enum OrgDomainOwnership { + Table, + Id, + OrgId, + Domain, + VerificationToken, + VerificationState, + VerificationFailureReason, + VerifiedAt, + ReleasedAt, + LastCheckedAt, + CreatedAt, + UpdatedAt, + MetaMapped, +} diff --git a/src/infra/src/table/migration/m20261030_000002_create_org_domain_ownership_req_table.rs b/src/infra/src/table/migration/m20261030_000002_create_org_domain_ownership_req_table.rs new file mode 100644 index 0000000000..96a950bc32 --- /dev/null +++ b/src/infra/src/table/migration/m20261030_000002_create_org_domain_ownership_req_table.rs @@ -0,0 +1,98 @@ +// Copyright 2026 OpenObserve Inc. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +use sea_orm_migration::prelude::*; + +#[derive(DeriveMigrationName)] +pub struct Migration; + +#[async_trait::async_trait] +impl MigrationTrait for Migration { + async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> { + manager.create_table(create_domaim_req_stmt()).await?; + Ok(()) + } + + async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> { + manager + .drop_table(Table::drop().table(OrgDomainRequests::Table).to_owned()) + .await?; + Ok(()) + } +} + +fn create_domaim_req_stmt() -> TableCreateStatement { + Table::create() + .table(OrgDomainRequests::Table) + .if_not_exists() + .col( + ColumnDef::new(OrgDomainRequests::Id) + .string_len(27) + .not_null() + .primary_key(), + ) + .col( + ColumnDef::new(OrgDomainRequests::OrgId) + .string_len(100) + .not_null(), + ) + .col( + ColumnDef::new(OrgDomainRequests::Domain) + .string_len(253) + .not_null(), + ) + .col( + ColumnDef::new(OrgDomainRequests::CreatedAt) + .big_integer() + .not_null(), + ) + .col( + ColumnDef::new(OrgDomainRequests::UpdatedAt) + .big_integer() + .not_null(), + ) + .col( + ColumnDef::new(OrgDomainRequests::RequestedBy) + .string_len(100) + .not_null(), + ) + .col( + ColumnDef::new(OrgDomainRequests::AnsweredBy) + .string_len(100) + .null(), + ) + .col(ColumnDef::new(OrgDomainRequests::Reason).string().null()) + .col( + ColumnDef::new(OrgDomainRequests::Status) + .integer() + .not_null() + .default(0), + ) + .to_owned() +} + +#[derive(DeriveIden)] +enum OrgDomainRequests { + Table, + Id, + OrgId, + Domain, + CreatedAt, + UpdatedAt, + RequestedBy, + AnsweredBy, + Reason, + Status, +} diff --git a/src/infra/src/table/migration/mod.rs b/src/infra/src/table/migration/mod.rs index de27c92e44..66549dfe4d 100644 --- a/src/infra/src/table/migration/mod.rs +++ b/src/infra/src/table/migration/mod.rs @@ -241,8 +241,12 @@ mod m20261025_000003_add_mute_targets_to_downtimes; mod m20261025_000004_create_slack_alert_messages; mod m20261025_000005_add_held_action_to_slack_link_requests; mod m20261026_000001_backfill_v1_rca_pointer; +mod m20261027_000001_alter_query_history; mod m20261028_000001_add_group_label_set_to_alert_states; mod m20261029_000001_widen_alert_dedup_fingerprint; +mod m20261030_000001_add_suppress_repeats_to_alerts; +mod m20261030_000001_create_org_domain_ownership_table; +mod m20261030_000002_create_org_domain_ownership_req_table; /// Shared body of the two `folder_id` migrations above; not a migration itself. mod workflow_folder_id; @@ -435,10 +439,13 @@ pub(crate) async fn create_rum_pa_tables(db: &sea_orm::DatabaseConnection) -> Re /// Apply only the Slack app tables, idempotently, for tests in any crate. #[cfg(any(test, feature = "test-utils"))] -pub(crate) async fn create_slack_tables(db: &C) -> Result<(), DbErr> { +pub(crate) async fn create_slack_tables(db: &sea_orm::DatabaseConnection) -> Result<(), DbErr> { + use sea_orm_migration::MigrationTrait; m20261018_000001_create_slack_tables::create_all(db).await?; m20261025_000004_create_slack_alert_messages::create_all(db).await?; - m20261025_000005_add_held_action_to_slack_link_requests::add_column(db).await + m20261025_000005_add_held_action_to_slack_link_requests::Migration + .up(&SchemaManager::new(db)) + .await } /// Apply only `alert_acks`, idempotently, for tests in any crate. @@ -501,6 +508,20 @@ pub(crate) async fn backfill_v1_rca_pointer(db: &sea_orm::DatabaseConnection) -> .await } +/// Apply the query_history table and its columns, idempotently, for tests in any crate. +#[cfg(any(test, feature = "test-utils"))] +pub(crate) async fn create_query_history_table( + db: &sea_orm::DatabaseConnection, +) -> Result<(), DbErr> { + let manager = SchemaManager::new(db); + m20261004_000001_create_query_history::Migration + .up(&manager) + .await?; + m20261027_000001_alter_query_history::Migration + .up(&manager) + .await +} + pub struct Migrator; #[async_trait::async_trait] @@ -732,6 +753,10 @@ impl MigratorTrait for Migrator { Box::new(m20261025_000005_add_held_action_to_slack_link_requests::Migration), Box::new(m20261028_000001_add_group_label_set_to_alert_states::Migration), Box::new(m20261029_000001_widen_alert_dedup_fingerprint::Migration), + Box::new(m20261027_000001_alter_query_history::Migration), + Box::new(m20261030_000001_add_suppress_repeats_to_alerts::Migration), + Box::new(m20261030_000001_create_org_domain_ownership_table::Migration), + Box::new(m20261030_000002_create_org_domain_ownership_req_table::Migration), ] } } @@ -825,6 +850,12 @@ mod tests { ), (127, "m20261028_000001_add_group_label_set_to_alert_states"), (128, "m20261029_000001_widen_alert_dedup_fingerprint"), + (129, "m20261027_000001_alter_query_history"), + (130, "m20261030_000001_add_suppress_repeats_to_alerts"), + ( + 131, + "m20261030_000002_create_org_domain_ownership_req_table", + ), ]; // Two branches that each claim the next version merge cleanly, so a reused number must fail diff --git a/src/infra/src/table/mod.rs b/src/infra/src/table/mod.rs index 3bfbc17c3f..feb95ad2d2 100644 --- a/src/infra/src/table/mod.rs +++ b/src/infra/src/table/mod.rs @@ -82,6 +82,8 @@ pub mod oncall_user_contacts; pub mod online_eval_jobs; pub mod org_ai_toolsets; pub mod org_cleanup_tasks; +pub mod org_domain_ownership; +pub mod org_domain_requests; pub mod org_ingestion_tokens; pub mod org_storage_providers; pub mod org_telephony; diff --git a/src/infra/src/table/org_domain_ownership.rs b/src/infra/src/table/org_domain_ownership.rs new file mode 100644 index 0000000000..42ee704cef --- /dev/null +++ b/src/infra/src/table/org_domain_ownership.rs @@ -0,0 +1,150 @@ +// Copyright 2026 OpenObserve Inc. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +use std::collections::HashMap; + +use config::utils::{rand::generate_random_string, time::now_micros}; +use sea_orm::{ + ActiveModelTrait, ActiveValue::Set, ColumnTrait, EntityTrait, IntoActiveModel, QueryFilter, + QuerySelect, sea_query::Expr, +}; + +use super::entity::{org_domain_ownership::*, prelude::OrgDomainOwnership}; +use crate::db::{get_orm_client_ro, get_orm_client_rw}; + +pub enum OwnershipState { + Pending = 0, + Verified = 1, + Failed = 2, +} +pub struct OwnershipRecord { + pub org_id: String, + pub domain: String, + pub state: OwnershipState, +} + +pub async fn list_active_domain_org_map() -> Result, anyhow::Error> { + let client = get_orm_client_ro().await; + let records = OrgDomainOwnership::find() + .select_only() + .column(Column::Domain) + .column(Column::OrgId) + .filter(Column::VerificationState.eq(OwnershipState::Verified as i32)) + .into_tuple::<(String, String)>() + .all(client) + .await?; + Ok(records.into_iter().collect()) +} + +pub async fn get_verified_org_for_domain(domain: &str) -> Result, anyhow::Error> { + let client = get_orm_client_ro().await; + let record = OrgDomainOwnership::find() + .select_only() + .column(Column::OrgId) + .filter(Column::Domain.eq(domain)) + .filter(Column::VerificationState.eq(OwnershipState::Verified as i32)) + .into_tuple::<(String,)>() + .one(client) + .await?; + Ok(record.map(|v| v.0)) +} + +pub async fn get_domains_for_org(org_id: &str) -> Result, anyhow::Error> { + let client = get_orm_client_ro().await; + let records = OrgDomainOwnership::find() + .filter(Column::OrgId.eq(org_id)) + .all(client) + .await?; + Ok(records) +} + +pub async fn save_org_domain_mapping( + record: OwnershipRecord, + meta_mapped: bool, +) -> Result<(), anyhow::Error> { + let client = get_orm_client_rw().await; + let now = now_micros(); + let token = format!("o2v_{}_{}", record.org_id, generate_random_string(32)); + + let m = ActiveModel { + id: Set(config::ider::generate()), + org_id: Set(record.org_id), + domain: Set(record.domain), + verification_token: Set(token), + verification_state: Set(record.state as i32), + verification_failure_reason: Set(None), + verified_at: Set(None), + released_at: Set(None), + last_checked_at: Set(None), + created_at: Set(now), + updated_at: Set(now), + meta_mapped: Set(meta_mapped), + }; + OrgDomainOwnership::insert(m).exec(client).await?; + Ok(()) +} + +pub async fn delete_linked_domain(org_id: &str, domain: &str) -> Result<(), anyhow::Error> { + let client = get_orm_client_rw().await; + OrgDomainOwnership::delete_many() + .filter(Column::OrgId.eq(org_id)) + .filter(Column::Domain.eq(domain)) + .exec(client) + .await?; + Ok(()) +} + +pub async fn get_domain_org_record( + org_id: &str, + domain: &str, +) -> Result, anyhow::Error> { + let client = get_orm_client_ro().await; + let res = OrgDomainOwnership::find() + .filter(Column::OrgId.eq(org_id)) + .filter(Column::Domain.eq(domain)) + .one(client) + .await?; + Ok(res) +} + +pub async fn save_domain_org_record(model: Model) -> Result<(), anyhow::Error> { + let client = get_orm_client_rw().await; + let am = model.clone().into_active_model().reset_all(); + am.update(client).await?; + Ok(()) +} + +pub async fn list_all() -> Result, anyhow::Error> { + let client = get_orm_client_ro().await; + let res = OrgDomainOwnership::find().all(client).await?; + Ok(res) +} + +pub async fn mark_meta_mapped(org_id: &str, domain: &str) -> Result<(), anyhow::Error> { + let now = now_micros(); + let client = get_orm_client_rw().await; + OrgDomainOwnership::update_many() + .filter(Column::OrgId.eq(org_id)) + .filter(Column::Domain.eq(domain)) + .col_expr(Column::MetaMapped, Expr::value(true)) + .col_expr(Column::LastCheckedAt, Expr::value(now)) + .col_expr( + Column::VerificationState, + Expr::value(OwnershipState::Verified as i32), + ) + .exec(client) + .await?; + Ok(()) +} diff --git a/src/infra/src/table/org_domain_requests.rs b/src/infra/src/table/org_domain_requests.rs new file mode 100644 index 0000000000..b482055ec5 --- /dev/null +++ b/src/infra/src/table/org_domain_requests.rs @@ -0,0 +1,107 @@ +// Copyright 2026 OpenObserve Inc. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +use config::utils::time::now_micros; +use sea_orm::{ColumnTrait, EntityTrait, IntoActiveModel, QueryFilter, sea_query::Expr}; + +use super::entity::{org_domain_requests::*, prelude::OrgDomainRequests}; +use crate::db::{get_orm_client_ro, get_orm_client_rw}; + +#[derive(Debug)] +pub enum Status { + Pending = 0, + Approved = 1, + Rejected = 2, +} +impl From for Status { + fn from(value: i32) -> Self { + match value { + 0 => Self::Pending, + 1 => Self::Approved, + 2 => Self::Rejected, + _ => Self::Pending, + } + } +} + +pub async fn save_org_domain_request(req: Model) -> Result<(), anyhow::Error> { + let client = get_orm_client_rw().await; + let m = req.into_active_model(); + OrgDomainRequests::insert(m).exec(client).await?; + Ok(()) +} + +pub async fn list_all() -> Result, anyhow::Error> { + let client = get_orm_client_ro().await; + let res = OrgDomainRequests::find().all(client).await?; + Ok(res) +} + +pub async fn list_by_org(org_id: &str) -> Result, anyhow::Error> { + let client = get_orm_client_ro().await; + let res = OrgDomainRequests::find() + .filter(Column::OrgId.eq(org_id)) + .all(client) + .await?; + Ok(res) +} + +pub async fn update_status( + id: &str, + user: &str, + new_status: Status, + reason: String, +) -> Result<(), anyhow::Error> { + let now = now_micros(); + let client = get_orm_client_rw().await; + OrgDomainRequests::update_many() + .filter(Column::Id.eq(id)) + .col_expr(Column::Status, Expr::value(new_status as i32)) + .col_expr(Column::AnsweredBy, Expr::value(user)) + .col_expr(Column::Reason, Expr::value(reason)) + .col_expr(Column::UpdatedAt, Expr::value(now)) + .exec(client) + .await?; + Ok(()) +} + +pub async fn get_by_id(id: &str) -> Result, anyhow::Error> { + let client = get_orm_client_ro().await; + let res = OrgDomainRequests::find() + .filter(Column::Id.eq(id)) + .one(client) + .await?; + Ok(res) +} + +pub async fn get_by_org_domain(org_id: &str, domain: &str) -> Result, anyhow::Error> { + let client = get_orm_client_ro().await; + let res = OrgDomainRequests::find() + .filter(Column::OrgId.eq(org_id)) + .filter(Column::Domain.eq(domain)) + .filter(Column::Status.ne(Status::Approved as i32)) + .one(client) + .await?; + Ok(res) +} + +pub async fn delete_by_org(org_id: &str) -> Result<(), anyhow::Error> { + let client = get_orm_client_rw().await; + OrgDomainRequests::delete_many() + .filter(Column::OrgId.eq(org_id)) + .exec(client) + .await?; + Ok(()) +} diff --git a/src/infra/src/table/query_history.rs b/src/infra/src/table/query_history.rs index 0e99051af3..a83d21635e 100644 --- a/src/infra/src/table/query_history.rs +++ b/src/infra/src/table/query_history.rs @@ -14,199 +14,173 @@ // along with this program. If not, see . use sea_orm::{ - ColumnTrait, Condition, ConnectionTrait, EntityTrait, PaginatorTrait, QueryFilter, QueryOrder, - QuerySelect, Set, TransactionTrait, - sea_query::{Expr, Func, LikeExpr, SimpleExpr}, + ColumnTrait, Condition, ConnectionTrait, DbBackend, EntityTrait, FromQueryResult, + PaginatorTrait, QueryFilter, QueryOrder, QuerySelect, QueryTrait, Set, Statement, + TransactionTrait, + sea_query::{Expr, Func, LikeExpr, OnConflict, Query, SimpleExpr}, }; use super::entity::query_history::{ActiveModel, Column, Entity, Model}; use crate::{ db::{get_orm_client_ro, get_orm_client_rw}, - errors, + errors, orm_err, }; -/// Recording prunes the user's oldest unstarred entries beyond this many. -pub const MAX_UNSTARRED_PER_USER: u64 = 1000; -/// Starring past this many is refused; recording never stars, so it is unaffected. +/// Recording a new row prunes the user's oldest unstarred rows of that source beyond this many. +pub const MAX_UNSTARRED_PER_SOURCE: u64 = 500; +/// Starring past this many across all sources is refused; recording never stars. pub const MAX_STARRED_PER_USER: u64 = 200; +/// Pages that record history; the handler refuses any other source. +pub const SOURCES: [&str; 5] = ["logs", "traces", "metrics", "dbm", "profiles"]; +/// What v1.1.x rows and clients that send no source are. +pub const DEFAULT_SOURCE: &str = "metrics"; +pub const STATUS_OK: &str = "ok"; +pub const STATUS_ERROR: &str = "error"; /// Outcome of starring or unstarring an entry. pub enum SetStarred { - Updated(Model), + Updated(Box), NotFound, /// Starring would take the user past `MAX_STARRED_PER_USER`. StarredCapReached, } -/// Records a query for this org and user; the same query as their latest entry refreshes it. -pub async fn record( - org_id: &str, - user_email: &str, - query: &str, - context: serde_json::Value, - now_us: i64, -) -> Result { - let client = get_orm_client_rw().await; - record_with(client, org_id, user_email, query, context, now_us).await +/// One run to record; repeats of the same source and key merge into one row. +pub struct NewRun<'a> { + pub org_id: &'a str, + pub user_email: &'a str, + pub source: &'a str, + pub query: &'a str, + /// The client's merge key; `None` or empty falls back to the query text. + pub query_key: Option<&'a str>, + pub streams: &'a [String], + pub status: &'a str, + pub context: serde_json::Value, + pub now_us: i64, } -pub async fn record_with( +/// List filters; every `None` or empty field matches all rows. +#[derive(Default)] +pub struct ListFilter { + pub sources: Vec, + pub status: Option, + /// Exact stream name. + pub stream: Option, + /// Case-insensitive literal substring of the query or the streams. + pub q: Option, + pub starred: Option, + /// Inclusive bounds on `updated_at`, in microseconds. + pub ran_from: Option, + pub ran_to: Option, +} + +/// Records a run: inserts a row, or merges into the row with the same org, user, source and key. +pub async fn record(run: NewRun<'_>) -> Result { + let client = get_orm_client_rw().await; + record_with(client, run).await +} + +pub async fn record_with( conn: &C, - org_id: &str, - user_email: &str, - query: &str, - context: serde_json::Value, - now_us: i64, + run: NewRun<'_>, ) -> Result { - let txn = conn.begin().await?; - let latest = Entity::find() - .filter(Column::OrgId.eq(org_id)) - .filter(Column::UserEmail.eq(user_email)) - .select_only() - .column(Column::Id) - .column(Column::Query) - .column(Column::Starred) - .order_by_desc(Column::CreatedAt) - .order_by_desc(Column::Id) - .into_tuple::<(String, String, bool)>() - .one(&txn) - .await?; - let model = match latest { - Some((id, latest_query, starred)) if latest_query == query => { - Entity::update_many() - .col_expr(Column::CreatedAt, Expr::value(now_us)) - .col_expr(Column::Context, Expr::value(context.clone())) - .filter(Column::Id.eq(&id)) - .exec(&txn) - .await?; - Model { - id, - org_id: org_id.to_string(), - user_email: user_email.to_string(), - query: latest_query, - context, - starred, - created_at: now_us, - } - } - _ => { - let model = Model { - id: config::ider::uuid(), - org_id: org_id.to_string(), - user_email: user_email.to_string(), - query: query.to_string(), - context, - starred: false, - created_at: now_us, - }; - Entity::insert(ActiveModel { - id: Set(model.id.clone()), - org_id: Set(model.org_id.clone()), - user_email: Set(model.user_email.clone()), - query: Set(model.query.clone()), - context: Set(model.context.clone()), - starred: Set(false), - created_at: Set(now_us), - }) - .exec(&txn) - .await?; - prune_unstarred_over_cap(&txn, org_id, user_email).await?; - model - } + let entry = ActiveModel { + id: Set(config::ider::uuid()), + org_id: Set(run.org_id.to_string()), + user_email: Set(run.user_email.to_string()), + query: Set(run.query.to_string()), + context: Set(run.context), + starred: Set(false), + created_at: Set(run.now_us), + source: Set(run.source.to_string()), + query_key: Set(Some(query_key_hash(run.query_key, run.query))), + streams: Set(encode_streams(run.streams)), + run_count: Set(1), + updated_at: Set(run.now_us), + status: Set(run.status.to_string()), }; - txn.commit().await?; + // One statement, so concurrent identical runs merge instead of racing to insert. + let mut stmt = Entity::insert(entry) + .on_conflict( + OnConflict::columns([ + Column::OrgId, + Column::UserEmail, + Column::Source, + Column::QueryKey, + ]) + .value( + Column::RunCount, + Expr::col((Entity, Column::RunCount)).add(1), + ) + .update_columns([Column::UpdatedAt, Column::Context, Column::Status]) + .to_owned(), + ) + .into_query(); + stmt.returning(Query::returning().all()); + + let Some(row) = conn + .query_one(conn.get_database_backend().build(&stmt)) + .await? + else { + return orm_err!("query history upsert returned no row"); + }; + let model = Model::from_query_result(&row, "")?; + // After the upsert has committed, so a prune also sees rows that concurrent runs inserted. + if model.run_count == 1 { + prune_unstarred_over_cap(conn, run.org_id, run.user_email, run.source, &model.id).await?; + } Ok(model) } -/// Lists one user's entries in one org, newest first. +/// Lists one user's entries in one org, most recently run first. pub async fn list( org_id: &str, user_email: &str, - starred: Option, - q: Option<&str>, + filter: &ListFilter, limit: u64, offset: u64, ) -> Result, errors::Error> { let client = get_orm_client_ro().await; - list_with(client, org_id, user_email, starred, q, limit, offset).await + list_with(client, org_id, user_email, filter, limit, offset).await } pub async fn list_with( conn: &C, org_id: &str, user_email: &str, - starred: Option, - q: Option<&str>, + filter: &ListFilter, limit: u64, offset: u64, ) -> Result, errors::Error> { - let mut select = Entity::find() - .filter(Column::OrgId.eq(org_id)) - .filter(Column::UserEmail.eq(user_email)); - if let Some(starred) = starred { - select = select.filter(Column::Starred.eq(starred)); - } - if let Some(q) = q.filter(|q| !q.is_empty()) { - select = select.filter(query_matches(q)); - } - Ok(select - .order_by_desc(Column::CreatedAt) - .order_by_desc(Column::Id) + Ok(list_query(org_id, user_email, filter) .limit(limit) .offset(offset) .all(conn) .await?) } -/// Deletes the user's unstarred rows from the `MAX_UNSTARRED_PER_USER + 1`-th newest onward. -async fn prune_unstarred_over_cap( +/// How many entries `list` would return with no limit. +pub async fn count( + org_id: &str, + user_email: &str, + filter: &ListFilter, +) -> Result { + let client = get_orm_client_ro().await; + count_with(client, org_id, user_email, filter).await +} + +pub async fn count_with( conn: &C, org_id: &str, user_email: &str, -) -> Result<(), errors::Error> { - let unstarred = Condition::all() - .add(Column::OrgId.eq(org_id)) - .add(Column::UserEmail.eq(user_email)) - .add(Column::Starred.eq(false)); - let Some((created_at, id)) = Entity::find() - .filter(unstarred.clone()) - .select_only() - .column(Column::CreatedAt) - .column(Column::Id) - .order_by_desc(Column::CreatedAt) - .order_by_desc(Column::Id) - .offset(MAX_UNSTARRED_PER_USER) - .into_tuple::<(i64, String)>() - .one(conn) - .await? - else { - return Ok(()); - }; - let at_or_before_boundary = Condition::any().add(Column::CreatedAt.lt(created_at)).add( - Condition::all() - .add(Column::CreatedAt.eq(created_at)) - .add(Column::Id.lte(id)), - ); - Entity::delete_many() - .filter(unstarred) - .filter(at_or_before_boundary) - .exec(conn) - .await?; - Ok(()) -} - -/// Case-insensitive literal substring match: `%`, `_` and `\` match only themselves. -fn query_matches(q: &str) -> SimpleExpr { - let mut pattern = String::with_capacity(q.len() + 2); - pattern.push('%'); - for c in q.to_lowercase().chars() { - if matches!(c, '\\' | '%' | '_') { - pattern.push('\\'); - } - pattern.push(c); - } - pattern.push('%'); - Expr::expr(Func::lower(Expr::col(Column::Query))).like(LikeExpr::new(pattern).escape('\\')) + filter: &ListFilter, +) -> Result { + Ok(Entity::find() + .filter(Column::OrgId.eq(org_id)) + .filter(Column::UserEmail.eq(user_email)) + .filter(list_condition(filter)) + .count(conn) + .await?) } /// Stars or unstars one of the user's entries; starring is refused at `MAX_STARRED_PER_USER`. @@ -220,29 +194,35 @@ pub async fn set_starred( set_starred_with(client, org_id, user_email, id, starred).await } -pub async fn set_starred_with( +pub async fn set_starred_with( conn: &C, org_id: &str, user_email: &str, id: &str, starred: bool, ) -> Result { - // MySQL rows_affected counts changed rows, so a re-star reports 0; "not found" needs a lookup. + let txn = conn.begin().await?; + let users_rows = Condition::all() + .add(Column::OrgId.eq(org_id)) + .add(Column::UserEmail.eq(user_email)); + // Only a star can pass the cap, so an unstar takes no lock. + if starred && let Some(lock) = star_lock(txn.get_database_backend(), org_id, user_email) { + txn.execute(lock).await?; + } + // MySQL rows_affected counts changed rows, so a re-star reports 0; the response needs the row. let Some(entry) = Entity::find() - .filter(Column::OrgId.eq(org_id)) - .filter(Column::UserEmail.eq(user_email)) + .filter(users_rows.clone()) .filter(Column::Id.eq(id)) - .one(conn) + .one(&txn) .await? else { return Ok(SetStarred::NotFound); }; if starred && !entry.starred { let starred_count = Entity::find() - .filter(Column::OrgId.eq(org_id)) - .filter(Column::UserEmail.eq(user_email)) + .filter(users_rows) .filter(Column::Starred.eq(true)) - .count(conn) + .count(&txn) .await?; if starred_count >= MAX_STARRED_PER_USER { return Ok(SetStarred::StarredCapReached); @@ -251,9 +231,10 @@ pub async fn set_starred_with( Entity::update_many() .col_expr(Column::Starred, Expr::value(starred)) .filter(Column::Id.eq(id)) - .exec(conn) + .exec(&txn) .await?; - Ok(SetStarred::Updated(Model { starred, ..entry })) + txn.commit().await?; + Ok(SetStarred::Updated(Box::new(Model { starred, ..entry }))) } /// Deletes one of the user's entries in the org; false when it is not theirs or does not exist. @@ -277,7 +258,37 @@ pub async fn delete_with( Ok(res.rows_affected > 0) } -/// Deletes unstarred entries older than `cutoff_us` across every org and user; starred are kept. +/// Deletes the user's entries in the org for `sources` (all when empty); starred only on request. +pub async fn clear( + org_id: &str, + user_email: &str, + sources: &[String], + include_starred: bool, +) -> Result { + let client = get_orm_client_rw().await; + clear_with(client, org_id, user_email, sources, include_starred).await +} + +pub async fn clear_with( + conn: &C, + org_id: &str, + user_email: &str, + sources: &[String], + include_starred: bool, +) -> Result { + let mut delete = Entity::delete_many() + .filter(Column::OrgId.eq(org_id)) + .filter(Column::UserEmail.eq(user_email)); + if !sources.is_empty() { + delete = delete.filter(Column::Source.is_in(sources.iter().cloned())); + } + if !include_starred { + delete = delete.filter(Column::Starred.eq(false)); + } + Ok(delete.exec(conn).await?.rows_affected) +} + +/// Deletes unstarred entries first and last run before `cutoff_us` across every org and user. pub async fn delete_unstarred_before(cutoff_us: i64) -> Result { let client = get_orm_client_rw().await; delete_unstarred_before_with(client, cutoff_us).await @@ -289,6 +300,8 @@ pub async fn delete_unstarred_before_with( ) -> Result { let res = Entity::delete_many() .filter(Column::Starred.eq(false)) + .filter(Column::UpdatedAt.lt(cutoff_us)) + // An older node in a rolling upgrade writes rows with updated_at = 0. .filter(Column::CreatedAt.lt(cutoff_us)) .exec(conn) .await?; @@ -345,151 +358,708 @@ pub async fn delete_by_email_with( Ok(res.rows_affected) } +/// Splits the stored `,a,b,` form back into stream names. +pub fn decode_streams(streams: &str) -> Vec { + streams + .split(',') + .filter(|s| !s.is_empty()) + .map(str::to_string) + .collect() +} + +/// Creates the table as the migrations leave it; for tests, since the migrator replays everything. +#[cfg(any(test, feature = "test-utils"))] +pub async fn create_table_for_tests( + db: &sea_orm::DatabaseConnection, +) -> Result<(), sea_orm::DbErr> { + super::migration::create_query_history_table(db).await +} + +/// Comma-wrapped so `LIKE '%,name,%'` matches whole stream names only. +fn encode_streams(streams: &[String]) -> String { + if streams.is_empty() { + return String::new(); + } + format!(",{},", streams.join(",")) +} + +fn query_key_hash(query_key: Option<&str>, query: &str) -> String { + sha256::digest(query_key.filter(|k| !k.is_empty()).unwrap_or(query)) +} + +fn list_condition(filter: &ListFilter) -> Condition { + let mut cond = Condition::all(); + if !filter.sources.is_empty() { + cond = cond.add(Column::Source.is_in(filter.sources.iter().cloned())); + } + if let Some(status) = &filter.status { + cond = cond.add(Column::Status.eq(status.as_str())); + } + if let Some(stream) = filter.stream.as_deref().filter(|s| !s.is_empty()) { + let pattern = format!("%,{},%", escape_like(stream)); + cond = cond.add(Expr::col(Column::Streams).like(LikeExpr::new(pattern).escape('\\'))); + } + if let Some(q) = filter.q.as_deref().filter(|q| !q.is_empty()) { + cond = cond.add( + Condition::any() + .add(contains_lowered(Column::Query, q)) + .add(contains_lowered(Column::Streams, q)), + ); + } + if let Some(starred) = filter.starred { + cond = cond.add(Column::Starred.eq(starred)); + } + if let Some(from) = filter.ran_from { + cond = cond.add(Column::UpdatedAt.gte(from)); + } + if let Some(to) = filter.ran_to { + cond = cond.add(Column::UpdatedAt.lte(to)); + } + cond +} + +/// Newest first; the id breaks ties so pages never overlap. +fn list_query(org_id: &str, user_email: &str, filter: &ListFilter) -> sea_orm::Select { + Entity::find() + .filter(Column::OrgId.eq(org_id)) + .filter(Column::UserEmail.eq(user_email)) + .filter(list_condition(filter)) + .order_by_desc(Column::UpdatedAt) + .order_by_desc(Column::Id) +} + +/// Keeps `kept_id` and the newest other unstarred rows of `source`, up to the per-source cap. +async fn prune_unstarred_over_cap( + conn: &C, + org_id: &str, + user_email: &str, + source: &str, + kept_id: &str, +) -> Result<(), errors::Error> { + // The run just recorded survives a timestamp tie or a clock that stepped back. + let unstarred = Condition::all() + .add(Column::OrgId.eq(org_id)) + .add(Column::UserEmail.eq(user_email)) + .add(Column::Source.eq(source)) + .add(Column::Starred.eq(false)) + .add(Column::Id.ne(kept_id)); + let Some((ran_at, id)) = Entity::find() + .filter(unstarred.clone()) + .select_only() + .column(Column::UpdatedAt) + .column(Column::Id) + .order_by_desc(Column::UpdatedAt) + .order_by_desc(Column::Id) + .offset(MAX_UNSTARRED_PER_SOURCE - 1) + .into_tuple::<(i64, String)>() + .one(conn) + .await? + else { + return Ok(()); + }; + let at_or_before_boundary = Condition::any().add(Column::UpdatedAt.lt(ran_at)).add( + Condition::all() + .add(Column::UpdatedAt.eq(ran_at)) + .add(Column::Id.lte(id)), + ); + Entity::delete_many() + .filter(unstarred) + .filter(at_or_before_boundary) + .exec(conn) + .await?; + Ok(()) +} + +/// Serializes one user's stars in an org, so the starred count below cannot go stale. +fn star_lock(backend: DbBackend, org_id: &str, user_email: &str) -> Option { + match backend { + DbBackend::Postgres => Some(Statement::from_sql_and_values( + backend, + "SELECT pg_advisory_xact_lock(hashtextextended($1, 0))", + [format!("query_history_star/{org_id}/{user_email}").into()], + )), + // MySQL has no transaction-scoped named lock, so it row-locks the user's rows instead. + DbBackend::MySql => Some(Statement::from_sql_and_values( + backend, + "SELECT COUNT(*) FROM query_history WHERE org_id = ? AND user_email = ? FOR UPDATE", + [org_id.into(), user_email.into()], + )), + // SQLite serializes every write transaction already. + DbBackend::Sqlite => None, + } +} + +/// Case-insensitive literal substring match: `%`, `_` and `\` match only themselves. +fn contains_lowered(column: Column, q: &str) -> SimpleExpr { + let pattern = format!("%{}%", escape_like(&q.to_lowercase())); + Expr::expr(Func::lower(Expr::col(column))).like(LikeExpr::new(pattern).escape('\\')) +} + +fn escape_like(text: &str) -> String { + let mut escaped = String::with_capacity(text.len()); + for c in text.chars() { + if matches!(c, '\\' | '%' | '_') { + escaped.push('\\'); + } + escaped.push(c); + } + escaped +} + #[cfg(test)] mod tests { - use sea_orm::{Database, DatabaseConnection, PaginatorTrait, Schema}; + use sea_orm::{Database, DatabaseConnection, PaginatorTrait}; use serde_json::json; use super::*; const A: &str = "a@x.com"; const B: &str = "b@x.com"; + const DAY_US: i64 = 86_400 * 1_000_000; async fn db() -> DatabaseConnection { let db = Database::connect("sqlite::memory:").await.unwrap(); - let backend = db.get_database_backend(); - let stmt = Schema::new(backend).create_table_from_entity(Entity); - db.execute(backend.build(&stmt)).await.unwrap(); + create_table_for_tests(&db).await.unwrap(); db } + fn run<'a>(user: &'a str, source: &'a str, query: &'a str, now_us: i64) -> NewRun<'a> { + NewRun { + org_id: "default", + user_email: user, + source, + query, + query_key: None, + streams: &[], + status: STATUS_OK, + context: json!({}), + now_us, + } + } + + async fn rec(db: &DatabaseConnection, run: NewRun<'_>) -> Model { + record_with(db, run).await.unwrap() + } + fn updated(outcome: SetStarred) -> Model { match outcome { - SetStarred::Updated(m) => m, + SetStarred::Updated(m) => *m, _ => panic!("expected the entry to be updated"), } } - async fn all(db: &DatabaseConnection, user: &str) -> Vec { - list_with(db, "default", user, None, None, 10_000, 0) + async fn listed(db: &DatabaseConnection, user: &str, filter: ListFilter) -> Vec { + list_with(db, "default", user, &filter, 10_000, 0) .await .unwrap() } + async fn all(db: &DatabaseConnection, user: &str) -> Vec { + listed(db, user, ListFilter::default()).await + } + + fn queries(rows: Vec) -> Vec { + rows.into_iter().map(|r| r.query).collect() + } + #[tokio::test] - async fn test_same_query_as_latest_updates_time_and_context() { + async fn test_same_key_merges_and_keeps_first_run_and_star() { let db = db().await; - let first = record_with(&db, "default", A, "up", json!({"chart": "line"}), 1) - .await - .unwrap(); - let again = record_with(&db, "default", A, "up", json!({"chart": "bar"}), 2) + let streams = vec!["k8s".to_string()]; + let first = rec( + &db, + NewRun { + query_key: Some("k8s|sql|up|"), + streams: &streams, + context: json!({"chart": "line"}), + ..run(A, "logs", "up", 1) + }, + ) + .await; + assert_eq!( + (first.run_count, first.created_at, first.updated_at), + (1, 1, 1) + ); + set_starred_with(&db, "default", A, &first.id, true) .await .unwrap(); + + let again = rec( + &db, + NewRun { + query_key: Some("k8s|sql|up|"), + status: STATUS_ERROR, + context: json!({"error": "boom"}), + ..run(A, "logs", "up", 5) + }, + ) + .await; assert_eq!(again.id, first.id); - let rows = all(&db, A).await; - assert_eq!(rows.len(), 1); - assert_eq!(rows[0].created_at, 2); - assert_eq!(rows[0].context, json!({"chart": "bar"})); - - record_with(&db, "default", A, "rate(x[5m])", json!({}), 3) - .await - .unwrap(); - record_with(&db, "default", A, "up", json!({}), 4) - .await - .unwrap(); - assert_eq!(all(&db, A).await.len(), 3, "only the latest entry dedupes"); - } - - #[tokio::test] - async fn test_dedupe_is_per_user() { - let db = db().await; - record_with(&db, "default", A, "up", json!({}), 1) - .await - .unwrap(); - record_with(&db, "default", B, "up", json!({}), 2) - .await - .unwrap(); + assert_eq!( + (again.run_count, again.created_at, again.updated_at), + (2, 1, 5) + ); + assert!(again.starred, "a merge keeps the star"); + assert_eq!(again.status, STATUS_ERROR); + assert_eq!(again.context, json!({"error": "boom"})); + assert_eq!(again.streams, ",k8s,", "the key already holds the streams"); assert_eq!(all(&db, A).await.len(), 1); - assert_eq!(all(&db, B).await.len(), 1); } #[tokio::test] - async fn test_cap_prunes_oldest_unstarred_and_keeps_starred() { + async fn test_the_same_search_on_two_days_is_one_row() { let db = db().await; - let starred = record_with(&db, "default", A, "starred", json!({}), 0) - .await - .unwrap(); + let key = Some("k8s|sql|up|"); + let monday = rec( + &db, + NewRun { + query_key: key, + ..run(A, "logs", "up", DAY_US) + }, + ) + .await; + let later = rec( + &db, + NewRun { + query_key: key, + ..run(A, "logs", "up", 40 * DAY_US) + }, + ) + .await; + assert_eq!(later.id, monday.id); + assert_eq!( + (later.run_count, later.created_at, later.updated_at), + (2, DAY_US, 40 * DAY_US) + ); + assert_eq!(all(&db, A).await.len(), 1); + } + + #[tokio::test] + async fn test_a_merge_sets_only_run_count_updated_at_context_and_status() { + let db = db().await; + let k8s = vec!["k8s".to_string()]; + let other = vec!["other".to_string()]; + let first = rec( + &db, + NewRun { + query_key: Some("k8s|sql|q|"), + streams: &k8s, + ..run(A, "logs", "SELECT 1", 1) + }, + ) + .await; + let merged = rec( + &db, + NewRun { + query_key: Some("k8s|sql|q|"), + streams: &other, + status: STATUS_ERROR, + context: json!({"error": "boom"}), + ..run(A, "logs", "SELECT 1 -- edited", 2) + }, + ) + .await; + assert_eq!(merged.id, first.id); + assert_eq!( + (merged.run_count, merged.created_at, merged.updated_at), + (2, 1, 2) + ); + assert_eq!( + (merged.status.as_str(), &merged.context), + (STATUS_ERROR, &json!({"error": "boom"})) + ); + assert_eq!( + (merged.query.as_str(), merged.streams.as_str()), + ("SELECT 1", ",k8s,"), + "query and streams stay as inserted" + ); + } + + #[tokio::test] + async fn test_a_merge_on_a_later_day_leaves_every_other_column_as_first_inserted() { + let db = db().await; + let k8s = vec!["k8s".to_string()]; + let first = rec( + &db, + NewRun { + query_key: Some("k8s|sql|q|"), + streams: &k8s, + context: json!({"run": 1}), + ..run(A, "logs", "q", DAY_US) + }, + ) + .await; + let starred = updated( + set_starred_with(&db, "default", A, &first.id, true) + .await + .unwrap(), + ); + let merged = rec( + &db, + NewRun { + query_key: Some("k8s|sql|q|"), + streams: &[], + status: STATUS_ERROR, + context: json!({"run": 2}), + ..run(A, "logs", "q edited", 3 * DAY_US) + }, + ) + .await; + assert_eq!( + merged, + Model { + run_count: 2, + updated_at: 3 * DAY_US, + context: json!({"run": 2}), + status: STATUS_ERROR.to_string(), + ..starred + } + ); + assert_eq!(all(&db, A).await, vec![merged]); + } + + #[tokio::test] + async fn test_merge_is_per_user_per_source_and_per_key() { + let db = db().await; + rec(&db, run(A, "logs", "up", 1)).await; + rec(&db, run(B, "logs", "up", 2)).await; + rec(&db, run(A, "traces", "up", 3)).await; + rec( + &db, + NewRun { + query_key: Some("up|2026-10-28"), + ..run(A, "logs", "up", 4) + }, + ) + .await; + assert_eq!(all(&db, A).await.len(), 3); + assert_eq!(all(&db, B).await.len(), 1); + let merged = rec(&db, run(A, "logs", "up", 5)).await; + assert_eq!(merged.run_count, 2, "no key falls back to the query text"); + } + + #[tokio::test] + async fn test_new_key_prunes_over_the_per_source_cap_and_keeps_starred() { + let db = db().await; + let starred = rec(&db, run(A, "logs", "starred", 0)).await; set_starred_with(&db, "default", A, &starred.id, true) .await .unwrap(); - for i in 1..=MAX_UNSTARRED_PER_USER as i64 { - record_with(&db, "default", A, &format!("q{i}"), json!({}), i) - .await - .unwrap(); + for i in 1..=MAX_UNSTARRED_PER_SOURCE as i64 { + rec(&db, run(A, "logs", &format!("q{i}"), i)).await; } - record_with(&db, "default", B, "other user", json!({}), 1) - .await - .unwrap(); - assert_eq!(all(&db, A).await.len(), 1001); + rec(&db, run(A, "traces", "other source", 0)).await; + rec(&db, run(B, "logs", "other user", 0)).await; + let logs = |rows: Vec| rows.into_iter().filter(|r| r.source == "logs").count(); + assert_eq!(logs(all(&db, A).await), 501); - record_with(&db, "default", A, "q1001", json!({}), 1001) - .await - .unwrap(); + rec(&db, run(A, "logs", "q1", 1_000)).await; + assert_eq!(logs(all(&db, A).await), 501, "a merge never prunes"); + + rec(&db, run(A, "logs", "q501", 1_001)).await; let rows = all(&db, A).await; - assert_eq!(rows.len(), 1001); - assert!(rows.iter().all(|r| r.query != "q1"), "oldest is pruned"); - assert!(rows.iter().any(|r| r.query == "q2")); + assert_eq!(logs(rows.clone()), 501); + assert!(rows.iter().all(|r| r.query != "q2"), "oldest is pruned"); + assert!( + rows.iter().any(|r| r.query == "q1"), + "the merge refreshed q1" + ); assert!(rows.iter().any(|r| r.id == starred.id && r.starred)); + assert!(rows.iter().any(|r| r.source == "traces")); assert_eq!(all(&db, B).await.len(), 1); } #[tokio::test] async fn test_cap_holds_when_timestamps_tie() { let db = db().await; - for i in 0..=MAX_UNSTARRED_PER_USER { - record_with(&db, "default", A, &format!("q{i}"), json!({}), 7) - .await - .unwrap(); + for i in 0..=MAX_UNSTARRED_PER_SOURCE { + rec(&db, run(A, "logs", &format!("q{i}"), 7)).await; } - assert_eq!(all(&db, A).await.len() as u64, MAX_UNSTARRED_PER_USER); + assert_eq!(all(&db, A).await.len() as u64, MAX_UNSTARRED_PER_SOURCE); } #[tokio::test] - async fn test_list_filters_newest_first() { + async fn test_reaper_uses_updated_at_and_keeps_starred() { let db = db().await; - let up = record_with(&db, "default", A, "up", json!({}), 1) + let now = 100 * DAY_US; + let cutoff = now - 14 * DAY_US; + let old = rec(&db, run(A, "logs", "old", now - 15 * DAY_US)).await; + let old_starred = rec(&db, run(A, "logs", "old starred", now - 15 * DAY_US)).await; + set_starred_with(&db, "default", A, &old_starred.id, true) .await .unwrap(); - record_with(&db, "default", A, "rate(http_total[5m])", json!({}), 2) - .await - .unwrap(); - record_with(&db, "default", A, "sum(http_total)", json!({}), 3) - .await - .unwrap(); - record_with(&db, "other", A, "http_total", json!({}), 4) + let rerun = rec(&db, run(A, "logs", "rerun", now - 20 * DAY_US)).await; + rec(&db, run(A, "logs", "rerun", now)).await; + + assert_eq!(delete_unstarred_before_with(&db, cutoff).await.unwrap(), 1); + let ids: Vec = all(&db, A).await.into_iter().map(|r| r.id).collect(); + assert!(!ids.contains(&old.id)); + assert!(ids.contains(&old_starred.id)); + assert!(ids.contains(&rerun.id), "a recent run keeps an old row"); + } + + async fn row_at(db: &DatabaseConnection, id: &str, created_at: i64, updated_at: i64) { + db.execute_unprepared(&format!( + "INSERT INTO query_history (id, org_id, user_email, query, context, created_at, \ + updated_at, source) VALUES ('{id}', 'default', '{A}', 'q {id}', '{{}}', {created_at}, \ + {updated_at}, 'logs')" + )) + .await + .unwrap(); + } + + #[tokio::test] + async fn test_the_reaper_needs_both_runs_before_the_cutoff() { + let db = db().await; + row_at(&db, "stale", 50, 60).await; + row_at(&db, "fresh", 50, 500).await; + row_at(&db, "old-node", 500, 0).await; + assert_eq!(delete_unstarred_before_with(&db, 100).await.unwrap(), 1); + let mut ids: Vec = all(&db, A).await.into_iter().map(|r| r.id).collect(); + ids.sort(); + assert_eq!(ids, vec!["fresh", "old-node"]); + } + + #[tokio::test] + async fn test_order_window_and_prune_read_updated_at_alone() { + let db = db().await; + for i in 1..=MAX_UNSTARRED_PER_SOURCE as i64 { + row_at(&db, &format!("r{i:04}"), 1_000_000 - i, i * 10).await; + } + let rows = all(&db, A).await; + assert_eq!( + rows.iter() + .take(2) + .map(|r| r.id.as_str()) + .collect::>(), + vec!["r0500", "r0499"], + "newest updated_at first, whatever created_at says" + ); + let window = ListFilter { + ran_from: Some(4_980), + ran_to: Some(4_990), + ..Default::default() + }; + assert_eq!( + listed(&db, A, window) + .await + .iter() + .map(|r| r.id.as_str()) + .collect::>(), + vec!["r0499", "r0498"] + ); + rec(&db, run(A, "logs", "new", 1)).await; + let ids: Vec = all(&db, A).await.into_iter().map(|r| r.id).collect(); + assert_eq!(ids.len() as u64, MAX_UNSTARRED_PER_SOURCE); + assert!( + !ids.contains(&"r0001".to_string()), + "lowest updated_at is pruned" + ); + assert!(ids.contains(&"r0002".to_string())); + } + + #[tokio::test] + async fn test_count_matches_the_list_filters() { + let db = db().await; + let starred = rec(&db, run(A, "logs", "a", 1)).await; + set_starred_with(&db, "default", A, &starred.id, true) .await .unwrap(); + rec(&db, run(A, "logs", "b", 2)).await; + rec(&db, run(A, "traces", "c", 3)).await; + rec(&db, run(B, "logs", "d", 4)).await; + let count = |sources: &[&str], starred: Option| ListFilter { + sources: sources.iter().map(|s| s.to_string()).collect(), + starred, + ..Default::default() + }; + assert_eq!( + count_with(&db, "default", A, &count(&[], None)) + .await + .unwrap(), + 3 + ); + assert_eq!( + count_with(&db, "default", A, &count(&["logs"], None)) + .await + .unwrap(), + 2 + ); + assert_eq!( + count_with(&db, "default", A, &count(&["logs"], Some(false))) + .await + .unwrap(), + 1 + ); + assert_eq!( + count_with(&db, "default", A, &count(&[], Some(true))) + .await + .unwrap(), + 1 + ); + } + + #[tokio::test] + async fn test_a_new_row_survives_the_cap_when_others_ran_later() { + let db = db().await; + for i in 0..MAX_UNSTARRED_PER_SOURCE as i64 { + rec(&db, run(A, "logs", &format!("q{i}"), 1_000 + i)).await; + } + // The clock stepped back, or every row ties: the new run must still be kept. + let new = rec(&db, run(A, "logs", "after the step back", 5)).await; + let rows = all(&db, A).await; + assert_eq!(rows.len() as u64, MAX_UNSTARRED_PER_SOURCE); + assert!(rows.iter().any(|r| r.id == new.id)); + assert!( + rows.iter().all(|r| r.query != "q0"), + "the oldest other row is pruned" + ); + + let tie = rec(&db, run(A, "logs", "tie", 1_001)).await; + let rows = all(&db, A).await; + assert_eq!(rows.len() as u64, MAX_UNSTARRED_PER_SOURCE); + assert!(rows.iter().any(|r| r.id == tie.id)); + } + + #[tokio::test] + async fn test_list_order_is_served_by_the_updated_at_indexes_on_sqlite() { + let db = db().await; + for i in 0..20 { + rec(&db, run(A, "logs", &format!("q{i}"), i)).await; + } + let filters = [ + ListFilter::default(), + ListFilter { + sources: vec!["logs".to_string()], + ..Default::default() + }, + ]; + for filter in filters { + let sql = list_query("default", A, &filter) + .limit(50) + .build(sea_orm::DbBackend::Sqlite) + .to_string(); + let plan: Vec = db + .query_all(sea_orm::Statement::from_string( + sea_orm::DbBackend::Sqlite, + format!("EXPLAIN QUERY PLAN {sql}"), + )) + .await + .unwrap() + .iter() + .map(|r| r.try_get::("", "detail").unwrap()) + .collect(); + let plan = plan.join(" | "); + assert!(plan.contains("updated_at_idx"), "{plan}"); + assert!(!plan.contains("TEMP B-TREE FOR ORDER BY"), "{plan}"); + } + } + + #[tokio::test] + async fn test_concurrent_new_rows_keep_the_per_source_cap() { + let db = db().await; + for i in 0..MAX_UNSTARRED_PER_SOURCE as i64 { + rec(&db, run(A, "logs", &format!("q{i}"), i)).await; + } + let queries: Vec = (0..8).map(|i| format!("new{i}")).collect(); + let results = futures::future::join_all( + queries + .iter() + .enumerate() + .map(|(i, q)| record_with(&db, run(A, "logs", q, 10_000 + i as i64))), + ) + .await; + assert!(results.iter().all(Result::is_ok)); + assert_eq!(all(&db, A).await.len() as u64, MAX_UNSTARRED_PER_SOURCE); + } + + #[tokio::test] + async fn test_list_orders_by_last_run_and_filters() { + let db = db().await; + let k8s = vec!["k8s_logs".to_string(), "default".to_string()]; + let app = vec!["app_logs".to_string()]; + let up = rec( + &db, + NewRun { + streams: &k8s, + ..run(A, "logs", "level = 'error'", 1) + }, + ) + .await; + rec( + &db, + NewRun { + streams: &app, + status: STATUS_ERROR, + ..run(A, "logs", "SELECT * FROM app", 2) + }, + ) + .await; + rec(&db, run(A, "traces", "service = 'cart'", 3)).await; + rec(&db, run(A, "metrics", "rate(http_total[5m])", 4)).await; + rec(&db, run(A, "logs", "level = 'error'", 5)).await; set_starred_with(&db, "default", A, &up.id, true) .await .unwrap(); - let queries = |rows: Vec| rows.into_iter().map(|r| r.query).collect::>(); + let f = |build: fn(&mut ListFilter)| { + let mut filter = ListFilter::default(); + build(&mut filter); + filter + }; assert_eq!( queries(all(&db, A).await), - vec!["sum(http_total)", "rate(http_total[5m])", "up"] + vec![ + "level = 'error'", + "rate(http_total[5m])", + "service = 'cart'", + "SELECT * FROM app" + ] ); - let starred = list_with(&db, "default", A, Some(true), None, 50, 0) - .await - .unwrap(); - assert_eq!(queries(starred), vec!["up"]); - let matched = list_with(&db, "default", A, None, Some("http_total"), 50, 0) - .await - .unwrap(); + let by_source = f(|f| f.sources = vec!["logs".into(), "traces".into()]); + assert_eq!(queries(listed(&db, A, by_source).await).len(), 3); + let failed = f(|f| f.status = Some(STATUS_ERROR.into())); assert_eq!( - queries(matched), - vec!["sum(http_total)", "rate(http_total[5m])"] + queries(listed(&db, A, failed).await), + vec!["SELECT * FROM app"] ); - let page = list_with(&db, "default", A, None, None, 1, 1) + let stream = f(|f| f.stream = Some("default".into())); + assert_eq!( + queries(listed(&db, A, stream).await), + vec!["level = 'error'"], + "a merge keeps the streams" + ); + let stream = f(|f| f.stream = Some("app_logs".into())); + assert_eq!( + queries(listed(&db, A, stream).await), + vec!["SELECT * FROM app"] + ); + let partial = f(|f| f.stream = Some("app".into())); + assert!(listed(&db, A, partial).await.is_empty(), "stream is exact"); + let q_stream = f(|f| f.q = Some("APP_LO".into())); + assert_eq!( + queries(listed(&db, A, q_stream).await), + vec!["SELECT * FROM app"] + ); + let q_query = f(|f| f.q = Some("HTTP".into())); + assert_eq!( + queries(listed(&db, A, q_query).await), + vec!["rate(http_total[5m])"] + ); + let starred = f(|f| f.starred = Some(true)); + assert_eq!( + queries(listed(&db, A, starred).await), + vec!["level = 'error'"] + ); + let window = f(|f| { + f.ran_from = Some(2); + f.ran_to = Some(3); + }); + assert_eq!( + queries(listed(&db, A, window).await), + vec!["service = 'cart'", "SELECT * FROM app"] + ); + let page = list_with(&db, "default", A, &ListFilter::default(), 1, 1) .await .unwrap(); assert_eq!(queries(page), vec!["rate(http_total[5m])"]); @@ -509,19 +1079,16 @@ mod tests { .into_iter() .enumerate() { - record_with(&db, "default", A, query, json!({}), i as i64) - .await - .unwrap(); + rec(&db, run(A, "metrics", query, i as i64)).await; } let search = |q: &'static str| { let db = &db; async move { - let mut rows: Vec = list_with(db, "default", A, None, Some(q), 50, 0) - .await - .unwrap() - .into_iter() - .map(|r| r.query) - .collect(); + let filter = ListFilter { + q: Some(q.to_string()), + ..Default::default() + }; + let mut rows = queries(listed(db, A, filter).await); rows.sort(); rows } @@ -535,9 +1102,9 @@ mod tests { // Postgres LIKE is case-sensitive, so the filter must lower both sides. #[test] fn test_search_sql_lowers_and_escapes_on_postgres() { - use sea_orm::{DbBackend, QueryTrait}; + use sea_orm::DbBackend; let sql = Entity::find() - .filter(query_matches("50%_X")) + .filter(contains_lowered(Column::Query, "50%_X")) .build(DbBackend::Postgres) .to_string(); assert!( @@ -546,12 +1113,95 @@ mod tests { ); } + #[test] + fn test_star_lock_is_narrow_on_each_backend() { + let pg = star_lock(DbBackend::Postgres, "default", A) + .unwrap() + .to_string(); + assert!(pg.contains("pg_advisory_xact_lock"), "{pg}"); + assert!( + !pg.contains("FOR UPDATE"), + "no rows are locked or read: {pg}" + ); + let mysql = star_lock(DbBackend::MySql, "default", A) + .unwrap() + .to_string(); + assert!( + mysql.starts_with("SELECT COUNT(*)") && mysql.ends_with("FOR UPDATE"), + "{mysql}" + ); + assert!(star_lock(DbBackend::Sqlite, "default", A).is_none()); + } + + #[test] + fn test_upsert_sql_targets_the_unique_key_on_postgres() { + use sea_orm::DbBackend; + let sql = Entity::insert(ActiveModel { + id: Set("x".into()), + ..Default::default() + }) + .on_conflict( + OnConflict::columns([Column::OrgId, Column::QueryKey]) + .value( + Column::RunCount, + Expr::col((Entity, Column::RunCount)).add(1), + ) + .to_owned(), + ) + .build(DbBackend::Postgres) + .to_string(); + assert!( + sql.contains(r#""run_count" = "query_history"."run_count" + 1"#), + "{sql}" + ); + } + + #[test] + fn test_streams_round_trip_through_the_comma_wrapped_form() { + let streams = vec!["a".to_string(), "b".to_string()]; + assert_eq!(encode_streams(&streams), ",a,b,"); + assert_eq!(decode_streams(",a,b,"), streams); + assert_eq!(encode_streams(&[]), ""); + assert!(decode_streams("").is_empty()); + } + + #[test] + fn test_query_key_is_a_sha256_of_the_key_or_the_query() { + assert_eq!(query_key_hash(Some("k"), "q"), sha256::digest("k")); + assert_eq!(query_key_hash(None, "q"), sha256::digest("q")); + assert_eq!(query_key_hash(Some(""), "q"), sha256::digest("q")); + assert_eq!(query_key_hash(None, "q").len(), 64); + } + + #[tokio::test] + async fn test_clear_scopes_by_source_and_keeps_starred_unless_asked() { + let db = db().await; + let star = rec(&db, run(A, "logs", "starred", 1)).await; + set_starred_with(&db, "default", A, &star.id, true) + .await + .unwrap(); + rec(&db, run(A, "logs", "a", 2)).await; + rec(&db, run(A, "traces", "b", 3)).await; + rec(&db, run(A, "metrics", "c", 4)).await; + rec(&db, run(B, "logs", "a", 5)).await; + + let logs = vec!["logs".to_string()]; + assert_eq!( + clear_with(&db, "default", A, &logs, false).await.unwrap(), + 1 + ); + assert_eq!(queries(all(&db, A).await), vec!["c", "b", "starred"]); + assert_eq!(clear_with(&db, "default", A, &[], false).await.unwrap(), 2); + assert_eq!(queries(all(&db, A).await), vec!["starred"]); + assert_eq!(clear_with(&db, "default", A, &[], true).await.unwrap(), 1); + assert!(all(&db, A).await.is_empty()); + assert_eq!(all(&db, B).await.len(), 1, "only the caller's rows"); + } + #[tokio::test] async fn test_by_id_operations_are_scoped_to_org_and_user() { let db = db().await; - let entry = record_with(&db, "default", A, "up", json!({}), 1) - .await - .unwrap(); + let entry = rec(&db, run(A, "metrics", "up", 1)).await; assert!(matches!( set_starred_with(&db, "default", B, &entry.id, true) @@ -583,9 +1233,7 @@ mod tests { #[tokio::test] async fn test_starring_twice_still_returns_the_entry() { let db = db().await; - let entry = record_with(&db, "default", A, "up", json!({}), 1) - .await - .unwrap(); + let entry = rec(&db, run(A, "metrics", "up", 1)).await; for _ in 0..2 { let starred = updated( set_starred_with(&db, "default", A, &entry.id, true) @@ -609,13 +1257,12 @@ mod tests { } #[tokio::test] - async fn test_starring_past_the_cap_is_refused_per_user() { + async fn test_starring_past_the_cap_is_refused_across_sources() { let db = db().await; let mut ids = Vec::new(); for i in 0..=MAX_STARRED_PER_USER as i64 { - let entry = record_with(&db, "default", A, &format!("q{i}"), json!({}), i) - .await - .unwrap(); + let source = SOURCES[i as usize % 3]; + let entry = rec(&db, run(A, source, &format!("q{i}"), i)).await; assert!(!entry.starred, "recording never stars"); ids.push(entry.id); } @@ -638,22 +1285,12 @@ mod tests { .starred, "re-starring an already starred entry at the cap still succeeds" ); - let other_user = record_with(&db, "default", B, "up", json!({}), 1) - .await - .unwrap(); + let other_user = rec(&db, run(B, "logs", "up", 1)).await; updated( set_starred_with(&db, "default", B, &other_user.id, true) .await .unwrap(), ); - let other_org = record_with(&db, "other", A, "up", json!({}), 1) - .await - .unwrap(); - updated( - set_starred_with(&db, "other", A, &other_org.id, true) - .await - .unwrap(), - ); updated( set_starred_with(&db, "default", A, &at_cap[0], false) @@ -672,68 +1309,167 @@ mod tests { } #[tokio::test] - async fn test_dedupe_keeps_the_starred_flag() { + async fn test_concurrent_stars_never_pass_the_cap() { let db = db().await; - let entry = record_with(&db, "default", A, "up", json!({}), 1) + let mut ids = Vec::new(); + for i in 0..MAX_STARRED_PER_USER as i64 + 5 { + ids.push(rec(&db, run(A, "logs", &format!("q{i}"), i)).await.id); + } + let (racing, below_cap) = ids.split_at(6); + for id in &below_cap[..MAX_STARRED_PER_USER as usize - 1] { + updated(set_starred_with(&db, "default", A, id, true).await.unwrap()); + } + let outcomes = futures::future::join_all( + racing + .iter() + .map(|id| set_starred_with(&db, "default", A, id, true)), + ) + .await; + let starred_now = outcomes + .into_iter() + .filter(|o| matches!(o, Ok(SetStarred::Updated(_)))) + .count(); + assert_eq!(starred_now, 1); + let starred = Entity::find() + .filter(Column::Starred.eq(true)) + .count(&db) .await .unwrap(); - updated( - set_starred_with(&db, "default", A, &entry.id, true) - .await - .unwrap(), - ); - let again = record_with(&db, "default", A, "up", json!({"chart": "bar"}), 2) - .await - .unwrap(); - assert_eq!(again.id, entry.id); - assert!(again.starred); - assert_eq!(again.org_id, "default"); - assert_eq!(again.user_email, A); - assert_eq!(again.query, "up"); + assert_eq!(starred, MAX_STARRED_PER_USER); } #[tokio::test] - async fn test_retention_deletes_only_old_unstarred_rows() { + async fn test_concurrent_first_stars_from_zero_stop_at_the_cap() { let db = db().await; - let day_us = 86_400 * 1_000_000; - let now = 100 * day_us; - let old = record_with(&db, "default", A, "old", json!({}), now - 15 * day_us) - .await - .unwrap(); - let old_starred = record_with( - &db, - "default", - A, - "old starred", - json!({}), - now - 15 * day_us, + let mut ids = Vec::new(); + for i in 0..MAX_STARRED_PER_USER as i64 + 10 { + ids.push(rec(&db, run(A, "logs", &format!("q{i}"), i)).await.id); + } + let outcomes = futures::future::join_all( + ids.iter() + .map(|id| set_starred_with(&db, "default", A, id, true)), ) - .await - .unwrap(); - set_starred_with(&db, "default", A, &old_starred.id, true) - .await - .unwrap(); - record_with(&db, "default", A, "fresh", json!({}), now) + .await; + let starred_now = outcomes + .iter() + .filter(|o| matches!(o, Ok(SetStarred::Updated(_)))) + .count(); + let refused = outcomes + .iter() + .filter(|o| matches!(o, Ok(SetStarred::StarredCapReached))) + .count(); + assert_eq!((starred_now, refused), (MAX_STARRED_PER_USER as usize, 10)); + let starred = Entity::find() + .filter(Column::Starred.eq(true)) + .count(&db) .await .unwrap(); + assert_eq!(starred, MAX_STARRED_PER_USER); + } - let deleted = delete_unstarred_before_with(&db, now - 14 * day_us) + /// One-connection pools on a throwaway CI Postgres schema, so calls truly race. + async fn postgres_pools( + n: usize, + ) -> Option<(DatabaseConnection, String, Vec)> { + let dsn = std::env::var("ZO_META_POSTGRES_DSN").ok()?; + // The SQLite lane sets the variable empty rather than unsetting it. + if dsn.trim().is_empty() { + return None; + } + let schema = format!("qh_test_{}", config::ider::generate().to_ascii_lowercase()); + let admin = Database::connect(&dsn).await.unwrap(); + admin + .execute_unprepared(&format!(r#"CREATE SCHEMA "{schema}""#)) .await .unwrap(); - assert_eq!(deleted, 1); - let ids: Vec = all(&db, A).await.into_iter().map(|r| r.id).collect(); - assert!(!ids.contains(&old.id)); - assert!(ids.contains(&old_starred.id)); - assert_eq!(ids.len(), 2); + let mut pools = Vec::new(); + for _ in 0..n { + let mut options = sea_orm::ConnectOptions::new(dsn.clone()); + options + .max_connections(1) + .min_connections(1) + .set_schema_search_path(schema.clone()); + pools.push(Database::connect(options).await.unwrap()); + } + create_table_for_tests(&pools[0]).await.unwrap(); + Some((admin, schema, pools)) + } + + async fn drop_schema(admin: DatabaseConnection, schema: &str, pools: Vec) { + for pool in pools { + pool.close().await.unwrap(); + } + admin + .execute_unprepared(&format!(r#"DROP SCHEMA "{schema}" CASCADE"#)) + .await + .unwrap(); + } + + #[tokio::test] + async fn test_postgres_concurrent_first_stars_stop_at_the_cap_when_ci_provides_it() { + let Some((admin, schema, pools)) = postgres_pools(8).await else { + return; + }; + let mut ids = Vec::new(); + for i in 0..MAX_STARRED_PER_USER as i64 + 16 { + ids.push(rec(&pools[0], run(A, "logs", &format!("q{i}"), i)).await.id); + } + let outcomes = futures::future::join_all( + ids.iter() + .enumerate() + .map(|(i, id)| set_starred_with(&pools[i % pools.len()], "default", A, id, true)), + ) + .await; + let starred_now = outcomes + .iter() + .filter(|o| matches!(o, Ok(SetStarred::Updated(_)))) + .count(); + assert_eq!(starred_now, MAX_STARRED_PER_USER as usize); + let starred = Entity::find() + .filter(Column::Starred.eq(true)) + .count(&pools[0]) + .await + .unwrap(); + assert_eq!(starred, MAX_STARRED_PER_USER); + drop_schema(admin, &schema, pools).await; + } + + #[tokio::test] + async fn test_postgres_concurrent_new_rows_keep_the_cap_when_ci_provides_it() { + let Some((admin, schema, pools)) = postgres_pools(8).await else { + return; + }; + for i in 0..MAX_UNSTARRED_PER_SOURCE as i64 { + rec(&pools[0], run(A, "logs", &format!("q{i}"), i)).await; + } + let queries: Vec = (0..16).map(|i| format!("new{i}")).collect(); + let results = futures::future::join_all(queries.iter().enumerate().map(|(i, q)| { + record_with( + &pools[i % pools.len()], + run(A, "logs", q, 10_000 + i as i64), + ) + })) + .await; + assert!(results.iter().all(Result::is_ok)); + assert_eq!( + all(&pools[0], A).await.len() as u64, + MAX_UNSTARRED_PER_SOURCE + ); + drop_schema(admin, &schema, pools).await; } #[tokio::test] async fn test_cleanup_by_org_user_and_email() { let db = db().await; for (org, user) in [("default", A), ("default", B), ("other", A), ("other", B)] { - record_with(&db, org, user, "up", json!({}), 1) - .await - .unwrap(); + rec( + &db, + NewRun { + org_id: org, + ..run(user, "logs", "up", 1) + }, + ) + .await; } assert_eq!(delete_by_user_with(&db, "default", A).await.unwrap(), 1); assert!(all(&db, A).await.is_empty()); diff --git a/src/infra/src/table/slack.rs b/src/infra/src/table/slack.rs index 6e2cd0e9e1..3fae8cb908 100644 --- a/src/infra/src/table/slack.rs +++ b/src/infra/src/table/slack.rs @@ -13,9 +13,9 @@ // You should have received a copy of the GNU Affero General Public License // along with this program. If not, see . -use sea_orm::{ConnectionTrait, DbErr}; +use sea_orm::{DatabaseConnection, DbErr}; /// Creates the Slack tables, `slack_alert_messages` included, without replaying the migrator. -pub async fn create_tables_for_test(conn: &C) -> Result<(), DbErr> { +pub async fn create_tables_for_test(conn: &DatabaseConnection) -> Result<(), DbErr> { super::migration::create_slack_tables(conn).await } diff --git a/src/infra/src/table/status_pages.rs b/src/infra/src/table/status_pages.rs index 23ef43bf0e..41dff4ebc6 100644 --- a/src/infra/src/table/status_pages.rs +++ b/src/infra/src/table/status_pages.rs @@ -31,21 +31,26 @@ use super::entity::{ status_page_audit_log, status_page_check_snoozes, status_page_component_checks, status_page_components, status_page_custom_domains, status_page_notice_components, status_page_notice_updates, status_page_notices, status_page_snapshots, status_pages, - synthetics_checks, + synthetics_checks, synthetics_runs, }; use crate::{ db::{get_orm_client_ro, get_orm_client_rw}, errors, }; -/// The three columns the auto-incident engine needs per mapped check — +/// The columns the auto-incident engine needs per mapped check — /// deliberately not the whole row (config/secrets are KB each). #[derive(Clone, Debug)] pub struct CheckStateRow { pub id: String, + pub org_id: String, pub last_check_status: i32, pub consecutive_failures: i32, pub last_triggered_at: i64, + /// `completed_at` of the check's latest completed run, filled by + /// [`latest_run_completions`]; `None` until read or when no run completed + /// inside the lookback window. + pub last_completed_at: Option, } pub async fn get_page_by_slug( @@ -117,6 +122,7 @@ pub async fn get_check_states( let mut q = synthetics_checks::Entity::find() .select_only() .column(synthetics_checks::Column::Id) + .column(synthetics_checks::Column::OrgId) .column(synthetics_checks::Column::LastCheckStatus) .column(synthetics_checks::Column::ConsecutiveFailures) .column(synthetics_checks::Column::LastTriggeredAt) @@ -126,22 +132,31 @@ pub async fn get_check_states( q = q.filter(synthetics_checks::Column::UpdatedAt.gt(watermark)); } let rows = q - .into_tuple::<(String, i32, i32, i64, i64)>() + .into_tuple::<(String, String, i32, i32, i64, i64)>() .all(conn) .await?; let mut max_updated = since.unwrap_or(0); let out = rows .into_iter() .map( - |(id, last_check_status, consecutive_failures, last_triggered_at, updated_at)| { + |( + id, + org_id, + last_check_status, + consecutive_failures, + last_triggered_at, + updated_at, + )| { if updated_at > max_updated { max_updated = updated_at; } CheckStateRow { id, + org_id, last_check_status, consecutive_failures, last_triggered_at, + last_completed_at: None, } }, ) @@ -149,6 +164,43 @@ pub async fn get_check_states( Ok((out, max_updated)) } +/// Latest `synthetics_runs.completed_at` per check, for runs scheduled at or +/// after `scheduled_since`. Keys are check ids; checks with no completed run in +/// the window are absent. +/// +/// Region-local like the runs table itself: it sees the runs this region's +/// scheduler enqueued, which is the region that holds the job-cluster claim and +/// therefore runs the incident engine. The `org_id` + `synthetics_id` + +/// `scheduled_ts` predicates match `synthetics_runs_monitor_idx`, so the read +/// is an index range per mapped check, not a table scan. +pub async fn latest_run_completions( + conn: &C, + checks: &[(String, String)], + scheduled_since: i64, +) -> Result, errors::Error> { + if checks.is_empty() { + return Ok(Default::default()); + } + let orgs: HashSet<&str> = checks.iter().map(|(o, _)| o.as_str()).collect(); + let ids: HashSet<&str> = checks.iter().map(|(_, id)| id.as_str()).collect(); + let rows = synthetics_runs::Entity::find() + .select_only() + .column(synthetics_runs::Column::SyntheticsId) + .column_as(synthetics_runs::Column::CompletedAt.max(), "completed_at") + .filter(synthetics_runs::Column::OrgId.is_in(orgs.into_iter().map(str::to_owned))) + .filter(synthetics_runs::Column::SyntheticsId.is_in(ids.into_iter().map(str::to_owned))) + .filter(synthetics_runs::Column::ScheduledTs.gte(scheduled_since)) + .filter(synthetics_runs::Column::CompletedAt.is_not_null()) + .group_by(synthetics_runs::Column::SyntheticsId) + .into_tuple::<(String, Option)>() + .all(conn) + .await?; + Ok(rows + .into_iter() + .filter_map(|(id, at)| at.map(|at| (id, at))) + .collect()) +} + /// Notices the engine and snapshots need: not deleted, and open or resolved /// after `resolved_cutoff` (90d for history; the merge window rides within). pub async fn list_notices_since( diff --git a/src/infra/src/table/synthetics_checks.rs b/src/infra/src/table/synthetics_checks.rs index 29ea7415e0..fa153b4583 100644 --- a/src/infra/src/table/synthetics_checks.rs +++ b/src/infra/src/table/synthetics_checks.rs @@ -269,6 +269,17 @@ pub async fn count( org_id: &str, params: &ListSyntheticsParams, ) -> Result { + if has_json_filters(params) { + // location/tag live in JSON text columns: filter in Rust so count and + // page agree. + let models = Entity::find() + .filter(Column::OrgId.eq(org_id)) + .apply_filters(params) + .order_by_asc(Column::Name) + .all(conn) + .await?; + return Ok(filter_and_paginate(models, params).1); + } let q = Entity::find() .filter(Column::OrgId.eq(org_id)) .apply_filters(params); @@ -1040,6 +1051,50 @@ async fn get_model( .await } +fn has_json_filters(params: &ListSyntheticsParams) -> bool { + params.location.is_some() || params.tag.is_some() +} + +fn json_array_contains(v: &serde_json::Value, needle: &str) -> bool { + v.as_array() + .is_some_and(|a| a.iter().any(|x| x.as_str() == Some(needle))) +} + +/// Applies the `location` / `tag` exact-match filters (JSON text columns, so +/// no DB operators) and then page/page_size over the filtered set. Returns the +/// requested page and the filtered total; `models` must already be ordered. +fn filter_and_paginate( + models: Vec, + params: &ListSyntheticsParams, +) -> (Vec, u64) { + let filtered: Vec<_> = models + .into_iter() + .filter(|m| { + params + .location + .as_deref() + .is_none_or(|l| json_array_contains(&m.locations, l)) + && params + .tag + .as_deref() + .is_none_or(|t| json_array_contains(&m.tags, t)) + }) + .collect(); + let total = filtered.len() as u64; + if let (Some(page_size), Some(page)) = (params.page_size, params.page) + && page_size > 0 + { + let start = page.saturating_mul(page_size) as usize; + let page_rows = filtered + .into_iter() + .skip(start) + .take(page_size as usize) + .collect(); + return (page_rows, total); + } + (filtered, total) +} + async fn list_models( conn: &C, org_id: &str, @@ -1050,6 +1105,11 @@ async fn list_models( .apply_filters(params) .order_by_asc(Column::Name); + if has_json_filters(params) { + let models = q.all(conn).await?; + return Ok(filter_and_paginate(models, params).0); + } + if let (Some(page_size), Some(page)) = (params.page_size, params.page) && page_size > 0 { @@ -1227,6 +1287,63 @@ pub(crate) mod tests { } } + #[test] + fn filter_and_paginate_combined_filters_and_pages() { + let mk = |id: &str, locs: &[&str], tags: &[&str]| { + let mut m = make_model(); + m.id = id.to_string(); + m.locations = serde_json::json!(locs); + m.tags = serde_json::json!(tags); + m + }; + let models = vec![ + mk("a", &["l1"], &["prod"]), + mk("b", &["l1", "l2"], &["prod", "x"]), + mk("c", &["l2"], &["prod"]), + mk("d", &["l1"], &["dev"]), + mk("e", &["l1"], &["prod"]), + ]; + let params = |loc: Option<&str>, tag: Option<&str>, page, size| ListSyntheticsParams { + location: loc.map(String::from), + tag: tag.map(String::from), + page, + page_size: size, + ..Default::default() + }; + // location only + let (rows, total) = + filter_and_paginate(models.clone(), ¶ms(Some("l2"), None, None, None)); + assert_eq!((rows.len(), total), (2, 2)); + // exact match, not substring + let (_, total) = filter_and_paginate(models.clone(), ¶ms(Some("l"), None, None, None)); + assert_eq!(total, 0); + // combined + let (rows, total) = filter_and_paginate( + models.clone(), + ¶ms(Some("l1"), Some("prod"), None, None), + ); + assert_eq!(total, 3); + assert_eq!( + rows.iter().map(|m| m.id.as_str()).collect::>(), + ["a", "b", "e"] + ); + // page boundaries: total stays 3, pages 2 + 1 + empty + let (rows, total) = filter_and_paginate( + models.clone(), + ¶ms(Some("l1"), Some("prod"), Some(0), Some(2)), + ); + assert_eq!((rows.len(), total), (2, 3)); + let (rows, total) = filter_and_paginate( + models.clone(), + ¶ms(Some("l1"), Some("prod"), Some(1), Some(2)), + ); + assert_eq!((rows.len(), total), (1, 3)); + assert_eq!(rows[0].id, "e"); + let (rows, total) = + filter_and_paginate(models, ¶ms(Some("l1"), Some("prod"), Some(2), Some(2))); + assert_eq!((rows.len(), total), (0, 3)); + } + #[test] fn test_try_from_model() { let check = Synthetic::try_from(make_model()).unwrap(); diff --git a/src/jobs/Cargo.toml b/src/jobs/Cargo.toml index 4de18ac87a..6b8931ef88 100644 --- a/src/jobs/Cargo.toml +++ b/src/jobs/Cargo.toml @@ -40,7 +40,8 @@ vectorscan = [ "search_service/vectorscan", "stream/vectorscan", ] -profiling = ["openobserve-core/profiling"] +cpu-profiling = ["openobserve-core/cpu-profiling"] +profiling = ["cpu-profiling", "openobserve-core/profiling"] [dependencies] audit = { workspace = true, optional = true } diff --git a/src/jobs/src/job/cloud.rs b/src/jobs/src/job/cloud.rs index 9d747302bd..27ab95de42 100644 --- a/src/jobs/src/job/cloud.rs +++ b/src/jobs/src/job/cloud.rs @@ -34,6 +34,7 @@ use db::{ use hashbrown::HashSet; use infra::table::org_users::get_admin; use o2_enterprise::enterprise::cloud::billings::{MeteringDetail, MeteringProvider}; +use openobserve_core::org_domain_ownership; use stream::get_streams; use crate::{ @@ -52,6 +53,8 @@ const TRIAL_QUOTA_FLUSH_INTERVAL: u64 = 10; /// Interval for external contract expiry checks (1 hour). const EXTERNAL_CONTRACT_CHECK_INTERVAL: u64 = 3600; +const ORG_DOMAIN_OWNERSHIP_CHECK_INTERVAL: u64 = 300; + /// (days-remaining threshold, stage stored after the warning fires). /// Walked in descending-urgency order so we send at most one warning per tick. const EXPIRY_WARNING_STAGES: &[( @@ -99,6 +102,7 @@ pub fn start() { tokio::spawn(async move { run_org_expiry_daily().await }); tokio::spawn(async move { run_ai_quota_check().await }); tokio::spawn(async move { run_external_contract_expiry_check().await }); + tokio::spawn(async move { run_org_domain_ownership_validation().await }); tokio::spawn(async move { run_budget_warning_emails().await }); } @@ -111,6 +115,55 @@ pub fn start_trial_quota_jobs() { }); } +async fn run_org_domain_ownership_validation() { + let mut interval = tokio::time::interval(tokio::time::Duration::from_secs( + ORG_DOMAIN_OWNERSHIP_CHECK_INTERVAL, + )); + loop { + interval.tick().await; + let records = match infra::table::org_domain_ownership::list_all().await { + Ok(v) => v, + Err(e) => { + log::error!( + "error listing org domain ownership records for verification, skipping : {e}" + ); + continue; + } + }; + for record in records { + if record.meta_mapped { + continue; + } + let original_state = record.verification_state; + let org_id = record.org_id.clone(); + let domain = record.domain.clone(); + let updated_record = match org_domain_ownership::verify(record).await { + Ok(v) => v, + Err(e) => { + log::error!( + "error verifying org domain ownership record for {org_id} domain {domain} : {e}" + ); + continue; + } + }; + let new_state = updated_record.verification_state; + if new_state != original_state { + log::info!( + "org domain ownership state for {org_id} domain {domain} changed from {original_state} to {new_state}" + ); + if let Err(e) = + org_domain_ownership::update_state(&org_id, &domain, new_state).await + { + log::error!( + "error updating in memory org domain ownership state for org {org_id} domain {domain} : {e}" + ); + continue; + } + } + } + } +} + async fn run_no_ingestion_period() { let mut interval = tokio::time::interval(tokio::time::Duration::from_secs( NO_INGESTION_REPORT_INTERVAL, diff --git a/src/jobs/src/job/mod.rs b/src/jobs/src/job/mod.rs index 6b4d189aa4..60926a9513 100644 --- a/src/jobs/src/job/mod.rs +++ b/src/jobs/src/job/mod.rs @@ -596,7 +596,11 @@ pub async fn init() -> Result<(), anyhow::Error> { .await?; #[cfg(feature = "cloud")] - tokio::task::spawn(o2_enterprise::enterprise::cloud::billings::watch()); + { + openobserve_core::org_domain_ownership::init().await; + tokio::task::spawn(o2_enterprise::enterprise::cloud::billings::watch()); + tokio::task::spawn(openobserve_core::org_domain_ownership::watch()); + } // check version db::metas::version::set() @@ -690,7 +694,7 @@ pub async fn init() -> Result<(), anyhow::Error> { return Ok(()); } - #[cfg(feature = "profiling")] + #[cfg(feature = "cpu-profiling")] tokio::task::spawn(openobserve_core::self_profiles::run()); // telemetry run diff --git a/src/jobs/src/job/slack_hooks.rs b/src/jobs/src/job/slack_hooks.rs index e541a2e6f1..57dd54e571 100644 --- a/src/jobs/src/job/slack_hooks.rs +++ b/src/jobs/src/job/slack_hooks.rs @@ -71,7 +71,7 @@ async fn mapped_domains(org_id: &str) -> anyhow::Result> { if db::org_status::is_blocked(org_id) { return Ok(vec![]); } - let settings = db::organization::get_org_setting(config::META_ORG_ID).await?; + let settings = db::organization::get_org_setting(org_id).await?; let mappings = settings .domain_org_mappings .iter() diff --git a/src/synthetics/src/callback_endpoint.rs b/src/synthetics/src/callback_endpoint.rs new file mode 100644 index 0000000000..ae73ad9624 --- /dev/null +++ b/src/synthetics/src/callback_endpoint.rs @@ -0,0 +1,260 @@ +// Copyright 2026 OpenObserve Inc. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +//! The callback endpoint probes are told to call back on +//! (`config::meta::synthetics::api_endpoint`), as it can safely appear in a +//! log line or a dead-letter message, plus a startup reachability warning for +//! Lambda venues (o2-enterprise#3165). + +use std::{ + net::{IpAddr, Ipv4Addr, Ipv6Addr}, + time::Duration, +}; + +/// Placeholder for an endpoint that does not parse, so no raw value (which may +/// carry credentials) is ever echoed. +pub const UNPARSEABLE_ENDPOINT: &str = ""; + +/// `scheme://host[:port]/path` of `raw`: no userinfo, no query, no fragment. +/// `None` when `raw` is not an absolute URL with a host. +pub fn sanitize_callback_url(raw: &str) -> Option { + let url = reqwest::Url::parse(raw.trim()).ok()?; + let host = url.host_str()?; + let port = url.port().map(|p| format!(":{p}")).unwrap_or_default(); + let path = url.path().trim_end_matches('/'); + Some(format!("{}://{host}{port}{path}", url.scheme())) +} + +/// The endpoint probes are currently told to call back on, sanitized. +pub fn current_callback_endpoint() -> String { + sanitize_callback_url(&config::meta::synthetics::api_endpoint()) + .unwrap_or_else(|| UNPARSEABLE_ENDPOINT.to_string()) +} + +/// Address classes a Lambda function outside the deployment's network usually +/// cannot reach. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum AddrClass { + Public, + Loopback, + Unspecified, + LinkLocal, + Private, +} + +pub fn classify_ip(ip: IpAddr) -> AddrClass { + match ip { + IpAddr::V4(v4) => classify_v4(v4), + IpAddr::V6(v6) => { + if let Some(v4) = v6.to_ipv4_mapped() { + return classify_v4(v4); + } + classify_v6(v6) + } + } +} + +fn classify_v4(ip: Ipv4Addr) -> AddrClass { + let o = ip.octets(); + if ip.is_loopback() { + AddrClass::Loopback + } else if ip.is_unspecified() { + AddrClass::Unspecified + } else if ip.is_link_local() { + AddrClass::LinkLocal + } else if ip.is_private() || (o[0] == 100 && (o[1] & 0xc0) == 64) { + // RFC 1918 plus shared address space (100.64.0.0/10, carrier NAT). + AddrClass::Private + } else { + AddrClass::Public + } +} + +fn classify_v6(ip: Ipv6Addr) -> AddrClass { + let first = ip.segments()[0]; + if ip.is_loopback() { + AddrClass::Loopback + } else if ip.is_unspecified() { + AddrClass::Unspecified + } else if (first & 0xffc0) == 0xfe80 { + AddrClass::LinkLocal + } else if (first & 0xfe00) == 0xfc00 { + // Unique local addresses, fc00::/7. + AddrClass::Private + } else { + AddrClass::Public + } +} + +/// The host part of a parsed URL as an IP literal, IPv6 brackets stripped. +fn host_ip(host: &str) -> Option { + host.trim_start_matches('[') + .trim_end_matches(']') + .parse() + .ok() +} + +const DNS_TIMEOUT: Duration = Duration::from_secs(2); + +/// Log-only check run when a Lambda venue is enabled: a Lambda function can +/// call back only on an endpoint reachable from its own network. Never rejects +/// anything, because a VPC-connected Lambda can legitimately reach private +/// addresses. Logs only the sanitized endpoint. +pub async fn warn_if_lambda_callback_may_be_unreachable() { + let raw = config::meta::synthetics::api_endpoint(); + let Some(shown) = sanitize_callback_url(&raw) else { + tracing::warn!( + "[synthetics] Lambda venue enabled but the callback endpoint \ + (ZO_SYNTHETICS_API_ENDPOINT, else ZO_WEB_URL) is empty or not an absolute URL; \ + Lambda probes may be unable to report results" + ); + return; + }; + let Ok(url) = reqwest::Url::parse(raw.trim()) else { + return; + }; + let Some(host) = url.host_str() else { return }; + let advice = "a Lambda probe can report only if this endpoint is reachable from the \ + Lambda function's network (a VPC-connected function can reach private \ + addresses; otherwise use a public endpoint)"; + + if let Some(ip) = host_ip(host) { + let class = classify_ip(ip); + if class != AddrClass::Public { + tracing::warn!( + endpoint = %shown, + address_class = ?class, + "[synthetics] callback endpoint is a {class:?} address; {advice}" + ); + } + return; + } + if host.eq_ignore_ascii_case("localhost") || host.to_ascii_lowercase().ends_with(".localhost") { + tracing::warn!( + endpoint = %shown, + "[synthetics] callback endpoint is localhost; {advice}" + ); + return; + } + + let port = url.port_or_known_default().unwrap_or(443); + let lookup = tokio::time::timeout( + DNS_TIMEOUT, + tokio::net::lookup_host((host.to_string(), port)), + ) + .await; + let addrs: Vec = match lookup { + Ok(Ok(it)) => it.map(|sa| sa.ip()).collect(), + Ok(Err(e)) => { + tracing::warn!( + endpoint = %shown, + "[synthetics] could not resolve the callback endpoint host from this node ({e}); \ + Lambda probes may be unable to report results if it does not resolve for them \ + either" + ); + return; + } + Err(_) => { + tracing::warn!( + endpoint = %shown, + "[synthetics] resolving the callback endpoint host timed out after {}s; \ + Lambda probes may be unable to report results if it does not resolve for them \ + either", + DNS_TIMEOUT.as_secs() + ); + return; + } + }; + let non_public = addrs + .iter() + .filter(|ip| classify_ip(**ip) != AddrClass::Public) + .count(); + if non_public == 0 { + return; + } + let kind = if non_public == addrs.len() { + "only private, loopback or link-local addresses" + } else { + "a mix of public and private addresses" + }; + tracing::warn!( + endpoint = %shown, + "[synthetics] callback endpoint host resolves to {kind}; {advice}" + ); +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn sanitize_strips_userinfo_query_and_fragment() { + assert_eq!( + sanitize_callback_url("https://user:secret@o2.example.com:8443/api/?token=abc#x"), + Some("https://o2.example.com:8443/api".to_string()) + ); + assert_eq!( + sanitize_callback_url("http://o2.example.com"), + Some("http://o2.example.com".to_string()) + ); + } + + #[test] + fn sanitize_keeps_ipv6_brackets() { + assert_eq!( + sanitize_callback_url("http://[fd00::1]:5080/base?sig=1"), + Some("http://[fd00::1]:5080/base".to_string()) + ); + } + + #[test] + fn sanitize_rejects_bad_urls() { + assert_eq!(sanitize_callback_url(""), None); + assert_eq!(sanitize_callback_url("not a url"), None); + assert_eq!(sanitize_callback_url("o2.example.com/api"), None); + assert_eq!(sanitize_callback_url("mailto:a@b.c"), None); + } + + #[test] + fn classify_ipv4() { + let c = |s: &str| classify_ip(s.parse().unwrap()); + assert_eq!(c("127.0.0.1"), AddrClass::Loopback); + assert_eq!(c("0.0.0.0"), AddrClass::Unspecified); + assert_eq!(c("169.254.169.254"), AddrClass::LinkLocal); + assert_eq!(c("10.1.2.3"), AddrClass::Private); + assert_eq!(c("172.16.0.1"), AddrClass::Private); + assert_eq!(c("192.168.1.1"), AddrClass::Private); + assert_eq!(c("100.64.0.1"), AddrClass::Private); + assert_eq!(c("8.8.8.8"), AddrClass::Public); + assert_eq!(c("172.32.0.1"), AddrClass::Public); + } + + #[test] + fn classify_ipv6() { + let c = |s: &str| classify_ip(s.parse().unwrap()); + assert_eq!(c("::1"), AddrClass::Loopback); + assert_eq!(c("::"), AddrClass::Unspecified); + assert_eq!(c("fe80::1"), AddrClass::LinkLocal); + assert_eq!(c("fd12:3456::1"), AddrClass::Private); + assert_eq!(c("::ffff:10.0.0.1"), AddrClass::Private); + assert_eq!(c("2606:4700::1111"), AddrClass::Public); + } + + #[test] + fn host_ip_parses_bracketed_ipv6() { + assert_eq!(host_ip("[::1]"), Some("::1".parse().unwrap())); + assert_eq!(host_ip("o2.example.com"), None); + } +} diff --git a/src/synthetics/src/dispatcher.rs b/src/synthetics/src/dispatcher.rs index 0f50f057fc..d4daa92592 100644 --- a/src/synthetics/src/dispatcher.rs +++ b/src/synthetics/src/dispatcher.rs @@ -35,7 +35,22 @@ use infra::{ }; const TICK: Duration = Duration::from_secs(2); -const LEASE_BATCH_SIZE: i64 = 10; + +/// Jobs per pool per lease call (`ZO_SYNTHETICS_LEASE_BATCH_SIZE`), read per tick. +fn lease_batch_size() -> i64 { + config::get_config().synthetics.lease_batch_size.max(1) +} + +/// Most jobs leased across all pools in one tick (`ZO_SYNTHETICS_MAX_INFLIGHT_PER_TICK`). +fn max_inflight_per_tick() -> i64 { + config::get_config().synthetics.max_inflight_per_tick.max(1) +} + +/// Size of the next lease call: a full batch, trimmed so the tick never leases +/// past its cap. Zero means the tick is out of budget. +fn next_batch_limit(batch: i64, leased_this_tick: i64, cap: i64) -> i64 { + batch.min(cap - leased_this_tick).max(0) +} /// How long a leased job may run before the reaper requeues it /// (`reaper::requeue_expired`). /// @@ -95,39 +110,88 @@ static INSTANCE_ID: LazyLock = LazyLock::new(|| { ) }); -pub async fn run() { +static POOL_ROTATION: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0); + +pub async fn run(stop: crate::StopToken) { let instance_id = INSTANCE_ID.as_str(); tracing::info!(instance_id = %instance_id, "[synthetics dispatcher] started"); + // o2-enterprise#3165: a Lambda probe can only report to a callback endpoint + // its own network reaches. Log-only; never blocks the dispatcher. + { + let cfg = config::get_config(); + if !cfg.synthetics.lambda_browser.is_empty() || !cfg.synthetics.lambda_net.is_empty() { + crate::callback_endpoint::warn_if_lambda_callback_may_be_unreachable().await; + } + } + loop { - tokio::time::sleep(TICK).await; + if stop.sleep(TICK).await { + tracing::info!("[synthetics dispatcher] stopped"); + return; + } let db = get_orm_client_rw().await; let now_us = config::utils::time::now_micros(); - for pool in dispatcher_pools().await { - let rows = match synthetics_jobs::lease_batch( - db, - &pool, - instance_id, - LEASE_BATCH_SIZE, - now_us, - lease_secs(), - None, // public pools are already type-segmented (net-* vs aws-browser) - ) - .await - { - Ok(r) => r, - Err(e) => { - tracing::error!(pool = %pool, "[synthetics dispatcher] lease_batch: {e}"); - continue; + // Round-robin across pools, one batch each per round. A pool whose batch + // came back full may have more due, so it stays in for another round; + // the per-tick cap bounds the total, and no pool gets a second batch + // before every other active pool has had its turn. + let batch = lease_batch_size(); + let cap = max_inflight_per_tick(); + let mut leased_this_tick: i64 = 0; + let mut active = dispatcher_pools().await; + // Rotate the starting pool every tick: with the cap reached, a fixed + // order would let the first pools take the whole budget every tick. + if !active.is_empty() { + let start = + POOL_ROTATION.fetch_add(1, std::sync::atomic::Ordering::Relaxed) % active.len(); + active.rotate_left(start); + } + while !active.is_empty() { + let mut still_active = Vec::with_capacity(active.len()); + for pool in active { + // Ownership lost mid-tick: lease nothing more. + if stop.is_cancelled() { + return; } - }; + let limit = next_batch_limit(batch, leased_this_tick, cap); + if limit == 0 { + break; + } + let rows = match synthetics_jobs::lease_batch( + db, + &pool, + instance_id, + limit, + now_us, + lease_secs(), + None, // public pools are already type-segmented (net-* vs aws-browser) + ) + .await + { + Ok(r) => r, + Err(e) => { + tracing::error!(pool = %pool, "[synthetics dispatcher] lease_batch: {e}"); + continue; + } + }; - for row in rows { - tokio::spawn(invoke_probe(row)); + let full = rows.len() as i64 >= limit; + leased_this_tick += rows.len() as i64; + for row in rows { + tokio::spawn(invoke_probe(row, stop.clone())); + } + if full { + still_active.push(pool); + } } + if leased_this_tick >= cap { + break; + } + active = still_active; } } } @@ -304,7 +368,13 @@ async fn mark_failure(row: &synthetics_jobs::LeasedRow, reason: Option<&str>) -> // nothing, while the first failure after a pass sends one message. Without // it the other regions' LIST would keep showing the last status they // themselves ran, or "Unknown". - match synthetics_checks::update_last_check_status(db, &row.synthetics_id, 4).await { + match synthetics_checks::update_last_check_status( + db, + &row.synthetics_id, + config::meta::status_pages::CHECK_STATUS_ERROR, + ) + .await + { Ok(true) => { #[cfg(feature = "enterprise")] if o2_enterprise::enterprise::common::config::get_config() @@ -449,7 +519,13 @@ async fn report_to_stream( } /// Invokes the browser probe Lambda for a leased check row. -async fn invoke_probe(row: synthetics_jobs::LeasedRow) { +async fn invoke_probe(row: synthetics_jobs::LeasedRow, stop: crate::StopToken) { + // Leased just before ownership moved: do not start the probe here. The + // lease expires and the new owner's reaper re-queues the row. + if stop.is_cancelled() { + tracing::info!(job_id = %row.id, "[synthetics dispatcher] ownership lost — not invoking"); + return; + } // Derive AWS region from the job's location field. let region = match aws_region_from_location(&row.location) { Some(r) => r.to_owned(), @@ -691,3 +767,16 @@ async fn handle_dispatch_failure( } } } + +#[cfg(test)] +mod lease_batch_tests { + use super::next_batch_limit; + + #[test] + fn batch_limit_is_trimmed_to_the_tick_cap() { + assert_eq!(next_batch_limit(10, 0, 100), 10); + assert_eq!(next_batch_limit(10, 95, 100), 5); + assert_eq!(next_batch_limit(10, 100, 100), 0); + assert_eq!(next_batch_limit(10, 120, 100), 0); + } +} diff --git a/src/synthetics/src/lib.rs b/src/synthetics/src/lib.rs index 11be7d03ba..d71b43f252 100644 --- a/src/synthetics/src/lib.rs +++ b/src/synthetics/src/lib.rs @@ -60,6 +60,7 @@ pub(crate) static CONFIG_SWAP_LOCK: tokio::sync::Mutex<()> = tokio::sync::Mutex: pub const BUILT_WITH_CLOUD: bool = cfg!(feature = "cloud"); pub mod alerting; +pub mod callback_endpoint; pub mod dispatcher; pub mod job_api; pub mod pool; @@ -157,52 +158,200 @@ pub async fn init(step_pool: Option) { // super-cluster KV client is built and no NATS read is issued on this path // (spec §14 — super cluster off must be byte-identical). #[cfg(not(feature = "enterprise"))] - spawn_workers(); + drop(spawn_workers(StopToken::never())); #[cfg(feature = "enterprise")] if !o2_enterprise::enterprise::common::config::get_config() .super_cluster .enabled { - spawn_workers(); + drop(spawn_workers(StopToken::never())); return; } #[cfg(feature = "enterprise")] - tokio::spawn(async move { - let mut announced = false; - while !holds_job_cluster_claim().await { - if !announced { - // Once, not per poll: a region that legitimately does not hold - // the claim is the steady state for the whole life of the - // process, and an every-10s line would bury it. - tracing::info!( - "[synthetics] another cluster holds the job-cluster claim (or none is \ - registered yet) — workers not started, re-checking every {}s", - JOB_CLUSTER_POLL.as_secs() - ); - announced = true; - } - tokio::time::sleep(JOB_CLUSTER_POLL).await; - } - tracing::info!("[synthetics] this cluster holds the job-cluster claim — starting workers"); - spawn_workers(); - }); + tokio::spawn(ownership_supervisor()); } -fn spawn_workers() { - tokio::spawn(scheduler::run()); - tokio::spawn(dispatcher::run()); - tokio::spawn(reaper::run()); - // POC scope: single cluster. Multi-region needs this split — the - // incident-engine half claim-gated, the snapshot half per region. - // Status pages ships with synthetics — no separate toggle. - tokio::spawn(status_pages::run()); - tokio::spawn(status_pages::run_domain_verifier()); - // Its own task, not a step on the reaper's tick: a pass is up to a thousand - // rows of outbound HTTP, and the reaper's lease bookkeeping cannot wait - // behind it. Its own kill switch too, checked per pass rather than here. - tokio::spawn(reaper::orphan::run()); +/// Cooperative stop signal handed to every claim-gated worker. +/// +/// A minimal `CancellationToken` on `tokio::sync::watch` (the crate has no +/// `tokio-util` dependency, and adding one is not worth a manifest change). +/// A dropped sender reads as cancelled, so a supervisor that goes away fails +/// closed rather than leaving workers running unsupervised. +#[derive(Clone)] +pub struct StopToken { + rx: tokio::sync::watch::Receiver, + // Only set by [`StopToken::never`]: keeps the channel open forever. + _keep: Option>>, +} + +impl StopToken { + /// A token plus the sender that cancels it. + pub fn new() -> (tokio::sync::watch::Sender, Self) { + let (tx, rx) = tokio::sync::watch::channel(false); + (tx, Self { rx, _keep: None }) + } + + /// A token that is never cancelled: the single-cluster path, where there + /// is no ownership to lose. + pub fn never() -> Self { + let (tx, rx) = tokio::sync::watch::channel(false); + Self { + rx, + _keep: Some(std::sync::Arc::new(tx)), + } + } + + pub fn is_cancelled(&self) -> bool { + *self.rx.borrow() || (self._keep.is_none() && self.rx.has_changed().is_err()) + } + + pub async fn cancelled(&self) { + let mut rx = self.rx.clone(); + // Err = sender dropped, which also means stop. + let _ = rx.wait_for(|v| *v).await; + } + + /// Sleeps for `d`; returns `true` when cancelled first (the caller exits + /// its loop at this tick boundary, never mid-pass). + pub async fn sleep(&self, d: std::time::Duration) -> bool { + tokio::select! { + _ = self.cancelled() => true, + _ = tokio::time::sleep(d) => self.is_cancelled(), + } + } +} + +/// Result of one read of the job-cluster claim. A failed read is `Unknown`, +/// never "held": the supervisor fails closed. +#[cfg_attr(not(feature = "enterprise"), allow(dead_code))] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum ClaimRead { + Held, + NotHeld, + Unknown, +} + +#[cfg_attr(not(feature = "enterprise"), allow(dead_code))] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum SupervisorAction { + Start, + Stop, + Hold, +} + +/// The supervisor's whole decision: a pure function of whether a worker +/// generation is running and what the claim read returned. Workers run exactly +/// while the claim is provably held here; `NotHeld` and `Unknown` both stop a +/// running generation (same fail-closed rule as the anomaly supervisor in +/// `jobs::job::mod.rs`). +#[cfg_attr(not(feature = "enterprise"), allow(dead_code))] +fn supervisor_action(running: bool, claim: ClaimRead) -> SupervisorAction { + match (running, claim) { + (false, ClaimRead::Held) => SupervisorAction::Start, + (true, ClaimRead::NotHeld | ClaimRead::Unknown) => SupervisorAction::Stop, + _ => SupervisorAction::Hold, + } +} + +/// Start/Stop supervisor for the claim-gated workers (o2-enterprise#3190). +/// +/// Replaces the old start-only latch, which spawned the workers once on +/// gaining the claim and never stopped them when the claim moved away, so two +/// regions could schedule, dispatch and reap the same checks. Each generation +/// gets its own [`StopToken`]; Stop cancels it and JOINS every worker before +/// the next Start can happen, so there are never two generations at once. +#[cfg(feature = "enterprise")] +async fn ownership_supervisor() { + let mut generation: u64 = 0; + let mut running: Option<( + tokio::sync::watch::Sender, + Vec>, + )> = None; + let mut last_claim: Option = None; + loop { + let claim = read_job_cluster_claim().await; + if last_claim != Some(claim) && claim == ClaimRead::NotHeld && running.is_none() { + tracing::info!( + "[synthetics] another cluster holds the job-cluster claim (or none is registered \ + yet) — workers not running here, re-checking every {}s", + JOB_CLUSTER_POLL.as_secs() + ); + } + last_claim = Some(claim); + match supervisor_action(running.is_some(), claim) { + SupervisorAction::Start => { + generation += 1; + tracing::info!( + generation, + "[synthetics] ownership acquired — starting workers" + ); + let (tx, token) = StopToken::new(); + running = Some((tx, spawn_workers(token))); + } + SupervisorAction::Stop => { + if claim == ClaimRead::Unknown { + tracing::warn!( + generation, + "[synthetics] ownership unknown (claim read failed) — stopping workers" + ); + } else { + tracing::info!(generation, "[synthetics] ownership lost — stopping workers"); + } + if let Some((tx, handles)) = running.take() { + let _ = tx.send(true); + for h in handles { + if let Err(e) = h.await { + tracing::error!(generation, "[synthetics] worker join failed: {e}"); + } + } + } + tracing::info!(generation, "[synthetics] workers stopped"); + } + SupervisorAction::Hold => {} + } + tokio::time::sleep(JOB_CLUSTER_POLL).await; + } +} + +#[cfg(feature = "enterprise")] +async fn read_job_cluster_claim() -> ClaimRead { + match holds_job_cluster_claim().await { + Some(true) => ClaimRead::Held, + Some(false) => ClaimRead::NotHeld, + None => ClaimRead::Unknown, + } +} + +/// Spawns one generation of workers. Every worker takes `stop` and leaves its +/// loop at the next tick boundary once it is cancelled. +/// +/// All of them are claim-gated, as they were under the old latch: the status +/// page task is still one loop (incident engine and snapshot together — the +/// per-region snapshot split is future work), and the domain verifier writes +/// shared verification state, so both stay owner-only. +fn spawn_workers(stop: StopToken) -> Vec> { + let verifier_stop = stop.clone(); + vec![ + tokio::spawn(scheduler::run(stop.clone())), + tokio::spawn(dispatcher::run(stop.clone())), + tokio::spawn(reaper::run(stop.clone())), + // Status pages ships with synthetics — no separate toggle. + tokio::spawn(status_pages::run(stop.clone())), + // The verifier lives in the enterprise crate and takes no token, so it + // is dropped at its next await point on cancel. + tokio::spawn(async move { + tokio::select! { + _ = verifier_stop.cancelled() => {} + _ = status_pages::run_domain_verifier() => {} + } + }), + // Its own task, not a step on the reaper's tick: a pass is up to a + // thousand rows of outbound HTTP, and the reaper's lease bookkeeping + // cannot wait behind it. Its own kill switch too, checked per pass. + tokio::spawn(reaper::orphan::run(stop)), + ] } /// One evaluation of [`should_start`] against the super-cluster KV. @@ -217,7 +366,7 @@ fn spawn_workers() { /// increments `SYNTHETICS_ORPHAN_SCANS_TOTAL` anywhere, which is exactly the /// dead-region signal `reaper::orphan` documents. #[cfg(feature = "enterprise")] -async fn holds_job_cluster_claim() -> bool { +async fn holds_job_cluster_claim() -> Option { use o2_enterprise::enterprise::super_cluster::kv; let local = config::get_cluster_name(); @@ -225,27 +374,27 @@ async fn holds_job_cluster_claim() -> bool { Ok(name) => name, Err(e) => { tracing::error!("[synthetics] could not read the job cluster, not starting yet: {e}"); - return false; + return None; } }; // Both settled without the cluster list, which is the more expensive read. // Must agree with `should_start`, which is where the rule is tested. if claim.is_empty() { - return false; + return Some(false); } if claim == local { - return true; + return Some(true); } let live = match kv::cluster::list_by_role_group(None).await { Ok(clusters) => clusters.into_iter().map(|c| c.name).collect::>(), Err(e) => { tracing::error!("[synthetics] could not list clusters, not starting yet: {e}"); - return false; + return None; } }; - should_start(true, &claim, &local, &live) + Some(should_start(true, &claim, &local, &live)) } /// Whether this cluster may start the synthetics workers. @@ -344,4 +493,62 @@ mod job_cluster_gate_tests { } #[cfg(test)] -mod tests {} +mod tests { + use super::{ClaimRead::*, StopToken, SupervisorAction::*, supervisor_action}; + + #[test] + fn starts_only_on_a_held_claim() { + assert_eq!(supervisor_action(false, Held), Start); + assert_eq!(supervisor_action(false, NotHeld), Hold); + assert_eq!(supervisor_action(false, Unknown), Hold); + } + + #[test] + fn a_running_generation_stops_on_loss_or_read_error() { + assert_eq!(supervisor_action(true, NotHeld), Stop); + assert_eq!(supervisor_action(true, Unknown), Stop); + assert_eq!(supervisor_action(true, Held), Hold); + } + + /// Flapping claim: replay the reads through the decision function and + /// check the generation never overlaps and every flip is acted on. + #[test] + fn flapping_claim_alternates_start_and_stop_without_overlap() { + let reads = [Held, Held, NotHeld, Held, Unknown, Unknown, Held, NotHeld]; + let mut running = false; + let mut starts = 0; + let mut stops = 0; + for r in reads { + match supervisor_action(running, r) { + Start => { + assert!(!running, "never two generations"); + running = true; + starts += 1; + } + Stop => { + assert!(running); + running = false; + stops += 1; + } + Hold => {} + } + } + assert_eq!((starts, stops), (3, 3)); + assert!(!running); + } + + #[tokio::test] + async fn stop_token_cancels_sleep_and_dropped_sender_fails_closed() { + let (tx, token) = StopToken::new(); + assert!(!token.is_cancelled()); + tx.send(true).unwrap(); + assert!(token.sleep(std::time::Duration::from_secs(60)).await); + + let (tx, token) = StopToken::new(); + drop(tx); + assert!(token.sleep(std::time::Duration::from_secs(60)).await); + + let never = StopToken::never(); + assert!(!never.sleep(std::time::Duration::from_millis(1)).await); + } +} diff --git a/src/synthetics/src/reaper/mod.rs b/src/synthetics/src/reaper/mod.rs index 0171cc8b46..561266c432 100644 --- a/src/synthetics/src/reaper/mod.rs +++ b/src/synthetics/src/reaper/mod.rs @@ -48,11 +48,14 @@ use infra::{ use crate::{MAX_DISPATCH_ATTEMPTS as MAX_ATTEMPTS, alerting::ERROR_SOURCE_DISPATCH}; const TICK: Duration = Duration::from_secs(30); -pub async fn run() { +pub async fn run(stop: crate::StopToken) { tracing::info!("[synthetics reaper] started"); loop { - tokio::time::sleep(TICK).await; + if stop.sleep(TICK).await { + tracing::info!("[synthetics reaper] stopped"); + return; + } let db = get_orm_client_rw().await; @@ -181,7 +184,13 @@ async fn handle_dead_letter( // check that is already in Error dead-letters silently, which is the common // case when a private agent has gone away and every run of every check it // served is expiring. - match synthetics_checks::update_last_check_status(db, &row.synthetics_id, 4).await { + match synthetics_checks::update_last_check_status( + db, + &row.synthetics_id, + config::meta::status_pages::CHECK_STATUS_ERROR, + ) + .await + { Ok(true) => { #[cfg(feature = "enterprise")] if o2_enterprise::enterprise::common::config::get_config() @@ -218,21 +227,16 @@ async fn handle_dead_letter( // points at a dead agent or a location nothing polls; the other two point at // a probe that took the job and stopped talking. They need different // responses, so they should not read identically in the record. - let error_msg = match row.reason { - synthetics_jobs::DeadLetterReason::AttemptsExhausted => format!( - "probe did not respond after {} dispatch attempts", - row.dispatch_attempts - ), - synthetics_jobs::DeadLetterReason::Expired => format!( - "probe held the check but did not respond before its schedule window closed \ - (after {} dispatch attempt(s)); the next scheduled run will retry", - row.dispatch_attempts - ), - synthetics_jobs::DeadLetterReason::NeverDispatched => { - "no probe claimed the check for this location before its schedule window closed" - .to_string() - } - }; + // Phase, attempt count and the callback endpoint (o2-enterprise#3165): a + // timeout says the probe did not report, not why, so the message names + // where it was told to report and stops short of a cause. + // + // The endpoint is the one resolved NOW, not at dispatch: the job row does + // not store the dispatch-time endpoint. Capturing it needs a column + // (deferred); until then a config change between dispatch and reaping + // would show the new value. + let callback = crate::callback_endpoint::current_callback_endpoint(); + let error_msg = dead_letter_message(row.reason, row.dispatch_attempts, &callback); // Above the token lookup, which returns early: the triggers half needs no token. usage_reporting::publish_triggers_usage(dead_letter_trigger(row, now_us, &error_msg)); @@ -264,6 +268,32 @@ async fn handle_dead_letter( write_results_stream(row, &ingest_token, &api_endpoint, now_us, &error_msg).await; } +/// The dead-letter text for one reason. `callback` must already be sanitized. +fn dead_letter_message( + reason: synthetics_jobs::DeadLetterReason, + dispatch_attempts: i32, + callback: &str, +) -> String { + match reason { + synthetics_jobs::DeadLetterReason::AttemptsExhausted => format!( + "probe did not respond after {dispatch_attempts} dispatch attempts \ + (phase: claimed, no result before the lease ran out); the probe may be unable to reach \ + the callback endpoint {callback}" + ), + synthetics_jobs::DeadLetterReason::Expired => format!( + "probe held the check but did not respond before its schedule window closed \ + (phase: claimed, lease expired without a result; after {dispatch_attempts} dispatch attempt(s)); \ + the probe may be unable to reach the callback endpoint {callback}; \ + the next scheduled run will retry" + ), + synthetics_jobs::DeadLetterReason::NeverDispatched => format!( + "no probe claimed the check for this location before its schedule window closed \ + (phase: never claimed; {dispatch_attempts} dispatch attempt(s)); a probe may be \ + unable to reach the callback endpoint {callback}" + ), + } +} + async fn write_results_stream( row: &synthetics_jobs::DeadLetteredRow, ingest_token: &str, @@ -393,4 +423,21 @@ mod tests { "an alert rule matches the SERIALIZED value, and this row writes `failed` today", ); } + + #[test] + fn dead_letter_messages_name_phase_attempts_and_endpoint() { + use infra::table::synthetics_jobs::DeadLetterReason as R; + let ep = "https://o2.example.com/api"; + for (reason, phase) in [ + (R::AttemptsExhausted, "phase: claimed, no result"), + (R::Expired, "phase: claimed, lease expired"), + (R::NeverDispatched, "phase: never claimed"), + ] { + let msg = super::dead_letter_message(reason, 3, ep); + assert!(msg.contains(phase), "{msg}"); + assert!(msg.contains("3 dispatch attempt"), "{msg}"); + assert!(msg.contains(ep), "{msg}"); + assert!(msg.contains("may be unable to reach"), "{msg}"); + } + } } diff --git a/src/synthetics/src/reaper/orphan.rs b/src/synthetics/src/reaper/orphan.rs index 1b77609475..80a886f474 100644 --- a/src/synthetics/src/reaper/orphan.rs +++ b/src/synthetics/src/reaper/orphan.rs @@ -296,7 +296,7 @@ async fn is_scan_leader() -> bool { /// slow scan stalls run accounting, and runs that never complete never notify. /// A separate loop cannot do that, and it is free to run at a cadence that /// suits the threshold instead of the lease timers. -pub async fn run() { +pub async fn run(stop: crate::StopToken) { tracing::info!("[synthetics orphan] started"); if !orphan_detection_enabled() { tracing::info!( @@ -310,7 +310,10 @@ pub async fn run() { let mut last_reported: HashMap = HashMap::new(); loop { - tokio::time::sleep(ORPHAN_TICK).await; + if stop.sleep(ORPHAN_TICK).await { + tracing::info!("[synthetics orphan] stopped"); + return; + } // Read per pass, not captured here: the operator turning this off is // reacting to something happening now. diff --git a/src/synthetics/src/scheduler.rs b/src/synthetics/src/scheduler.rs index 0f2fd36dff..3551138882 100644 --- a/src/synthetics/src/scheduler.rs +++ b/src/synthetics/src/scheduler.rs @@ -279,11 +279,14 @@ pub fn trial_gate_decision( } } -pub async fn run() { +pub async fn run(stop: crate::StopToken) { tracing::info!("[synthetics scheduler] started"); loop { - tokio::time::sleep(TICK).await; + if stop.sleep(TICK).await { + tracing::info!("[synthetics scheduler] stopped"); + return; + } let db = get_orm_client_rw().await; diff --git a/src/synthetics/src/service/composition_lock.rs b/src/synthetics/src/service/composition_lock.rs index f3c0299968..5b28f5d4d4 100644 --- a/src/synthetics/src/service/composition_lock.rs +++ b/src/synthetics/src/service/composition_lock.rs @@ -30,6 +30,27 @@ enum Inner { Distributed(Option), } +/// Longest a save or delete waits for the org-wide composition lock locally. +/// The existing lock-wait config (`ZO_NATS_LOCK_WAIT_TIMEOUT`, 3600s) belongs to +/// NATS and is far too long for a request, so this is a fixed bound. A stuck +/// holder then fails the request with a retryable error instead of hanging it. +const LOCAL_LOCK_WAIT: std::time::Duration = std::time::Duration::from_secs(30); + +async fn lock_local( + key: &str, + wait: std::time::Duration, +) -> Result, infra::errors::Error> { + let holder = infra::local_lock::lock(key).await?; + tokio::time::timeout(wait, holder.lock_owned()) + .await + .map_err(|_| { + infra::errors::Error::Message(format!( + "timed out after {}s waiting for the synthetics composition lock; retry shortly", + wait.as_secs() + )) + }) +} + pub async fn lock(org_id: &str) -> Result { #[cfg(test)] { @@ -41,8 +62,7 @@ pub async fn lock(org_id: &str) -> Result bool { #[derive(Debug, Deserialize)] pub struct UpdateLocationRequest { pub label: String, - #[serde(default = "default_true")] - pub enabled: bool, + /// Omitted keeps the stored value; an explicit `false` disables. + #[serde(default)] + pub enabled: Option, +} + +/// The `enabled` value to store: the request's, else the row's current one. +fn resolve_enabled(requested: Option, stored: bool) -> bool { + requested.unwrap_or(stored) } /// Legacy compat: old UIs sent bare AWS regions ("us-east-1") for public @@ -595,8 +601,9 @@ pub async fn update_location( id: &str, req: UpdateLocationRequest, ) -> anyhow::Result { - location_for_write(org_id, is_root, id).await?; - synthetics_locations::update(id, &req.label, req.enabled).await?; + let existing = location_for_write(org_id, is_root, id).await?; + let enabled = resolve_enabled(req.enabled, existing.enabled); + synthetics_locations::update(id, &req.label, enabled).await?; // The two columns the write touched, not the row: sending a row would carry // `down_notified_at` with it, and that flag is the claim on the right to // send one "location down" notification. @@ -606,9 +613,7 @@ pub async fn update_location( .enabled { o2_enterprise::enterprise::super_cluster::queue::synthetics_location_update( - id, - &req.label, - req.enabled, + id, &req.label, enabled, ) .await?; } @@ -725,6 +730,20 @@ async fn private_location_install( mod tests { use super::*; + #[test] + fn label_only_update_preserves_disabled() { + let req: UpdateLocationRequest = serde_json::from_str(r#"{"label":"new"}"#).unwrap(); + assert_eq!(req.enabled, None); + assert!(!resolve_enabled(req.enabled, false)); + assert!(resolve_enabled(req.enabled, true)); + let req: UpdateLocationRequest = + serde_json::from_str(r#"{"label":"new","enabled":false}"#).unwrap(); + assert!(!resolve_enabled(req.enabled, true)); + let req: UpdateLocationRequest = + serde_json::from_str(r#"{"label":"new","enabled":true}"#).unwrap(); + assert!(resolve_enabled(req.enabled, false)); + } + /// Moved from the enterprise config module with the reader it pins. /// /// Decides whether a location shows Offline and whether the staleness diff --git a/src/synthetics/src/status_pages.rs b/src/synthetics/src/status_pages.rs index b52e60e3a0..6d4877b65b 100644 --- a/src/synthetics/src/status_pages.rs +++ b/src/synthetics/src/status_pages.rs @@ -91,11 +91,20 @@ pub async fn preview_snapshot( .push(u); } + // Same state read as the rebuilder, so a preview of an all-Error component + // shows NoData exactly as the published snapshot does. + let preview_check_ids: Vec = mappings + .iter() + .map(|m| m.synthetics_id.clone()) + .collect::>() + .into_iter() + .collect(); + let (states, _) = table::get_check_states(conn, &preview_check_ids, None).await?; let data = TickData { pages: vec![page.clone()], components, mappings, - states: Vec::new(), + states, snoozed: HashSet::new(), notices: notices.clone(), }; @@ -146,7 +155,7 @@ pub async fn refresh_snapshot(org_id: &str, page_id: &str) -> Result<(), infra:: .await } -pub async fn run() { +pub async fn run(stop: crate::StopToken) { let interval = config::get_config() .synthetics .status_page_rebuild_interval @@ -164,7 +173,10 @@ pub async fn run() { // deletions and refreshes `last_triggered_at` for the dead-probe rule. let mut rt = RebuilderState::default(); loop { - tokio::time::sleep(std::time::Duration::from_secs(interval)).await; + if stop.sleep(std::time::Duration::from_secs(interval)).await { + tracing::info!("[status_pages] rebuilder stopped"); + return; + } if let Err(e) = tick(&mut hashes, &mut down_streaks, &mut rt).await { tracing::error!("[status_pages] rebuild tick failed: {e}"); } @@ -183,6 +195,106 @@ struct RebuilderState { /// Highest `updated_at` consumed so far (the delta watermark). watermark: i64, tick_count: u64, + /// Per check, the `completed_at` of the last run this loop counted. + seen_runs: HashMap, +} + +/// How far back (by `scheduled_ts`) the rebuilder looks for completed runs. +/// Only completions that are new since the previous tick matter, and a probe +/// run completes within its job timeout (minutes), so an hour covers every +/// normal result with room for a delayed tick. A run that completes later than +/// this — a long-expired dead letter — ends as Error, which never advances a +/// streak anyway. +const RUN_COMPLETION_LOOKBACK_MICROS: i64 = 3_600_000_000; + +/// Records one tick's view of a check and says whether it is a new result. +/// +/// Streaks count completed RUNS, not rebuild ticks: the rebuilder ticks far +/// more often than most checks run, so counting ticks confirmed a single DOWN +/// as three failures. The marker is the check's latest +/// `synthetics_runs.completed_at`, so one completed run counts exactly once. +/// (`last_triggered_at` moves at dispatch, before the result exists, so it +/// counted the previous result again when the next run was dispatched.) No +/// completed run in the lookback window means nothing new to count. +fn observe_run( + down_streaks: &mut HashMap, + seen_runs: &mut HashMap, + s: &table::CheckStateRow, +) -> bool { + let fresh = match s.last_completed_at { + Some(at) => seen_runs.insert(s.id.clone(), at) != Some(at), + None => false, + }; + if s.last_check_status == engine::CHECK_STATUS_DOWN { + if fresh { + *down_streaks.entry(s.id.clone()).or_insert(0) += 1; + } + } else { + down_streaks.remove(&s.id); + } + fresh +} + +/// Per-check thresholds from the pages the check is mapped to. A check on +/// several pages uses the STRICTEST settings (max of each value), so one +/// result stream never opens or resolves an org-wide notice earlier than any +/// of its pages allows. +fn thresholds_by_check(data: &TickData) -> HashMap { + let page_of_component: HashMap<&str, &str> = data + .components + .iter() + .map(|c| (c.id.as_str(), c.status_page_id.as_str())) + .collect(); + let page_th: HashMap<&str, engine::Thresholds> = data + .pages + .iter() + .map(|p| (p.id.as_str(), page_thresholds(p))) + .collect(); + let mut out: HashMap = HashMap::new(); + for m in &data.mappings { + let Some(th) = page_of_component + .get(m.component_id.as_str()) + .and_then(|pid| page_th.get(pid)) + else { + continue; + }; + out.entry(m.synthetics_id.clone()) + .and_modify(|cur| *cur = strictest(*cur, *th)) + .or_insert(*th); + } + out +} + +fn page_thresholds(p: &infra::table::entity::status_pages::Model) -> engine::Thresholds { + let d = engine::Thresholds::default(); + engine::Thresholds { + confirm_failures: if p.confirm_failures > 0 { + p.confirm_failures + } else { + d.confirm_failures + }, + confirm_recovery: if p.confirm_recovery > 0 { + p.confirm_recovery + } else { + d.confirm_recovery + }, + confirm_after_micros: p + .confirm_after_secs + .filter(|s| *s > 0) + .and_then(|s| (s as i64).checked_mul(1_000_000)), + } +} + +fn strictest(a: engine::Thresholds, b: engine::Thresholds) -> engine::Thresholds { + engine::Thresholds { + confirm_failures: a.confirm_failures.max(b.confirm_failures), + confirm_recovery: a.confirm_recovery.max(b.confirm_recovery), + // Only time-confirm when every page allows it; then the longest wait. + confirm_after_micros: match (a.confirm_after_micros, b.confirm_after_micros) { + (Some(x), Some(y)) => Some(x.max(y)), + _ => None, + }, + } } async fn tick( @@ -193,14 +305,15 @@ async fn tick( let conn = get_orm_client_rw().await; let now = config::utils::time::now_micros(); let data = load(conn, now, rt).await?; + let mut fresh: HashSet = HashSet::new(); + rt.seen_runs + .retain(|id, _| data.states.iter().any(|s| &s.id == id)); for s in &data.states { - if s.last_check_status == engine::CHECK_STATUS_DOWN { - *down_streaks.entry(s.id.clone()).or_insert(0) += 1; - } else { - down_streaks.remove(&s.id); + if observe_run(down_streaks, &mut rt.seen_runs, s) { + fresh.insert(s.id.clone()); } } - apply_engine_actions(conn, &data, down_streaks, now).await?; + apply_engine_actions(conn, &data, down_streaks, &fresh, now).await?; // Re-read notices only if actions changed them; a second bounded read per // tick is cheaper than tracking per-action deltas in the POC. let notices = table::list_notices_since(conn, now - RESOLVED_LOOKBACK_MICROS).await?; @@ -239,7 +352,13 @@ async fn load( // Delta by default; full sweep on the Nth tick. The sweep replaces the // cache wholesale (so deleted/unmapped checks drop out); the delta merges // only changed rows over the carried cache. - let full_sweep = rt.tick_count.is_multiple_of(FULL_SWEEP_TICKS); + // Always a full read of the mapped checks' state. The delta read keys on + // `updated_at`, which only moves on a status TRANSITION; a check that keeps + // failing (or keeps passing) advances only `last_triggered_at`, so the + // run-based streaks in `observe_run` would see a repeated result only on + // the periodic sweep and confirm many minutes late. The read is three + // scalar columns of the mapped checks only. + let full_sweep = FULL_SWEEP_TICKS > 0; rt.tick_count = rt.tick_count.wrapping_add(1); let since = if full_sweep { None } else { Some(rt.watermark) }; let (changed, max_updated) = table::get_check_states(conn, &check_ids, since).await?; @@ -260,7 +379,16 @@ async fn load( rt.check_state.retain(|id, _| mapped.contains(id.as_str())); } rt.watermark = rt.watermark.max(max_updated); - let states: Vec = rt.check_state.values().cloned().collect(); + let mut states: Vec = rt.check_state.values().cloned().collect(); + let keys: Vec<(String, String)> = states + .iter() + .map(|s| (s.org_id.clone(), s.id.clone())) + .collect(); + let completions = + table::latest_run_completions(conn, &keys, now - RUN_COMPLETION_LOOKBACK_MICROS).await?; + for s in &mut states { + s.last_completed_at = completions.get(&s.id).copied(); + } let mut snoozed: HashSet = table::list_active_snoozes(conn, now) .await? @@ -315,6 +443,7 @@ async fn apply_engine_actions( conn: &sea_orm::DatabaseConnection, data: &TickData, down_streaks: &HashMap, + fresh: &HashSet, now: i64, ) -> Result<(), infra::errors::Error> { let checks: Vec = data @@ -327,6 +456,7 @@ async fn apply_engine_actions( consecutive_failures: down_streaks.get(&s.id).copied().unwrap_or(0), first_fail_at: None, failing_since: None, + fresh_result: fresh.contains(&s.id), }) .collect(); @@ -358,12 +488,13 @@ async fn apply_engine_actions( } } - let actions = engine::incident_actions( + let th_by_check = thresholds_by_check(data); + let actions = engine::incident_actions_with( &checks, &open_by_check, &recently_resolved, &data.snoozed, - &engine::Thresholds::default(), + |id| th_by_check.get(id).copied().unwrap_or_default(), now, ); for action in actions { @@ -616,8 +747,19 @@ fn build_page_snapshot( }); } } - let has_checks = data.mappings.iter().any(|m| m.component_id == component.id); - if !has_checks { + let mapped: Vec<&str> = data + .mappings + .iter() + .filter(|m| m.component_id == component.id) + .map(|m| m.synthetics_id.as_str()) + .collect(); + if mapped.is_empty() { + status = engine::ComponentStatus::NoData; + } else if status == engine::ComponentStatus::Operational + && all_checks_in_error(&mapped, &data.states) + { + // Every probe is failing to produce a verdict: we cannot say the + // service is up, so show "no data" rather than plain Operational. status = engine::ComponentStatus::NoData; } current_components.push(engine::CurrentComponent { @@ -649,7 +791,24 @@ fn build_page_snapshot( generated_at: now, components: history_components, }; - (current, history, page_notice_ids.into_iter().collect()) + // Sorted: HashSet order is random per process, and this list feeds the + // serialized snapshot whose hash decides whether to rewrite it. + (current, history, sorted_ids(page_notice_ids)) +} + +fn sorted_ids(set: HashSet) -> Vec { + let mut ids: Vec = set.into_iter().collect(); + ids.sort_unstable(); + ids +} + +fn all_checks_in_error(mapped: &[&str], states: &[table::CheckStateRow]) -> bool { + mapped.iter().all(|id| { + states + .iter() + .find(|s| s.id == *id) + .is_some_and(|s| s.last_check_status == engine::CHECK_STATUS_ERROR) + }) } fn public_notice( @@ -720,6 +879,116 @@ pub async fn run_domain_verifier() { #[cfg(not(feature = "enterprise"))] pub async fn run_domain_verifier() {} +#[cfg(test)] +mod fast_pass_tests { + use super::*; + + fn row(id: &str, status: i32, completed: i64) -> table::CheckStateRow { + row_at(id, status, Some(completed)) + } + + fn row_at(id: &str, status: i32, completed: Option) -> table::CheckStateRow { + table::CheckStateRow { + id: id.into(), + org_id: "org".into(), + last_check_status: status, + consecutive_failures: 0, + last_triggered_at: 0, + last_completed_at: completed, + } + } + + /// The double count Codex found: dispatching the next run used to move the + /// marker while the status still showed the previous result, so one DOWN + /// counted twice and `confirm_failures = 2` opened on a single failure. + #[test] + fn dispatch_without_completion_does_not_count_again() { + let (mut d, mut seen) = (HashMap::new(), HashMap::new()); + let down = engine::CHECK_STATUS_DOWN; + assert!(observe_run(&mut d, &mut seen, &row("c", down, 100))); + // Next run dispatched (last_triggered_at moves), not yet completed. + let mut dispatched = row("c", down, 100); + dispatched.last_triggered_at = 999; + assert!(!observe_run(&mut d, &mut seen, &dispatched)); + assert_eq!(d["c"], 1); + } + + /// The latest completion falling out of the lookback window is not a new + /// result, and its return is not one either. + #[test] + fn no_completion_in_window_is_not_fresh() { + let (mut d, mut seen) = (HashMap::new(), HashMap::new()); + let down = engine::CHECK_STATUS_DOWN; + assert!(!observe_run(&mut d, &mut seen, &row_at("c", down, None))); + assert!(!d.contains_key("c")); + assert!(observe_run(&mut d, &mut seen, &row("c", down, 100))); + assert!(!observe_run(&mut d, &mut seen, &row_at("c", down, None))); + assert!(!observe_run(&mut d, &mut seen, &row("c", down, 100))); + assert_eq!(d["c"], 1); + } + + #[test] + fn down_streak_counts_runs_not_ticks() { + let (mut d, mut seen) = (HashMap::new(), HashMap::new()); + let down = engine::CHECK_STATUS_DOWN; + assert!(observe_run(&mut d, &mut seen, &row("c", down, 100))); + for _ in 0..10 { + assert!(!observe_run(&mut d, &mut seen, &row("c", down, 100))); + } + assert_eq!(d["c"], 1); + assert!(observe_run(&mut d, &mut seen, &row("c", down, 200))); + assert_eq!(d["c"], 2); + observe_run( + &mut d, + &mut seen, + &row("c", engine::CHECK_STATUS_ERROR, 300), + ); + assert!(!d.contains_key("c")); + } + + #[test] + fn strictest_takes_max_and_requires_time_confirm_everywhere() { + let a = engine::Thresholds { + confirm_failures: 3, + confirm_recovery: 4, + confirm_after_micros: Some(10), + }; + let b = engine::Thresholds { + confirm_failures: 7, + confirm_recovery: 2, + confirm_after_micros: None, + }; + let s = strictest(a, b); + assert_eq!((s.confirm_failures, s.confirm_recovery), (7, 4)); + assert_eq!(s.confirm_after_micros, None); + let c = engine::Thresholds { + confirm_after_micros: Some(30), + ..a + }; + assert_eq!(strictest(a, c).confirm_after_micros, Some(30)); + } + + #[test] + fn notice_ids_serialize_in_a_stable_order() { + let ids = ["n3", "n1", "n2", "n9", "n0"]; + let fwd: HashSet = ids.iter().map(|s| s.to_string()).collect(); + let rev: HashSet = ids.iter().rev().map(|s| s.to_string()).collect(); + let a = serde_json::to_string(&sorted_ids(fwd)).unwrap(); + let b = serde_json::to_string(&sorted_ids(rev)).unwrap(); + assert_eq!(a, b); + assert_eq!(a, r#"["n0","n1","n2","n3","n9"]"#); + } + + #[test] + fn all_error_component_is_not_operational() { + let e = engine::CHECK_STATUS_ERROR; + let states = vec![row("a", e, 1), row("b", e, 1), row("c", 1, 1)]; + assert!(all_checks_in_error(&["a", "b"], &states)); + assert!(!all_checks_in_error(&["a", "c"], &states)); + assert!(!all_checks_in_error(&["a", "missing"], &states)); + } +} + #[cfg(test)] mod tests { use sea_orm::{DatabaseBackend, MockDatabase}; diff --git a/src/usage_reporting/src/lib.rs b/src/usage_reporting/src/lib.rs index 3170840a29..bf31554d2d 100644 --- a/src/usage_reporting/src/lib.rs +++ b/src/usage_reporting/src/lib.rs @@ -684,6 +684,7 @@ mod tests { ack_source: None, taken_over_from: None, delivery_attempted: None, + repeat_suppressed: None, trigger_source: None, triggered_by: None, recipient_count: None, diff --git a/tests/api-testing/README.md b/tests/api-testing/README.md index acd0946e49..5df6f73109 100644 --- a/tests/api-testing/README.md +++ b/tests/api-testing/README.md @@ -115,6 +115,7 @@ file should pass strict ruff. |---|---| | `framework` | Self-tests for `support/` framework. Run with `-m framework`. | | `enterprise` | Needs an enterprise build (e.g. the on-call suite, whose routes an OSS server answers 404). Run with `-m enterprise`; deselect with `-m "not enterprise"`. | +| `slow` | Long-running test (e.g. the profiles 100k-stack ingest, ~2 min). Runs in the normal suite; deselect with `-m "not slow"`. | | `order(N)` | Run order pin (legacy — avoid in new tests). | ## Troubleshooting diff --git a/tests/api-testing/pyproject.toml b/tests/api-testing/pyproject.toml index fa9873c0cd..75ca6ad646 100644 --- a/tests/api-testing/pyproject.toml +++ b/tests/api-testing/pyproject.toml @@ -63,6 +63,7 @@ markers = [ "order: mark test to run in a specific order", "framework: self-tests for the support/ framework (Phase 2)", "enterprise: needs an enterprise build with the feature flag on (on-call). Run with -m enterprise.", + "slow: long-running test (e.g. profiles 100k-stack ingest); runs in the normal suite. Deselect with -m \"not slow\".", ] [tool.ruff] diff --git a/tests/api-testing/tests/profiles/__init__.py b/tests/api-testing/tests/profiles/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/api-testing/tests/profiles/conftest.py b/tests/api-testing/tests/profiles/conftest.py new file mode 100644 index 0000000000..31c08059fb --- /dev/null +++ b/tests/api-testing/tests/profiles/conftest.py @@ -0,0 +1,85 @@ +"""Fixtures for the Continuous Profiling (profiles) API tests.""" + +from __future__ import annotations + +import logging +from collections.abc import Callable, Generator + +import pytest + +from support.client import OpenObserveClient +from support.factories import unique_name + +from .profiles_helpers import ( + ALL_CPU_TOTAL, + CHECKOUT_TOTAL, + PAYMENTS, + US, + RefData, + cpu, + ingest_json, + now_us, + reference_model, + reference_t0_ns, + to_json, + wait_merge_total, + window, +) + +log = logging.getLogger(__name__) + + +def _delete_profiles_stream(client: OpenObserveClient, name: str) -> None: + try: + resp = client.streams.delete(name, type_="profiles") + except Exception as e: + log.warning("profiles stream cleanup failed for %s: %s", name, e) + return + if resp.status_code not in (200, 404): + log.warning("profiles stream cleanup failed for %s: %s %s", name, resp.status_code, resp.text[:200]) + + +@pytest.fixture +def new_stream(client: OpenObserveClient) -> Generator[Callable[[str], str], None, None]: + """Factory for unique profiles stream names; every name handed out is deleted after the test.""" + names: list[str] = [] + + def _make(prefix: str = "prof") -> str: + name = unique_name(prefix) + names.append(name) + return name + + yield _make + for name in names: + _delete_profiles_stream(client, name) + + +@pytest.fixture +def prof_stream(new_stream: Callable[[str], str]) -> str: + return new_stream("prof") + + +@pytest.fixture(scope="module") +def ref(client: OpenObserveClient) -> Generator[RefData, None, None]: + """Reference dataset (feature §7) in its own stream, visible before any test reads it.""" + stream = unique_name("prof_ref") + t0_ns = reference_t0_ns() + data = RefData(stream=stream, t0_us=t0_ns // 1000) + try: + resp = ingest_json(client, stream, to_json(reference_model(t0_ns))) + assert resp.status_code == 200, f"reference ingest failed: {resp.status_code} {resp.text}" + assert resp.json() == {}, f"reference ingest partially rejected: {resp.text}" + wait_merge_total(client, stream, data.win(**cpu(service_name="checkout")), CHECKOUT_TOTAL) + wait_merge_total(client, stream, data.win(**cpu()), ALL_CPU_TOTAL) + total, unit = PAYMENTS["inuse_space"] + wait_merge_total(client, stream, data.win(profile_type="inuse_space", profile_unit=unit), total) + yield data + finally: + _delete_profiles_stream(client, stream) + + +@pytest.fixture +def empty_window() -> dict: + """A one-hour window three days ago; no profile in these tests is ever stored there.""" + end = now_us() - 3 * 24 * 3600 * US + return window(end - 3600 * US, end) diff --git a/tests/api-testing/tests/profiles/profiles_helpers.py b/tests/api-testing/tests/profiles/profiles_helpers.py new file mode 100644 index 0000000000..0b49257d21 --- /dev/null +++ b/tests/api-testing/tests/profiles/profiles_helpers.py @@ -0,0 +1,728 @@ +"""OTLP profiles builder (JSON + hand-encoded protobuf, as the Python proto package lacks v1development), query wrappers, reference dataset.""" + +from __future__ import annotations + +import copy +import json +import sys +import time +from dataclasses import dataclass +from typing import Any + +import requests + +from support.client import OpenObserveClient +from support.wait import WaitTimeout, wait_until + +NS = 1_000_000_000 +US = 1_000_000 +MIN_US = 60 * US +PARTIAL_MSG = "Some profiles were rejected due to out-of-window timestamps, malformed samples, or empty samples" +BAD_CT_BODY = {"code": 3, "message": "Bad Request: Content-Type must be application/json or application/x-protobuf"} +GRPC_EXPORT = "/opentelemetry.proto.collector.profiles.v1development.ProfilesService/Export" +# Older than any ZO_INGEST_ALLOWED_UPTO in use (CI 48 h, local up to 4000 h). +STALE_NS = 400 * 24 * 3600 * NS + +# A 1000-frame stack nests ~2000 JSON levels in merge/diff trees, past the default decoder limit. +sys.setrecursionlimit(max(sys.getrecursionlimit(), 20_000)) + + +def now_ns() -> int: + return time.time_ns() + + +def now_us() -> int: + return time.time_ns() // 1000 + + +class ProfilesBuilder: + """Collects one shared dictionary plus resources; `model()` returns a snake_case request dict.""" + + def __init__(self) -> None: + self.strings: list[str] = [""] + self._str: dict[str, int] = {"": 0} + self.functions: list[dict] = [{}] + self._fn: dict[tuple, int] = {} + self.locations: list[dict] = [{}] + self._loc: dict[str, int] = {} + self.stacks: list[dict] = [{"location_indices": []}] + self._stack: dict[tuple, int] = {} + self.links: list[dict] = [{}] + self.attributes: list[dict] = [{}] + self.mappings: list[dict] = [{}] + self.resources: list[dict] = [] + + def s(self, text: str) -> int: + if text not in self._str: + self._str[text] = len(self.strings) + self.strings.append(text) + return self._str[text] + + def func(self, name: str | None, system_name: str | None = None) -> int: + key = (name, system_name) + if key not in self._fn: + entry: dict[str, int] = {} + if name: + entry["name_strindex"] = self.s(name) + if system_name: + entry["system_name_strindex"] = self.s(system_name) + self._fn[key] = len(self.functions) + self.functions.append(entry) + return self._fn[key] + + def loc_fn(self, *names: str) -> int: + """Symbolized location; several names = inlined lines, in line order.""" + key = "fn:" + "\x00".join(names) + if key not in self._loc: + self._loc[key] = len(self.locations) + self.locations.append({"lines": [{"function_index": self.func(n)} for n in names]}) + return self._loc[key] + + def loc_raw(self, **fields: Any) -> int: + self.locations.append(dict(fields)) + return len(self.locations) - 1 + + def mapping(self, filename: str, memory_start: int, file_offset: int = 0, attrs: list[int] | None = None) -> int: + entry = { + "memory_start": memory_start, + "memory_limit": memory_start + 0x100000, + "file_offset": file_offset, + "filename_strindex": self.s(filename), + } + if attrs: + entry["attribute_indices"] = attrs + self.mappings.append(entry) + return len(self.mappings) - 1 + + def attr(self, key: str, value: Any) -> int: + self.attributes.append({"key_strindex": self.s(key), "value": any_value(value)}) + return len(self.attributes) - 1 + + def stack(self, frames: list[str]) -> int: + """Stack of symbolized frames, leaf first.""" + return self.stack_locs([self.loc_fn(f) for f in frames]) + + def stack_locs(self, locs: list[int]) -> int: + key = tuple(locs) + if key not in self._stack: + self._stack[key] = len(self.stacks) + self.stacks.append({"location_indices": list(locs)}) + return self._stack[key] + + def link(self, trace_hex: str, span_hex: str) -> int: + self.links.append({"trace_id": bytes.fromhex(trace_hex), "span_id": bytes.fromhex(span_hex)}) + return len(self.links) - 1 + + def sample( + self, + frames: list[str] | None = None, + *, + stack: int | None = None, + values=(), + timestamps=(), + attrs: dict[str, Any] | None = None, + link: int = 0, + ) -> dict: + idx = stack if stack is not None else (self.stack(frames) if frames else 0) + out: dict[str, Any] = {"stack_index": idx, "values": list(values), "timestamps_unix_nano": list(timestamps)} + if attrs: + out["attribute_indices"] = [self.attr(k, v) for k, v in attrs.items()] + if link: + out["link_index"] = link + return out + + def stacks_samples(self, stacks: list[tuple[list[str], int]]) -> list[dict]: + return [self.sample(frames, values=[value]) for frames, value in stacks] + + def profile( + self, + samples: list[dict], + *, + type_: str = "cpu", + unit: str = "nanoseconds", + time_ns: int = 0, + duration_ns: int = 10 * NS, + period_type: tuple[str, str] | None = None, + period: int = 0, + profile_id: bytes | None = None, + ) -> dict: + out: dict[str, Any] = { + "sample_type": {"type_strindex": self.s(type_), "unit_strindex": self.s(unit)}, + "samples": samples, + "time_unix_nano": time_ns, + "duration_nano": duration_ns, + } + if period_type: + out["period_type"] = {"type_strindex": self.s(period_type[0]), "unit_strindex": self.s(period_type[1])} + if period: + out["period"] = period + if profile_id is not None: + out["profile_id"] = profile_id + return out + + def resource( + self, + profiles: list[dict], + attrs: dict[str, Any] | None = None, + *, + scope_name: str = "", + scope_version: str = "", + ) -> dict: + res = { + "resource": {"attributes": [{"key": k, "value": any_value(v)} for k, v in (attrs or {}).items()]}, + "scope_profiles": [{"scope": {"name": scope_name, "version": scope_version}, "profiles": profiles}], + } + self.resources.append(res) + return res + + def model(self) -> dict: + return copy.deepcopy( + { + "resource_profiles": self.resources, + "dictionary": { + "mapping_table": self.mappings, + "location_table": self.locations, + "function_table": self.functions, + "link_table": self.links, + "string_table": self.strings, + "attribute_table": self.attributes, + "stack_table": self.stacks, + }, + } + ) + + +def any_value(value: Any) -> dict: + if isinstance(value, bool): + return {"bool_value": value} + if isinstance(value, int): + return {"int_value": value} + if isinstance(value, list): + return {"array_value": {"values": [any_value(v) for v in value]}} + if isinstance(value, dict): + return value + return {"string_value": value} + + +def simple_request( + stacks: list[tuple[list[str], int]], + *, + service: str = "svc-a", + type_: str = "cpu", + unit: str = "nanoseconds", + time_ns: int | None = None, + attrs: dict[str, Any] | None = None, +) -> dict: + b = ProfilesBuilder() + t = time_ns if time_ns is not None else now_ns() - 60 * NS + b.resource( + [b.profile(b.stacks_samples(stacks), type_=type_, unit=unit, time_ns=t)], + {"service.name": service, **(attrs or {})}, + ) + return b.model() + + +def _camel(key: str) -> str: + head, *rest = key.split("_") + return head + "".join(p.capitalize() for p in rest) + + +def to_json(model: Any, key: str = "") -> Any: + """snake_case model -> OTLP/JSON (camelCase, hex trace/span/profile ids, int64 as strings).""" + if isinstance(model, dict): + return {_camel(k): to_json(v, k) for k, v in model.items()} + if isinstance(model, list): + return [to_json(v, key) for v in model] + if isinstance(model, bytes): + return model.hex() + if key in ("values", "timestamps_unix_nano", "time_unix_nano", "duration_nano", "int_value") and isinstance( + model, int + ): + return str(model) + return model + + +def _varint(n: int) -> bytes: + n &= (1 << 64) - 1 + out = bytearray() + while True: + b = n & 0x7F + n >>= 7 + if n: + out.append(b | 0x80) + else: + out.append(b) + return bytes(out) + + +def _tag(field: int, wire: int) -> bytes: + return _varint((field << 3) | wire) + + +def _int(field: int, v: int) -> bytes: + return _tag(field, 0) + _varint(v) if v else b"" + + +def _len(field: int, data: bytes) -> bytes: + return _tag(field, 2) + _varint(len(data)) + data + + +def _str(field: int, s: str) -> bytes: + return _len(field, s.encode()) if s else b"" + + +def _bytes(field: int, b: bytes) -> bytes: + return _len(field, b) if b else b"" + + +def _fixed64(field: int, v: int) -> bytes: + return _tag(field, 1) + (v & ((1 << 64) - 1)).to_bytes(8, "little") if v else b"" + + +def _packed(field: int, vals: list[int]) -> bytes: + return _len(field, b"".join(_varint(v) for v in vals)) if vals else b"" + + +def _packed_fixed64(field: int, vals: list[int]) -> bytes: + return _len(field, b"".join(v.to_bytes(8, "little") for v in vals)) if vals else b"" + + +def _pb_any(v: dict | None) -> bytes: + if not v: + return b"" + if "string_value" in v: + return _len(1, v["string_value"].encode()) + if "bool_value" in v: + return _tag(2, 0) + _varint(int(v["bool_value"])) + if "int_value" in v: + return _tag(3, 0) + _varint(v["int_value"]) + if "array_value" in v: + return _len(5, b"".join(_len(1, _pb_any(x)) for x in v["array_value"]["values"])) + if "string_value_strindex" in v: + return _tag(8, 0) + _varint(v["string_value_strindex"]) + raise ValueError(f"unsupported AnyValue {v}") + + +def _pb_kv(kv: dict) -> bytes: + return ( + _str(1, kv.get("key", "")) + + (_len(2, _pb_any(kv["value"])) if kv.get("value") else b"") + + _int(3, kv.get("key_strindex", 0)) + ) + + +def _pb_value_type(vt: dict | None) -> bytes: + return _int(1, vt.get("type_strindex", 0)) + _int(2, vt.get("unit_strindex", 0)) if vt else b"" + + +def _pb_sample(s: dict) -> bytes: + return ( + _int(1, s.get("stack_index", 0)) + + _packed(2, s.get("attribute_indices", [])) + + _int(3, s.get("link_index", 0)) + + _packed(4, s.get("values", [])) + + _packed_fixed64(5, s.get("timestamps_unix_nano", [])) + ) + + +def _pb_profile(p: dict) -> bytes: + out = _len(1, _pb_value_type(p.get("sample_type"))) if p.get("sample_type") else b"" + out += b"".join(_len(2, _pb_sample(s)) for s in p.get("samples", [])) + out += _fixed64(3, p.get("time_unix_nano", 0)) + _int(4, p.get("duration_nano", 0)) + out += _len(5, _pb_value_type(p["period_type"])) if p.get("period_type") else b"" + out += _int(6, p.get("period", 0)) + _bytes(7, p.get("profile_id", b"")) + return out + _packed(11, p.get("attribute_indices", [])) + + +def _pb_dictionary(d: dict) -> bytes: + out = b"" + for m in d.get("mapping_table", []): + out += _len( + 1, + _int(1, m.get("memory_start", 0)) + + _int(2, m.get("memory_limit", 0)) + + _int(3, m.get("file_offset", 0)) + + _int(4, m.get("filename_strindex", 0)) + + _packed(5, m.get("attribute_indices", [])), + ) + for loc in d.get("location_table", []): + lines = b"".join(_len(3, _int(1, ln.get("function_index", 0))) for ln in loc.get("lines", [])) + out += _len( + 2, + _int(1, loc.get("mapping_index", 0)) + + _int(2, loc.get("address", 0)) + + lines + + _packed(4, loc.get("attribute_indices", [])), + ) + for f in d.get("function_table", []): + out += _len( + 3, + _int(1, f.get("name_strindex", 0)) + + _int(2, f.get("system_name_strindex", 0)) + + _int(3, f.get("filename_strindex", 0)), + ) + for link in d.get("link_table", []): + out += _len(4, _bytes(1, link.get("trace_id", b"")) + _bytes(2, link.get("span_id", b""))) + for s in d.get("string_table", []): + out += _len(5, s.encode()) + for a in d.get("attribute_table", []): + out += _len( + 6, + _int(1, a.get("key_strindex", 0)) + + (_len(2, _pb_any(a["value"])) if a.get("value") else b"") + + _int(3, a.get("unit_strindex", 0)), + ) + for st in d.get("stack_table", []): + out += _len(7, _packed(1, st.get("location_indices", []))) + return out + + +def to_proto(model: dict) -> bytes: + out = b"" + for rp in model.get("resource_profiles", []): + res = b"".join(_len(1, _pb_kv(kv)) for kv in rp.get("resource", {}).get("attributes", [])) + body = _len(1, res) + for sp in rp.get("scope_profiles", []): + scope = sp.get("scope", {}) + sbody = _len(1, _str(1, scope.get("name", "")) + _str(2, scope.get("version", ""))) + sbody += b"".join(_len(2, _pb_profile(p)) for p in sp.get("profiles", [])) + body += _len(2, sbody) + out += _len(1, body) + if "dictionary" in model: + out += _len(2, _pb_dictionary(model["dictionary"])) + return out + + +def decode_fields(data: bytes) -> dict[int, list]: + """Minimal protobuf reader for the export response: {field: [varint | bytes]}.""" + fields: dict[int, list] = {} + i = 0 + while i < len(data): + key, i = _read_varint(data, i) + field, wire = key >> 3, key & 7 + if wire == 0: + val, i = _read_varint(data, i) + elif wire == 2: + ln, i = _read_varint(data, i) + val, i = data[i : i + ln], i + ln + else: + raise ValueError(f"unexpected wire type {wire}") + fields.setdefault(field, []).append(val) + return fields + + +def _read_varint(data: bytes, i: int) -> tuple[int, int]: + shift = result = 0 + while True: + b = data[i] + i += 1 + result |= (b & 0x7F) << shift + if not b & 0x80: + return result, i + shift += 7 + + +def snappy_raw(data: bytes) -> bytes: + """Raw (unframed) snappy block made only of literal chunks; valid input for any snappy decoder.""" + out = bytearray(_varint(len(data))) + for i in range(0, len(data), 65536): + chunk = data[i : i + 65536] + n = len(chunk) - 1 + if n < 60: + out.append(n << 2) + elif n < 256: + out += bytes([60 << 2, n]) + else: + out += bytes([61 << 2]) + n.to_bytes(2, "little") + out += chunk + return bytes(out) + + +def ingest_json( + client: OpenObserveClient, + stream: str | None, + body: Any, + *, + path: str = "v1/profiles", + content_type: str = "application/json", + **kw: Any, +) -> requests.Response: + data = body if isinstance(body, bytes | str) else json.dumps(body) + headers = {"Content-Type": content_type, **kw.pop("headers", {})} + if stream is not None: + headers["stream-name"] = stream + return client.post(path, data=data, headers=headers, **kw) + + +def ingest_proto(client: OpenObserveClient, stream: str | None, body: bytes | dict, **kw: Any) -> requests.Response: + data = body if isinstance(body, bytes) else to_proto(body) + return ingest_json(client, stream, data, content_type="application/x-protobuf", **kw) + + +def query( + client: OpenObserveClient, stream: str, endpoint: str, body: dict | None = None, *, method: str = "POST", **kw: Any +) -> requests.Response: + return client.request(method, f"{stream}/profiles/{endpoint}", json=body, **kw) + + +def merge(client: OpenObserveClient, stream: str, body: dict, **kw: Any) -> requests.Response: + return query(client, stream, "merge", body, **kw) + + +def merge_ok(client: OpenObserveClient, stream: str, body: dict) -> dict: + resp = merge(client, stream, body) + assert resp.status_code == 200, f"merge {body} -> {resp.status_code} {resp.text[:500]}" + return resp.json() + + +def wait_merge_total( + client: OpenObserveClient, + stream: str, + body: dict, + expected: int, + *, + timeout: float = 60, + request_timeout: float = 30, +) -> dict: + """Poll merge until `total == expected`; times out loudly with the last body.""" + last: dict = {} + + def _ready() -> dict | None: + nonlocal last + resp = merge(client, stream, body, timeout=request_timeout) + last = {"status": resp.status_code, "body": resp.text[:300]} + if resp.status_code == 200 and resp.json().get("total") == expected: + return resp.json() + return None + + try: + return wait_until(_ready, timeout=timeout, interval=0.5, msg=f"merge total {expected} on {stream}") + except WaitTimeout as e: + raise WaitTimeout(f"{e}; last merge response: {last}") from e + + +def search_rows( + client: OpenObserveClient, stream: str, *, start_us: int, end_us: int, where: str = "", size: int = 2000 +) -> list[dict]: + sql = f'SELECT * FROM "{stream}"' + (f" WHERE {where}" if where else "") + " ORDER BY _timestamp ASC" + resp = client.post( + "_search?type=profiles", + json={"query": {"sql": sql, "start_time": start_us, "end_time": end_us, "from": 0, "size": size}}, + ) + assert resp.status_code == 200, f"_search on {stream} -> {resp.status_code} {resp.text[:300]}" + return resp.json().get("hits", []) + + +def wait_rows( + client: OpenObserveClient, + stream: str, + count: int, + *, + start_us: int | None = None, + end_us: int | None = None, + where: str = "", + timeout: float = 60, +) -> list[dict]: + """Poll `_search` until exactly `count` rows are visible.""" + start = start_us if start_us is not None else now_us() - 6 * 3600 * US + end = end_us if end_us is not None else now_us() + 26 * 3600 * US + return wait_until( + lambda: (rows := search_rows(client, stream, start_us=start, end_us=end, where=where)) + and len(rows) == count + and rows, + timeout=timeout, + msg=f"{count} rows in {stream} where {where!r}", + ) + + +def profiles_streams(client: OpenObserveClient) -> list[str]: + resp = client.get("streams?type=profiles") + assert resp.status_code == 200, resp.text + return [s["name"] for s in resp.json().get("list", [])] + + +def count_nodes(node: dict) -> int: + return 1 + sum(count_nodes(c) for c in node.get("children") or []) + + +def child(node: dict, name: str) -> dict: + matches = [c for c in node.get("children") or [] if c["name"] == name] + assert len(matches) == 1, ( + f"expected one child {name!r} under {node['name']!r}, got {[c['name'] for c in node.get('children') or []]}" + ) + return matches[0] + + +def assert_tree_sums(node: dict) -> None: + kids = node.get("children") or [] + assert node["total"] == node["self"] + sum(c["total"] for c in kids), ( + f"node {node['name']} total != self + children" + ) + for c in kids: + assert_tree_sums(c) + + +def window(start_us: int, end_us: int, **extra: Any) -> dict: + return {"start_time": start_us, "end_time": end_us, **extra} + + +def recent_window(**extra: Any) -> dict: + """A window that covers every per-test profile (now - 6h .. now + 26h).""" + return window(now_us() - 6 * 3600 * US, now_us() + 26 * 3600 * US, **extra) + + +def cpu(**extra: Any) -> dict: + return {"profile_type": "cpu", "profile_unit": "nanoseconds", **extra} + + +TRACE_ID = "0af7651916cd43dd8448eb211c80319c" +SPAN_ID = "b7ad6b7169203331" +XSS = "" +CHECKOUT_TOTAL = 12_000_000_000 +ALL_CPU_TOTAL = 21_015_000_000 +CATALOG_TOTAL = 5_515_000_000 +PAYMENTS = { + "cpu": (3_500_000_000, "nanoseconds"), + "alloc_space": (1_572_864, "bytes"), + "inuse_space": (6_291_456, "bytes"), +} + + +@dataclass(frozen=True) +class RefData: + stream: str + t0_us: int + + def win(self, **extra: Any) -> dict: + """T0 - 1m .. T0 + 9m, the window that holds every reference profile.""" + return window(self.t0_us - MIN_US, self.t0_us + 9 * MIN_US, **extra) + + def bucket(self, minutes: int) -> int: + """Start (us) of the one-minute bucket that holds T0 + `minutes`.""" + return self.t0_us - 5 * US + minutes * MIN_US + + +def reference_t0_ns() -> int: + minute = now_ns() // (60 * NS) * 60 * NS + return minute - 10 * 60 * NS + 5 * NS + + +def reference_model(t0_ns: int) -> dict: + b = ProfilesBuilder() + m = 60 * NS + + def res(svc, ver, env, inst, host): + return { + "service.name": svc, + "service.version": ver, + "deployment.environment": env, + "service.instance.id": inst, + "host.name": host, + } + + order = ["handle_order", "main"] + c1 = [(["db_query", *order], 2 * NS), (["json_encode", *order], NS), (order, NS)] + b.resource( + [b.profile(b.stacks_samples(c1), time_ns=t0_ns)], res("checkout", "v1.4.0", "prod", "checkout-1", "host-a") + ) + + proc = ["process", "main"] + b.resource( + [ + b.profile( + b.stacks_samples([(["charge_card", *proc], 1_200_000_000), (["validate", *proc], 800_000_000)]), + time_ns=t0_ns + 2 * m, + ), + b.profile( + b.stacks_samples([(["alloc_buffer", *proc], 1_048_576), (["alloc_json", *proc], 524_288)]), + type_="alloc_space", + unit="bytes", + time_ns=t0_ns + 2 * m, + ), + b.profile( + b.stacks_samples([(["cache_put", *proc], 2_097_152)]), + type_="inuse_space", + unit="bytes", + time_ns=t0_ns + 2 * m, + ), + ], + res("payments", "v1.4.0", "prod", "payments-1", "host-d"), + ) + + link = b.link(TRACE_ID, SPAN_ID) + c2 = b.stacks_samples( + [ + (["db_query", *order], 4 * NS), + (["json_encode", *order], 3 * NS), + (order, 500_000_000), + ([XSS, *order], 100_000_000), + ] + ) + c2.append(b.sample(["process_payment", *order], values=[250_000_000], link=link)) + b.resource([b.profile(c2, time_ns=t0_ns + 4 * m)], res("checkout", "v1.5.0", "prod", "checkout-2", "host-b")) + + b.resource( + [ + b.profile(b.stacks_samples([(["charge_card", *proc], 1_500_000_000)]), time_ns=t0_ns + 5 * m), + b.profile( + b.stacks_samples([(["cache_put", *proc], 4_194_304)]), + type_="inuse_space", + unit="bytes", + time_ns=t0_ns + 5 * m, + ), + ], + res("payments", "v1.5.0", "prod", "payments-2", "host-e"), + ) + + b.resource( + [b.profile(b.stacks_samples([(["gc_sweep", "runtime_main"], 150_000_000)]), time_ns=t0_ns + 6 * m)], + res("checkout", "v1.5.0", "stage", "checkout-3", "host-c"), + ) + + k1 = [([f"wide_fn_{n:03d}", "wide_root", "catalog_main"], (n + 1) * 100_000) for n in range(300)] + k1.append(([f"deep_{i:04d}" for i in range(1100)], NS)) + b.resource( + [b.profile(b.stacks_samples(k1), time_ns=t0_ns + 7 * m)], + res("catalog", "v1.5.0", "stage", "catalog-1", "host-f"), + ) + return b.model() + + +def ingest_ok(client: OpenObserveClient, stream: str | None, model: dict) -> None: + """JSON ingest that must be fully accepted (200 and an empty OTLP response).""" + resp = ingest_json(client, stream, to_json(model)) + assert resp.status_code == 200, f"ingest -> {resp.status_code} {resp.text[:500]}" + assert resp.json() == {}, f"expected full success, got {resp.text[:500]}" + + +def by_stack(rows: list[dict]) -> dict[str, dict]: + out = {r.get("stack", ""): r for r in rows} + assert len(out) == len(rows), f"duplicate stacks in {rows}" + return out + + +MIB = 1_048_576 + + +def snapshot_model(minute_ns: int, threads: tuple[str, str] | None = None) -> dict: + """One instance, two inuse snapshots (1 MiB at +5 s, 3 MiB at +25 s) inside the minute starting at `minute_ns`.""" + b = ProfilesBuilder() + profiles = [] + for i, (offset, value) in enumerate(((5, MIB), (25, 3 * MIB))): + attrs = {"thread.name": threads[i]} if threads else None + sample = b.sample(["snap_leaf", "snap_root"], values=[value], attrs=attrs) + profiles.append(b.profile([sample], type_="inuse_space", unit="bytes", time_ns=minute_ns + offset * NS)) + b.resource(profiles, {"service.name": "svc-snap", "service.instance.id": "snap-1", "host.name": "h1"}) + return b.model() + + +def native_model(frame_type: str | None = None) -> dict: + """Two unsymbolized frames of /usr/lib/libqa.so.1 (build id deadbeefcafe) under symbolized root `a9_root`, value 9.""" + b = ProfilesBuilder() + build_id = b.attr("process.executable.build_id.htlhash", "deadbeefcafe") + mapping = b.mapping("/usr/lib/libqa.so.1", 0x1000, 0, attrs=[build_id]) + loc_attrs = [b.attr("profile.frame.type", frame_type)] if frame_type else [] + locs = [b.loc_raw(mapping_index=mapping, address=addr, attribute_indices=loc_attrs) for addr in (0x1110, 0x1120)] + stack = b.stack_locs([*locs, b.loc_fn("a9_root")]) + b.resource([b.profile([b.sample(stack=stack, values=[9])], time_ns=now_ns() - 60 * NS)], {"service.name": "svc-a"}) + return b.model() diff --git a/tests/api-testing/tests/profiles/test_profiles_diff.py b/tests/api-testing/tests/profiles/test_profiles_diff.py new file mode 100644 index 0000000000..5c2a934784 --- /dev/null +++ b/tests/api-testing/tests/profiles/test_profiles_diff.py @@ -0,0 +1,221 @@ +"""Profiles `diff` API: before/after call tree and share-change ranking (feature §3.6).""" + +from __future__ import annotations + +import uuid + +import pytest +import requests + +from support.client import OpenObserveClient + +from .profiles_helpers import ( + MIB, + MIN_US, + NS, + US, + XSS, + RefData, + child, + cpu, + ingest_ok, + now_ns, + query, + snapshot_model, + wait_rows, + window, +) + +GROUP_1875 = {"gc_sweep", "handle_order", "main", "runtime_main"} + + +def _side(ref: RefData, version: str, **extra) -> dict: + return ref.win(**cpu(service_name="checkout", filters=[{"key": "service_version", "value": version}], **extra)) + + +def _diff(client: OpenObserveClient, stream: str, body: dict) -> requests.Response: + return query(client, stream, "diff", body) + + +def _diff_ok(client: OpenObserveClient, stream: str, body: dict) -> dict: + resp = _diff(client, stream, body) + assert resp.status_code == 200, f"diff -> {resp.status_code} {resp.text[:500]}" + return resp.json() + + +def _versions(client: OpenObserveClient, ref: RefData, before: str = "v1.4.0", after: str = "v1.5.0", **extra) -> dict: + return _diff_ok(client, ref.stream, {"baseline": _side(ref, before), "comparison": _side(ref, after), **extra}) + + +def _assert_400(resp: requests.Response, message: str) -> None: + assert resp.status_code == 400, f"{resp.status_code} {resp.text}" + assert resp.json()["message"] == message + + +def test_dif01_checkout_versions_totals(client: OpenObserveClient, ref: RefData): + body = _versions(client, ref) + assert (body["baseline_total"], body["comparison_total"]) == (4_000_000_000, 8_000_000_000) + assert (body["unit"], body["profile_type"]) == ("nanoseconds", "cpu") + assert (body["truncated"], body["tree_truncated"]) == (False, False) + + +def test_dif02_top_ordered_by_share_change(client: OpenObserveClient, ref: RefData): + top = _versions(client, ref)["top"] + names = [t["name"] for t in top] + assert len(top) == 8 + assert top[0] == { + "name": "json_encode", + "before_self": 1_000_000_000, + "before_total": 1_000_000_000, + "after_self": 3_000_000_000, + "after_total": 3_000_000_000, + } + assert top[1] == { + "name": "process_payment", + "before_self": 0, + "before_total": 0, + "after_self": 250_000_000, + "after_total": 250_000_000, + } + # The four +-1.875 pt changes are f64 shares that need not tie exactly, so only the set is fixed. + assert set(names[2:6]) == GROUP_1875 + assert names[6:] == [XSS, "db_query"] + + +def test_dif03_node_values_both_sides(client: OpenObserveClient, ref: RefData): + root = _versions(client, ref)["root"] + assert (root["total"], root["before_total"]) == (8_000_000_000, 4_000_000_000) + order = child(child(root, "main"), "handle_order") + assert (order["total"], order["before_total"]) == (7_850_000_000, 4_000_000_000) + encode = child(order, "json_encode") + assert (encode["self"], encode["before_self"]) == (3_000_000_000, 1_000_000_000) + gc = child(child(root, "runtime_main"), "gc_sweep") + assert (gc["total"], gc["before_total"]) == (150_000_000, 0) + + +def test_dif04_swap_sides(client: OpenObserveClient, ref: RefData): + body = _versions(client, ref, before="v1.5.0", after="v1.4.0") + assert (body["baseline_total"], body["comparison_total"]) == (8_000_000_000, 4_000_000_000) + assert body["top"][0]["name"] == "json_encode" + assert (body["top"][0]["before_total"], body["top"][0]["after_total"]) == (3_000_000_000, 1_000_000_000) + + +def test_dif05_one_side_without_data(client: OpenObserveClient, ref: RefData): + body = _versions(client, ref, after="v9.9.9") + assert (body["baseline_total"], body["comparison_total"]) == (4_000_000_000, 0) + + +def test_dif06_different_windows_per_side(client: OpenObserveClient, ref: RefData): + def minute(n: int) -> dict: + return window(ref.bucket(n), ref.bucket(n) + MIN_US, **cpu(service_name="checkout")) + + body = _diff_ok(client, ref.stream, {"baseline": minute(0), "comparison": minute(4)}) + assert (body["baseline_total"], body["comparison_total"]) == (4_000_000_000, 7_850_000_000) + + +def test_dif07_max_nodes_cut_keeps_both_totals(client: OpenObserveClient, ref: RefData): + body = _versions(client, ref, max_nodes=3) + assert body["tree_truncated"] is True + assert (body["baseline_total"], body["comparison_total"]) == (4_000_000_000, 8_000_000_000) + assert [(c["name"], c["total"], c["before_total"]) for c in body["root"]["children"]] == [ + ("other", 8_000_000_000, 4_000_000_000) + ] + + +def test_dif08_per_side_max_nodes_ignored(client: OpenObserveClient, ref: RefData): + body = _diff_ok( + client, ref.stream, {"baseline": _side(ref, "v1.4.0", max_nodes=3), "comparison": _side(ref, "v1.5.0")} + ) + assert body["tree_truncated"] is False + + +@pytest.mark.parametrize( + "comparison", + [{"profile_type": "alloc_space", "profile_unit": "bytes"}, {"profile_type": "cpu", "profile_unit": "milliseconds"}], + ids=["dif09-type", "dif10-unit"], +) +def test_dif09_10_mismatched_type_or_unit(client: OpenObserveClient, ref: RefData, comparison: dict): + resp = _diff( + client, ref.stream, {"baseline": _side(ref, "v1.4.0"), "comparison": {**_side(ref, "v1.5.0"), **comparison}} + ) + _assert_400(resp, "baseline and comparison must use the same profile_type and profile_unit") + + +def test_dif11_comparison_missing_type(client: OpenObserveClient, ref: RefData): + comparison = _side(ref, "v1.5.0") + del comparison["profile_type"] + _assert_400( + _diff(client, ref.stream, {"baseline": _side(ref, "v1.4.0"), "comparison": comparison}), + "profile_type is required", + ) + + +def test_dif12_baseline_missing_unit(client: OpenObserveClient, ref: RefData): + baseline = _side(ref, "v1.4.0") + del baseline["profile_unit"] + _assert_400( + _diff(client, ref.stream, {"baseline": baseline, "comparison": _side(ref, "v1.5.0")}), + "profile_unit is required", + ) + + +def test_dif13_missing_baseline(client: OpenObserveClient, ref: RefData): + resp = _diff(client, ref.stream, {"comparison": _side(ref, "v1.5.0")}) + assert resp.status_code == 400, resp.text + message = resp.json()["message"] + assert message.startswith("Failed to deserialize"), message + assert "missing field `baseline`" in message, message + + +def test_dif14_bad_filter_on_one_side(client: OpenObserveClient, ref: RefData): + comparison = _side(ref, "v1.5.0") + comparison["filters"] = [{"key": "stack", "value": "x"}] + _assert_400( + _diff(client, ref.stream, {"baseline": _side(ref, "v1.4.0"), "comparison": comparison}), + "invalid filter key: stack", + ) + + +def test_dif15_bad_time_on_one_side(client: OpenObserveClient, ref: RefData): + baseline = _side(ref, "v1.4.0") + baseline["start_time"] = baseline["end_time"] + _assert_400( + _diff(client, ref.stream, {"baseline": baseline, "comparison": _side(ref, "v1.5.0")}), + "start_time must be less than end_time", + ) + + +def test_dif16_get_not_allowed(client: OpenObserveClient, ref: RefData): + resp = query(client, ref.stream, "diff", method="GET") + assert resp.status_code == 405, resp.text + + +def test_dif17_inuse_diff_per_snapshot(client: OpenObserveClient, prof_stream: str): + minute = (now_ns() - 3 * 60 * NS) // (60 * NS) * 60 * NS + ingest_ok(client, prof_stream, snapshot_model(minute)) + wait_rows(client, prof_stream, 2) + inuse = {"profile_type": "inuse_space", "profile_unit": "bytes"} + m = minute // 1000 + body = _diff_ok( + client, + prof_stream, + {"baseline": window(m, m + 15 * US, **inuse), "comparison": window(m + 15 * US, m + 40 * US, **inuse)}, + ) + assert (body["baseline_total"], body["comparison_total"]) == (MIB, 3 * MIB) + + +def test_dif18_unknown_stream(client: OpenObserveClient, ref: RefData): + stream = f"no_such_{uuid.uuid4().hex[:8]}" + resp = _diff(client, stream, {"baseline": _side(ref, "v1.4.0"), "comparison": _side(ref, "v1.5.0")}) + assert resp.status_code == 400, resp.text + assert resp.json()["code"] == 20002 + assert resp.json()["message"] == f"Search stream not found: {stream}" + + +def test_dif19_no_auth(client: OpenObserveClient, ref: RefData): + resp = requests.post( + client.url(f"{ref.stream}/profiles/diff"), + json={"baseline": _side(ref, "v1.4.0"), "comparison": _side(ref, "v1.5.0")}, + timeout=30, + ) + assert resp.status_code == 401, resp.text diff --git a/tests/api-testing/tests/profiles/test_profiles_grpc.py b/tests/api-testing/tests/profiles/test_profiles_grpc.py new file mode 100644 index 0000000000..8f51828239 --- /dev/null +++ b/tests/api-testing/tests/profiles/test_profiles_grpc.py @@ -0,0 +1,96 @@ +"""Profiles over OTLP/gRPC: raw-bytes unary call, since the Python proto package lacks the v1development schema.""" + +from __future__ import annotations + +import uuid +from collections.abc import Generator + +import grpc +import pytest + +from support.client import OpenObserveClient +from support.otlp_grpc import grpc_target, metadata + +from .profiles_helpers import ( + GRPC_EXPORT, + NS, + PARTIAL_MSG, + ProfilesBuilder, + cpu, + decode_fields, + now_ns, + recent_window, + simple_request, + to_proto, + wait_merge_total, +) + +A_STACKS = [(["g_leaf", "g_work", "g_main"], 300), (["g_work", "g_main"], 100)] + + +@pytest.fixture(scope="module") +def export() -> Generator[grpc.UnaryUnaryMultiCallable, None, None]: + channel = grpc.insecure_channel(grpc_target()) + try: + yield channel.unary_unary(GRPC_EXPORT) + finally: + channel.close() + + +def _call(export: grpc.UnaryUnaryMultiCallable, body: bytes, md: list[tuple[str, str]]) -> bytes: + return export(body, metadata=md, timeout=30) + + +def _rpc_error(export: grpc.UnaryUnaryMultiCallable, body: bytes, md: list[tuple[str, str]]) -> grpc.RpcError: + with pytest.raises(grpc.RpcError) as info: + _call(export, body, md) + return info.value + + +def test_grpc01_export_happy_path(client: OpenObserveClient, export, prof_stream: str): + resp = _call(export, to_proto(simple_request(A_STACKS)), metadata(prof_stream)) + assert resp == b"", "a fully accepted export answers an empty ExportProfilesServiceResponse" + body = wait_merge_total(client, prof_stream, recent_window(**cpu()), 400) + assert [t["name"] for t in body["top"]] == ["g_leaf", "g_work", "g_main"] + + +def test_grpc02_missing_organization(export, prof_stream: str): + md = [kv for kv in metadata(prof_stream) if kv[0] != "organization"] + err = _rpc_error(export, to_proto(simple_request(A_STACKS)), md) + assert err.code() == grpc.StatusCode.INVALID_ARGUMENT, err + assert err.details().startswith("Please specify organization id with header key 'organization'"), err.details() + + +def test_grpc03_validation_error_is_invalid_argument(export, prof_stream: str): + model = simple_request(A_STACKS) + model["dictionary"]["stack_table"][0] = {"location_indices": [1]} + err = _rpc_error(export, to_proto(model), metadata(prof_stream)) + assert err.code() == grpc.StatusCode.INVALID_ARGUMENT, err + assert err.details() == "dictionary.stack_table[0] must be the zero value" + + +def test_grpc04_partial_success(client: OpenObserveClient, export, prof_stream: str): + b = ProfilesBuilder() + t = now_ns() - 60 * NS + bad = b.sample(["bad_leaf"], values=[1, 2], timestamps=[t]) + b.resource([b.profile([bad, b.sample(["good_leaf"], values=[50])], time_ns=t)], {"service.name": "svc-g"}) + resp = _call(export, to_proto(b.model()), metadata(prof_stream)) + partial = decode_fields(decode_fields(resp)[1][0]) + assert partial[1] == [1] + assert partial[2] == [PARTIAL_MSG.encode()] + wait_merge_total(client, prof_stream, recent_window(**cpu()), 50) + + +def test_grpc05_no_stream_name_writes_default(client: OpenObserveClient, export): + service = f"svc-grpc-noheader-{uuid.uuid4().hex[:8]}" + md = [kv for kv in metadata("unused") if kv[0] != "stream-name"] + # `default` is shared and is not deleted: a deleted stream answers 503 "being deleted" to the next run's ingest. + resp = _call(export, to_proto(simple_request([(["gd_leaf", "gd_main"], 33)], service=service)), md) + assert resp == b"" + wait_merge_total(client, "default", recent_window(**cpu(service_name=service)), 33) + + +def test_grpc06_unauthenticated(export, prof_stream: str): + md = [kv for kv in metadata(prof_stream) if kv[0] != "authorization"] + err = _rpc_error(export, to_proto(simple_request(A_STACKS)), md) + assert err.code() == grpc.StatusCode.UNAUTHENTICATED, err diff --git a/tests/api-testing/tests/profiles/test_profiles_ingest.py b/tests/api-testing/tests/profiles/test_profiles_ingest.py new file mode 100644 index 0000000000..6b53488f2f --- /dev/null +++ b/tests/api-testing/tests/profiles/test_profiles_ingest.py @@ -0,0 +1,563 @@ +"""Profiles ingestion: formats, auth, compression, stream naming and the stored row shape (feature §2.1-§2.4).""" + +from __future__ import annotations + +import gzip +import json +import os +import uuid + +import pytest +import requests + +from support.client import OpenObserveClient + +from .profiles_helpers import ( + NS, + US, + ProfilesBuilder, + by_stack, + child, + cpu, + ingest_json, + ingest_ok, + ingest_proto, + merge_ok, + native_model, + now_ns, + now_us, + profiles_streams, + query, + recent_window, + simple_request, + snappy_raw, + to_json, + to_proto, + wait_merge_total, + wait_rows, +) + +A_STACKS = [(["a_leaf", "a_work", "a_main"], 300), (["a_work", "a_main"], 100)] + + +def _one_profile(b: ProfilesBuilder, samples: list[dict], attrs: dict | None = None, **profile_kw) -> dict: + b.resource( + [b.profile(samples, time_ns=profile_kw.pop("time_ns", now_ns() - 60 * NS), **profile_kw)], + {"service.name": "svc-a", **(attrs or {})}, + ) + return b.model() + + +def _deep(n: int) -> list[str]: + return [f"deep_{i:04d}" for i in range(n)] + + +def _depth_rows(client: OpenObserveClient, stream: str, frames: int) -> dict: + ingest_ok(client, stream, simple_request([(_deep(frames), 1)])) + return wait_rows(client, stream, 1)[0] + + +def test_ing01_json_ingest(client: OpenObserveClient, prof_stream: str): + resp = ingest_json(client, prof_stream, to_json(simple_request(A_STACKS))) + assert resp.status_code == 200, resp.text + assert resp.json() == {} + wait_merge_total(client, prof_stream, recent_window(**cpu()), 400) + + +def test_ing02_v1development_path(client: OpenObserveClient, prof_stream: str): + resp = ingest_json(client, prof_stream, to_json(simple_request(A_STACKS)), path="v1development/profiles") + assert resp.status_code == 200, resp.text + assert resp.json() == {} + wait_merge_total(client, prof_stream, recent_window(**cpu()), 400) + + +def test_ing03_protobuf_ingest(client: OpenObserveClient, prof_stream: str): + resp = ingest_proto(client, prof_stream, simple_request(A_STACKS)) + assert resp.status_code == 200, resp.text + assert resp.headers["content-type"].startswith("application/x-protobuf") + assert resp.content == b"", "a fully accepted proto export answers an empty ExportProfilesServiceResponse" + wait_merge_total(client, prof_stream, recent_window(**cpu()), 400) + + +def test_ing04_json_with_charset(client: OpenObserveClient, prof_stream: str): + resp = ingest_json( + client, prof_stream, to_json(simple_request(A_STACKS)), content_type="application/json; charset=utf-8" + ) + assert resp.status_code == 200, resp.text + assert resp.json() == {} + wait_merge_total(client, prof_stream, recent_window(**cpu()), 400) + + +def test_ing05_gzip_body(client: OpenObserveClient, prof_stream: str): + body = gzip.compress(json.dumps(to_json(simple_request(A_STACKS))).encode()) + resp = ingest_json(client, prof_stream, body, headers={"Content-Encoding": "gzip"}) + assert resp.status_code == 200, resp.text + assert resp.json() == {} + wait_merge_total(client, prof_stream, recent_window(**cpu()), 400) + + +def test_ing06_snappy_body(client: OpenObserveClient, prof_stream: str): + body = snappy_raw(to_proto(simple_request(A_STACKS))) + resp = ingest_proto(client, prof_stream, body, headers={"Content-Encoding": "snappy"}) + assert resp.status_code == 200, resp.text + assert resp.content == b"" + wait_merge_total(client, prof_stream, recent_window(**cpu()), 400) + + +def test_ing07_no_stream_header_writes_default(client: OpenObserveClient): + service = f"svc-noheader-{uuid.uuid4().hex[:8]}" + # `default` is shared and is not deleted: a deleted stream answers 503 "being deleted" to the next run's ingest. + resp = ingest_json(client, None, to_json(simple_request([(["nh_leaf", "nh_main"], 77)], service=service))) + assert resp.status_code == 200, resp.text + assert resp.json() == {} + wait_merge_total(client, "default", recent_window(**cpu(service_name=service)), 77) + + +@pytest.mark.skipif( + os.environ.get("ZO_FORMAT_STREAM_NAME_TO_LOWERCASE", "true").lower() == "false", + reason="stream names keep their case when ZO_FORMAT_STREAM_NAME_TO_LOWERCASE=false", +) +def test_ing08_upper_case_stream_name_lowered(client: OpenObserveClient, new_stream): + lower = new_stream("prof_up") + resp = ingest_json(client, lower.upper(), to_json(simple_request(A_STACKS))) + assert resp.status_code == 200, resp.text + wait_merge_total(client, lower, recent_window(**cpu()), 400) + names = profiles_streams(client) + assert lower in names + assert lower.upper() not in names + + +def test_ing09_stored_row_shape(client: OpenObserveClient, prof_stream: str): + ingest_ok(client, prof_stream, simple_request(A_STACKS)) + rows = by_stack(wait_rows(client, prof_stream, 2)) + row = rows["a_leaf;a_work;a_main"] + assert row["value"] == 300 + assert row["frame_count"] == 3 + assert row["profile_type"] == "cpu" + assert row["profile_unit"] == "nanoseconds" + assert row["service_name"] == "svc-a" + assert row["extractor_version"] == "v1" + assert row["duration_nanos"] == 10 * NS + assert rows["a_work;a_main"]["value"] == 100 + assert rows["a_work;a_main"]["frame_count"] == 2 + schema = client.get(f"streams/{prof_stream}/schema?type=profiles") + assert schema.status_code == 200, schema.text + fields = {f["name"] for f in schema.json()["schema"]} + assert "event_id" not in fields + assert {"stack", "value", "frame_count", "profile_type", "profile_unit", "service_name"} <= fields + + +def test_ing10_paired_values_and_timestamps(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + t = now_ns() + stamps = [t - 3 * NS + 111_000, t - 2 * NS + 222_000, t - NS + 333_000] + ingest_ok(client, prof_stream, _one_profile(b, [b.sample(["p_leaf"], values=[10, 20, 30], timestamps=stamps)])) + rows = wait_rows(client, prof_stream, 3) + assert [(r["_timestamp"], r["value"]) for r in rows] == [ + (s // 1000, v) for s, v in zip(stamps, [10, 20, 30], strict=True) + ] + + +def test_ing11_timestamps_only_value_one(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + t = now_ns() + stamps = [t - 2 * NS, t - NS] + ingest_ok(client, prof_stream, _one_profile(b, [b.sample(["t_leaf"], timestamps=stamps)])) + rows = wait_rows(client, prof_stream, 2) + assert [(r["_timestamp"], r["value"]) for r in rows] == [(s // 1000, 1) for s in stamps] + + +def test_ing12_values_only_summed_at_profile_time(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + t = now_ns() - 30 * NS + 123_456 + ingest_ok(client, prof_stream, _one_profile(b, [b.sample(["v_leaf"], values=[5, 6, 7])], time_ns=t)) + rows = wait_rows(client, prof_stream, 1) + assert rows[0]["value"] == 18 + assert rows[0]["_timestamp"] == t // 1000 + + +def test_ing13_zero_profile_time_is_now(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + before = now_us() + ingest_ok(client, prof_stream, _one_profile(b, [b.sample(["z_leaf"], values=[3])], time_ns=0)) + rows = wait_rows(client, prof_stream, 1) + assert abs(rows[0]["_timestamp"] - before) <= 60 * US, rows[0] + + +def test_ing14_zero_sample_timestamp_uses_profile_time(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + t = now_ns() - 40 * NS + 7_000 + ingest_ok(client, prof_stream, _one_profile(b, [b.sample(["zt_leaf"], values=[7], timestamps=[0])], time_ns=t)) + rows = wait_rows(client, prof_stream, 1) + assert rows[0]["value"] == 7 + assert rows[0]["_timestamp"] == t // 1000 + + +def test_ing15_resource_attributes_to_columns(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + attrs = { + "host.name": "host-15", + "deployment.environment": "qa", + "process.pid": 4242, + "service.version": "v15", + "service.instance.id": "inst-15", + "k8s.pod.name": "pod-15", + "team.owner": "team-a", + } + ingest_ok(client, prof_stream, _one_profile(b, [b.sample(["r_leaf"], values=[1])], attrs)) + row = wait_rows(client, prof_stream, 1)[0] + assert row["host_name"] == "host-15" + assert row["deployment_environment"] == "qa" + assert str(row["process_pid"]) == "4242" + assert row["service_version"] == "v15" + assert row["service_instance_id"] == "inst-15" + assert row["k8s_pod_name"] == "pod-15" + assert row["team_owner"] == "team-a" + + +def test_ing16_empty_resource_attribute_skipped(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + ingest_ok(client, prof_stream, _one_profile(b, [b.sample(["e_leaf"], values=[1])], {"team.owner": ""})) + row = wait_rows(client, prof_stream, 1)[0] + assert "team_owner" not in row + fields = {f["name"] for f in client.get(f"streams/{prof_stream}/schema?type=profiles").json()["schema"]} + assert "team_owner" not in fields + + +def test_ing17_resource_attrs_cannot_overwrite_reserved(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + attrs = {"stack": "evil", "value": "evil", "profile.type": "x"} + ingest_ok(client, prof_stream, _one_profile(b, [b.sample(["real_leaf", "real_root"], values=[9])], attrs)) + row = wait_rows(client, prof_stream, 1)[0] + assert row["stack"] == "real_leaf;real_root" + assert row["value"] == 9 + assert row["profile_type"] == "cpu" + + +def test_ing18_sample_attributes_to_thread_columns(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + attrs = {"thread.id": 42, "thread.name": "worker-1", "cpu.logical_number": 3, "custom.tag": "x"} + ingest_ok(client, prof_stream, _one_profile(b, [b.sample(["th_leaf"], values=[1], attrs=attrs)])) + row = wait_rows(client, prof_stream, 1)[0] + assert row["thread_id"] == "42" + assert row["thread_name"] == "worker-1" + assert row["cpu_logical_number"] == "3" + assert row["custom_tag"] == "x" + + +def test_ing19_thread_id_aliases(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + samples = [ + b.sample(["os_leaf"], values=[1], attrs={"os.thread.id": "101"}), + b.sample(["us_leaf"], values=[1], attrs={"thread_id": "202"}), + ] + ingest_ok(client, prof_stream, _one_profile(b, samples)) + rows = by_stack(wait_rows(client, prof_stream, 2)) + assert rows["os_leaf"]["thread_id"] == "101" + assert rows["us_leaf"]["thread_id"] == "202" + + +def test_ing20_trace_link_lowercased_only_on_linked_sample(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + link = b.link("0af7651916cd43dd8448eb211c80319c", "b7ad6b7169203331") + body = to_json(_one_profile(b, [b.sample(["linked"], values=[5], link=link), b.sample(["plain"], values=[6])])) + body["dictionary"]["linkTable"][link] = { + "traceId": "0AF7651916CD43DD8448EB211C80319C", + "spanId": "B7AD6B7169203331", + } + resp = ingest_json(client, prof_stream, body) + assert resp.status_code == 200, resp.text + rows = by_stack(wait_rows(client, prof_stream, 2)) + assert rows["linked"]["trace_id"] == "0af7651916cd43dd8448eb211c80319c" + assert rows["linked"]["span_id"] == "b7ad6b7169203331" + assert "trace_id" not in rows["plain"] + assert "span_id" not in rows["plain"] + + +def test_ing21_profile_duration_once_per_instance(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + samples = [b.sample([f"d_leaf_{i}"], values=[i + 1]) for i in range(3)] + ingest_ok(client, prof_stream, _one_profile(b, samples)) + rows = wait_rows(client, prof_stream, 3) + assert [r.get("profile_duration_nanos") for r in rows].count(10 * NS) == 1 + assert sum("profile_duration_nanos" in r for r in rows) == 1 + + +def test_ing22_profile_duration_per_instance_in_batch(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + t = now_ns() - 60 * NS + for inst in ("a", "b"): + samples = [b.sample([f"{inst}_leaf_{i}"], values=[1]) for i in range(2)] + b.resource([b.profile(samples, time_ns=t)], {"service.name": "svc-a", "service.instance.id": f"inst-{inst}"}) + ingest_ok(client, prof_stream, b.model()) + rows = wait_rows(client, prof_stream, 4) + timed = sorted(r["service_instance_id"] for r in rows if r.get("profile_duration_nanos") == 10 * NS) + assert timed == ["inst-a", "inst-b"] + + +def test_ing23_profile_id_hex_and_zero_omitted(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + t = now_ns() - 60 * NS + pid = bytes(range(1, 17)) + b.resource( + [ + b.profile([b.sample(["pid_a"], values=[1])], time_ns=t, profile_id=pid), + b.profile([b.sample(["pid_b"], values=[1])], time_ns=t, profile_id=bytes(16)), + ], + {"service.name": "svc-a"}, + ) + ingest_ok(client, prof_stream, b.model()) + rows = by_stack(wait_rows(client, prof_stream, 2)) + assert rows["pid_a"]["profile_id"] == pid.hex() + assert "profile_id" not in rows["pid_b"] + + +def test_ing24_period_fields(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + t = now_ns() - 60 * NS + b.resource( + [ + b.profile( + [b.sample(["per_ns"], values=[1])], time_ns=t, period_type=("cpu", "nanoseconds"), period=10_000_000 + ), + b.profile([b.sample(["per_count"], values=[1])], time_ns=t, period_type=("samples", "count"), period=99), + ], + {"service.name": "svc-a"}, + ) + ingest_ok(client, prof_stream, b.model()) + rows = by_stack(wait_rows(client, prof_stream, 2)) + ns_row, count_row = rows["per_ns"], rows["per_count"] + assert (ns_row["period_nanos"], ns_row["period_type"], ns_row["period_unit"]) == (10_000_000, "cpu", "nanoseconds") + assert "period" not in ns_row + assert (count_row["period"], count_row["period_type"], count_row["period_unit"]) == (99, "samples", "count") + assert "period_nanos" not in count_row + + +def test_ing25_scope_name_and_version(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + b.resource( + [b.profile([b.sample(["sc_leaf"], values=[1])], time_ns=now_ns() - 60 * NS)], + {"service.name": "svc-a"}, + scope_name="pyroscope", + scope_version="1.2", + ) + ingest_ok(client, prof_stream, b.model()) + row = wait_rows(client, prof_stream, 1)[0] + assert row["otel_scope_name"] == "pyroscope" + assert row["otel_scope_version"] == "1.2" + + +def _service_fallback(client: OpenObserveClient, stream: str, attrs: dict) -> str: + b = ProfilesBuilder() + b.resource([b.profile([b.sample(["fb_leaf"], values=[1])], time_ns=now_ns() - 60 * NS)], attrs) + ingest_ok(client, stream, b.model()) + return wait_rows(client, stream, 1)[0]["service_name"] + + +def test_ing26_missing_service_uses_executable(client: OpenObserveClient, prof_stream: str): + assert _service_fallback(client, prof_stream, {"process.executable.name": "mybin"}) == "unknown_service:mybin" + w = recent_window() + meta = query( + client, prof_stream, "meta", method="GET", params={"start_time": w["start_time"], "end_time": w["end_time"]} + ) + assert meta.status_code == 200, meta.text + assert meta.json()["services"] == ["unknown_service:mybin"] + + +def test_ing27_missing_service_no_executable(client: OpenObserveClient, prof_stream: str): + assert _service_fallback(client, prof_stream, {}) == "unknown_service" + + +def test_ing28_empty_service_name_uses_executable(client: OpenObserveClient, prof_stream: str): + attrs = {"service.name": "", "process.executable.name": "emptysvcbin"} + assert _service_fallback(client, prof_stream, attrs) == "unknown_service:emptysvcbin" + + +def test_ing29_deep_stack_truncated(client: OpenObserveClient, prof_stream: str): + row = _depth_rows(client, prof_stream, 1100) + frames = row["stack"].split(";") + assert row["frame_count"] == 1000 + assert len(frames) == 1000 + assert frames[0] == "deep_0000" + assert frames[-2:] == ["deep_0998", "[truncated]"] + + +def test_ing30_exactly_1000_frames_kept(client: OpenObserveClient, prof_stream: str): + row = _depth_rows(client, prof_stream, 1000) + assert row["frame_count"] == 1000 + assert row["stack"] == ";".join(_deep(1000)) + + +def test_ing31_1001_frames_truncated(client: OpenObserveClient, prof_stream: str): + row = _depth_rows(client, prof_stream, 1001) + assert row["frame_count"] == 1000 + assert row["stack"] == ";".join(_deep(999) + ["[truncated]"]) + + +def test_ing32_inlined_lines_become_frames(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + stack = b.stack_locs([b.loc_fn("inner", "outer"), b.loc_fn("main32")]) + ingest_ok(client, prof_stream, _one_profile(b, [b.sample(stack=stack, values=[4])])) + row = wait_rows(client, prof_stream, 1)[0] + assert row["stack"] == "inner;outer;main32" + assert row["frame_count"] == 3 + + +def test_ing33_function_without_name_uses_system_name(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + loc = b.loc_raw(lines=[{"function_index": b.func(None, "sys_fn")}]) + stack = b.stack_locs([loc, b.loc_fn("main33")]) + ingest_ok(client, prof_stream, _one_profile(b, [b.sample(stack=stack, values=[4])])) + assert wait_rows(client, prof_stream, 1)[0]["stack"] == "sys_fn;main33" + + +def test_ing34_unsymbolized_frames_file_and_build_id(client: OpenObserveClient, prof_stream: str): + ingest_ok(client, prof_stream, native_model()) + row = wait_rows(client, prof_stream, 1)[0] + assert row["stack"] == "libqa.so.1@deadbeefcafe+0x110;libqa.so.1@deadbeefcafe+0x120;a9_root" + root = merge_ok(client, prof_stream, recent_window(**cpu()))["root"] + native = child(child(root, "a9_root"), "[libqa.so.1]") + assert (native["self"], native["total"]) == (9, 9) + assert not native.get("children") + + +def test_ing35_unsymbolized_frames_with_frame_type(client: OpenObserveClient, prof_stream: str): + ingest_ok(client, prof_stream, native_model("native")) + row = wait_rows(client, prof_stream, 1)[0] + assert row["stack"] == "libqa.so.1@deadbeefcafe+0x110/native;libqa.so.1@deadbeefcafe+0x120/native;a9_root" + root = merge_ok(client, prof_stream, recent_window(**cpu()))["root"] + assert child(child(root, "a9_root"), "[libqa.so.1]")["self"] == 9 + + +def test_ing36_location_without_mapping_folds_to_unknown(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + stack = b.stack_locs([b.loc_raw(address=0xABC), b.loc_fn("root36")]) + ingest_ok(client, prof_stream, _one_profile(b, [b.sample(stack=stack, values=[6])])) + assert wait_rows(client, prof_stream, 1)[0]["stack"] == "0xabc;root36" + root = merge_ok(client, prof_stream, recent_window(**cpu()))["root"] + assert child(child(root, "root36"), "[unknown]")["self"] == 6 + + +def test_ing37_empty_stack_credits_root(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + ingest_ok(client, prof_stream, _one_profile(b, [b.sample(stack=0, values=[5]), b.sample(["es_leaf"], values=[10])])) + rows = by_stack(wait_rows(client, prof_stream, 2)) + assert (rows[""]["value"], rows[""]["frame_count"]) == (5, 0) + body = wait_merge_total(client, prof_stream, recent_window(**cpu()), 15) + assert body["root"]["self"] == 5 + assert [t["name"] for t in body["top"]] == ["es_leaf"] + + +def test_ing38_profile_without_samples_not_rejected(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + b.resource([b.profile([], time_ns=now_ns())], {"service.name": "svc-a"}) + resp = ingest_json(client, prof_stream, to_json(b.model())) + assert resp.status_code == 200, resp.text + assert resp.json() == {} + assert prof_stream not in profiles_streams(client) + + +def test_ing39_empty_request_bodies(client: OpenObserveClient, prof_stream: str): + for body in ({}, {"resourceProfiles": []}): + resp = ingest_json(client, prof_stream, body) + assert resp.status_code == 200, f"{body} -> {resp.text}" + assert resp.json() == {} + assert prof_stream not in profiles_streams(client) + + +def test_ing40_ingestion_passcode_auth(client: OpenObserveClient, prof_stream: str): + resp = client.get("passcode") + assert resp.status_code == 200, resp.text + data = resp.json()["data"] + body = json.dumps(to_json(simple_request(A_STACKS))) + resp = requests.post( + client.url("v1/profiles"), + data=body, + auth=(data["user"], data["passcode"]), + headers={"Content-Type": "application/json", "stream-name": prof_stream}, + timeout=30, + ) + assert resp.status_code == 200, resp.text + assert resp.json() == {} + wait_merge_total(client, prof_stream, recent_window(**cpu()), 400) + + +def test_ing41_no_auth(client: OpenObserveClient, prof_stream: str): + resp = requests.post( + client.url("v1/profiles"), + data=json.dumps(to_json(simple_request(A_STACKS))), + headers={"Content-Type": "application/json", "stream-name": prof_stream}, + timeout=30, + ) + assert resp.status_code == 401, resp.text + assert prof_stream not in profiles_streams(client) + + +def test_ing42_wrong_password(client: OpenObserveClient, prof_stream: str): + resp = requests.post( + client.url("v1/profiles"), + data=json.dumps(to_json(simple_request(A_STACKS))), + auth=(client.email, "wrong-password"), + headers={"Content-Type": "application/json", "stream-name": prof_stream}, + timeout=30, + ) + assert resp.status_code == 401, resp.text + assert prof_stream not in profiles_streams(client) + + +def test_ing43_values_sum_saturates(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + ingest_ok(client, prof_stream, _one_profile(b, [b.sample(["sat_leaf"], values=[9223372036854775807, 1])])) + assert wait_rows(client, prof_stream, 1)[0]["value"] == 9223372036854775807 + + +def test_ing44_negative_value_kept(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + ingest_ok(client, prof_stream, _one_profile(b, [b.sample(["neg_leaf"], values=[-5])])) + assert wait_rows(client, prof_stream, 1)[0]["value"] == -5 + + +# Product bug: profiles never apply ZO_COLS_PER_RECORD_LIMIT; strict xfail turns red once o2-enterprise#3123 fixes it, then drop the marker. +@pytest.mark.xfail(strict=True, reason="Known product bug o2-enterprise#3123: profiles ignore ZO_COLS_PER_RECORD_LIMIT") +def test_ing45_too_many_columns_rejected(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + attrs = {f"extra.attr_{i:04d}": f"v{i}" for i in range(1010)} + resp = ingest_json(client, prof_stream, to_json(_one_profile(b, [b.sample(["wide_leaf"], values=[1])], attrs))) + assert resp.status_code == 400, resp.text[:500] + assert resp.json()["code"] == 3 + assert "ZO_COLS_PER_RECORD_LIMIT" in resp.json()["message"] + + +def test_ing47_first_write_to_new_stream_succeeds(client: OpenObserveClient, prof_stream: str): + assert prof_stream not in profiles_streams(client) + resp = ingest_proto(client, prof_stream, simple_request(A_STACKS)) + assert resp.status_code == 200, f"first write to a new stream must not fail (#15956): {resp.text}" + wait_merge_total(client, prof_stream, recent_window(**cpu()), 400) + + +def test_ing48_two_profile_types_in_one_resource(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + t = now_ns() - 60 * NS + b.resource( + [ + b.profile(b.stacks_samples([(["cpu_leaf", "m_root"], 700)]), time_ns=t), + b.profile( + b.stacks_samples([(["alloc_leaf", "m_root"], 4096)]), type_="alloc_space", unit="bytes", time_ns=t + ), + ], + {"service.name": "svc-a"}, + ) + ingest_ok(client, prof_stream, b.model()) + w = recent_window() + wait_merge_total(client, prof_stream, recent_window(**cpu()), 700) + alloc = wait_merge_total(client, prof_stream, recent_window(profile_type="alloc_space", profile_unit="bytes"), 4096) + assert alloc["unit"] == "bytes" + assert [t["name"] for t in alloc["top"]] == ["alloc_leaf", "m_root"] + meta = query( + client, prof_stream, "meta", method="GET", params={"start_time": w["start_time"], "end_time": w["end_time"]} + ) + assert meta.status_code == 200, meta.text + assert meta.json()["profile_types"] == [ + {"type": "alloc_space", "unit": "bytes"}, + {"type": "cpu", "unit": "nanoseconds"}, + ] diff --git a/tests/api-testing/tests/profiles/test_profiles_limits.py b/tests/api-testing/tests/profiles/test_profiles_limits.py new file mode 100644 index 0000000000..7320b86e17 --- /dev/null +++ b/tests/api-testing/tests/profiles/test_profiles_limits.py @@ -0,0 +1,50 @@ +"""Profiles result-size limit: more than 99 999 unique stacks marks merge and diff `truncated` (feature §3.5, §3.7).""" + +from __future__ import annotations + +import pytest + +from support.client import OpenObserveClient + +from .profiles_helpers import ( + NS, + ProfilesBuilder, + child, + cpu, + ingest_proto, + now_ns, + query, + recent_window, + wait_merge_total, +) + +LIGHT_STACKS = 100_000 +HEAVY_VALUE = 1_000_000 +TOTAL = LIGHT_STACKS + HEAVY_VALUE +RESULT_SIZE = 99_999 + + +@pytest.mark.slow +def test_lim01_more_unique_stacks_than_result_size(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + samples = [b.sample([f"lim_{i:06d}", "lim_root"], values=[1]) for i in range(LIGHT_STACKS)] + samples.append(b.sample(["heavy_leaf", "lim_root"], values=[HEAVY_VALUE])) + b.resource([b.profile(samples, time_ns=now_ns() - 60 * NS)], {"service.name": "svc-lim"}) + resp = ingest_proto(client, prof_stream, b.model(), timeout=180) + assert resp.status_code == 200, resp.text[:500] + assert resp.content == b"" + + # A debug build needs ~35 s to merge 100k unique stacks. + body = wait_merge_total(client, prof_stream, recent_window(**cpu()), TOTAL, timeout=240, request_timeout=180) + assert body["truncated"] is True + assert body["total"] == TOTAL + assert body["merged_total"] == HEAVY_VALUE + RESULT_SIZE - 1 + assert child(child(body["root"], "lim_root"), "heavy_leaf")["self"] == HEAVY_VALUE + assert body["top"][0] == {"name": "heavy_leaf", "self": HEAVY_VALUE, "total": HEAVY_VALUE} + + side = recent_window(**cpu()) + resp = query(client, prof_stream, "diff", {"baseline": side, "comparison": side}, timeout=240) + assert resp.status_code == 200, resp.text[:500] + diff = resp.json() + assert diff["truncated"] is True + assert diff["baseline_total"] == diff["comparison_total"] == TOTAL diff --git a/tests/api-testing/tests/profiles/test_profiles_merge.py b/tests/api-testing/tests/profiles/test_profiles_merge.py new file mode 100644 index 0000000000..da1fb549c5 --- /dev/null +++ b/tests/api-testing/tests/profiles/test_profiles_merge.py @@ -0,0 +1,379 @@ +"""Profiles `merge` API: call tree, top functions, max_nodes cut, depth cap, native folding, filters, errors (feature §3.5).""" + +from __future__ import annotations + +import uuid + +import pytest +import requests + +from support.client import OpenObserveClient + +from .profiles_helpers import ( + ALL_CPU_TOTAL, + CATALOG_TOTAL, + CHECKOUT_TOTAL, + MIB, + NS, + PAYMENTS, + SPAN_ID, + TRACE_ID, + XSS, + ProfilesBuilder, + RefData, + assert_tree_sums, + child, + count_nodes, + cpu, + ingest_ok, + merge, + merge_ok, + native_model, + now_ns, + query, + recent_window, + simple_request, + snapshot_model, + wait_merge_total, + wait_rows, +) + +MAX_MAX_NODES = 16_384 +CHECKOUT_TOP = [ + {"name": "db_query", "self": 6_000_000_000, "total": 6_000_000_000}, + {"name": "json_encode", "self": 4_000_000_000, "total": 4_000_000_000}, + {"name": "handle_order", "self": 1_500_000_000, "total": 11_850_000_000}, + {"name": "process_payment", "self": 250_000_000, "total": 250_000_000}, + {"name": "gc_sweep", "self": 150_000_000, "total": 150_000_000}, + {"name": XSS, "self": 100_000_000, "total": 100_000_000}, + {"name": "main", "self": 0, "total": 11_850_000_000}, + {"name": "runtime_main", "self": 0, "total": 150_000_000}, +] + + +def _checkout(ref: RefData, **extra) -> dict: + return ref.win(**cpu(service_name="checkout", **extra)) + + +def _catalog(ref: RefData, **extra) -> dict: + return ref.win(**cpu(service_name="catalog", **extra)) + + +def _names(node: dict) -> list[str]: + return sorted(c["name"] for c in node.get("children") or []) + + +def _walk(node: dict): + yield node + for c in node.get("children") or []: + yield from _walk(c) + + +def _assert_400(resp: requests.Response, message: str) -> None: + assert resp.status_code == 400, f"{resp.status_code} {resp.text}" + assert resp.json()["message"] == message + + +def _ingest_stacks(client: OpenObserveClient, stream: str, stacks: list[tuple[list[str], int]], total: int) -> dict: + ingest_ok(client, stream, simple_request(stacks)) + return wait_merge_total(client, stream, recent_window(**cpu()), total) + + +def test_mrg01_checkout_totals(client: OpenObserveClient, ref: RefData): + body = merge_ok(client, ref.stream, _checkout(ref)) + assert body["total"] == body["merged_total"] == CHECKOUT_TOTAL + assert body["truncated"] is False + assert body["tree_truncated"] is False + assert (body["unit"], body["profile_type"]) == ("nanoseconds", "cpu") + + +def test_mrg02_tree_shape(client: OpenObserveClient, ref: RefData): + root = merge_ok(client, ref.stream, _checkout(ref))["root"] + assert (root["name"], root["total"], root["self"]) == ("total", CHECKOUT_TOTAL, 0) + assert _names(root) == ["main", "runtime_main"] + assert child(root, "main")["total"] == 11_850_000_000 + assert child(root, "runtime_main")["total"] == 150_000_000 + order = child(child(root, "main"), "handle_order") + assert (order["total"], order["self"]) == (11_850_000_000, 1_500_000_000) + totals = {c["name"]: c["total"] for c in order["children"]} + assert totals == { + "db_query": 6_000_000_000, + "json_encode": 4_000_000_000, + "process_payment": 250_000_000, + XSS: 100_000_000, + } + assert child(child(root, "runtime_main"), "gc_sweep")["self"] == 150_000_000 + assert_tree_sums(root) + + +def test_mrg03_top_exact_and_ordered(client: OpenObserveClient, ref: RefData): + top = merge_ok(client, ref.stream, _checkout(ref))["top"] + assert top == CHECKOUT_TOP + + +def test_mrg04_top_tie_break(client: OpenObserveClient, prof_stream: str): + body = _ingest_stacks(client, prof_stream, [(["a"], 5), (["b"], 5), (["bc", "b"], 4), (["c"], 5)], 19) + assert [(t["name"], t["self"], t["total"]) for t in body["top"]] == [ + ("b", 5, 9), + ("a", 5, 5), + ("c", 5, 5), + ("bc", 4, 4), + ] + + +def test_mrg05_recursion_counted_once(client: OpenObserveClient, prof_stream: str): + body = _ingest_stacks(client, prof_stream, [(["f", "g", "f", "main"], 10)], 10) + assert {t["name"]: (t["self"], t["total"]) for t in body["top"]} == {"f": (10, 10), "g": (0, 10), "main": (0, 10)} + + +def test_mrg06_get_equals_post(client: OpenObserveClient, ref: RefData): + post = merge_ok(client, ref.stream, _checkout(ref)) + resp = query(client, ref.stream, "merge", method="GET", params=_checkout(ref)) + assert resp.status_code == 200, resp.text + get = resp.json() + assert (get["root"], get["total"], get["top"]) == (post["root"], post["total"], post["top"]) + + +@pytest.mark.parametrize("ptype", list(PAYMENTS)) +def test_mrg07_payments_per_type(client: OpenObserveClient, ref: RefData, ptype: str): + total, unit = PAYMENTS[ptype] + body = merge_ok(client, ref.stream, ref.win(service_name="payments", profile_type=ptype, profile_unit=unit)) + assert (body["total"], body["unit"], body["profile_type"]) == (total, unit, ptype) + + +def test_mrg08_all_services_cpu(client: OpenObserveClient, ref: RefData): + body = merge_ok(client, ref.stream, ref.win(**cpu())) + assert body["total"] == ALL_CPU_TOTAL + assert _names(body["root"]) == ["[truncated]", "catalog_main", "main", "runtime_main"] + + +def test_mrg09_filters_are_anded(client: OpenObserveClient, ref: RefData): + v15 = {"key": "service_version", "value": "v1.5.0"} + assert merge_ok(client, ref.stream, _checkout(ref, filters=[v15]))["total"] == 8_000_000_000 + prod = {"key": "deployment_environment", "value": "prod"} + assert merge_ok(client, ref.stream, _checkout(ref, filters=[v15, prod]))["total"] == 7_850_000_000 + + +def test_mrg10_dotted_key_default_op(client: OpenObserveClient, ref: RefData): + body = merge_ok(client, ref.stream, _checkout(ref, filters=[{"key": "service.version", "value": "v1.4.0"}])) + assert body["total"] == 4_000_000_000 + + +def test_mrg11_get_filters_list(client: OpenObserveClient, ref: RefData): + params = _checkout(ref, filters="service_version=v1.4.0,deployment_environment=prod") + resp = query(client, ref.stream, "merge", method="GET", params=params) + assert resp.status_code == 200, resp.text + assert resp.json()["total"] == 4_000_000_000 + + +def test_mrg12_two_values_of_one_label_is_empty(client: OpenObserveClient, ref: RefData): + filters = [{"key": "service_version", "value": "v1.4.0"}, {"key": "service_version", "value": "v1.5.0"}] + body = merge_ok(client, ref.stream, _checkout(ref, filters=filters)) + assert (body["total"], body["top"]) == (0, []) + + +def test_mrg13_trace_and_span_filter(client: OpenObserveClient, ref: RefData): + filters = [{"key": "trace_id", "value": TRACE_ID}, {"key": "span_id", "value": SPAN_ID}] + body = merge_ok(client, ref.stream, _checkout(ref, filters=filters)) + assert body["total"] == 250_000_000 + node, path = body["root"], [] + while node.get("children"): + assert len(node["children"]) == 1, node["children"] + node = node["children"][0] + path.append(node["name"]) + assert path == ["main", "handle_order", "process_payment"] + assert node["self"] == 250_000_000 + + +def test_mrg14_quoted_filter_value_is_safe(client: OpenObserveClient, ref: RefData): + body = merge_ok(client, ref.stream, _checkout(ref, filters=[{"key": "service_version", "value": "v1' OR '1'='1"}])) + assert (body["total"], body["top"]) == (0, []) + + +def test_mrg15_max_nodes_50(client: OpenObserveClient, ref: RefData): + body = merge_ok(client, ref.stream, _catalog(ref, max_nodes=50)) + assert count_nodes(body["root"]) == 50 + assert (body["tree_truncated"], body["truncated"], body["total"]) == (True, False, CATALOG_TOTAL) + others = [n for n in _walk(body["root"]) if n["name"] == "other"] + assert others, "a cut tree folds dropped siblings into an `other` node" + for node in _walk(body["root"]): + assert [c["name"] for c in node.get("children") or []].count("other") <= 1 + for node in others: + assert node["self"] == node["total"] + assert not node.get("children") + assert_tree_sums(body["root"]) + + +def test_mrg16_max_nodes_3(client: OpenObserveClient, ref: RefData): + body = merge_ok(client, ref.stream, _catalog(ref, max_nodes=3)) + # Both root children have children, and keeping one also reserves a slot for its own `other`: 4 > 3. + assert [(c["name"], c["total"]) for c in body["root"]["children"]] == [("other", CATALOG_TOTAL)] + assert (body["tree_truncated"], body["total"]) == (True, CATALOG_TOTAL) + assert_tree_sums(body["root"]) + + +@pytest.mark.parametrize("max_nodes", [0, 1]) +def test_mrg17_max_nodes_clamped_to_minimal_tree(client: OpenObserveClient, ref: RefData, max_nodes: int): + body = merge_ok(client, ref.stream, _catalog(ref, max_nodes=max_nodes)) + assert body["tree_truncated"] is True + assert [(c["name"], c["total"]) for c in body["root"]["children"]] == [("other", CATALOG_TOTAL)] + + +def test_mrg18_max_nodes_clamped_to_maximum(client: OpenObserveClient, prof_stream: str): + body = _ingest_stacks(client, prof_stream, [([f"mx_{i:05d}", "mx_root"], 1) for i in range(20_000)], 20_000) + assert count_nodes(body["root"]) == 2048 + clamped = merge_ok(client, prof_stream, recent_window(**cpu(max_nodes=999_999))) + assert count_nodes(clamped["root"]) == MAX_MAX_NODES + assert clamped["tree_truncated"] is True + assert clamped["total"] == 20_000 + assert_tree_sums(clamped["root"]) + + +def test_mrg18_full_tree_under_maximum_not_cut(client: OpenObserveClient, ref: RefData): + body = merge_ok(client, ref.stream, _catalog(ref, max_nodes=999_999)) + assert count_nodes(body["root"]) == 1303 + assert body["tree_truncated"] is False + + +def test_mrg19_default_max_nodes_2048(client: OpenObserveClient, prof_stream: str): + body = _ingest_stacks(client, prof_stream, [([f"wide_fn_{i:04d}", "wide_root"], 1) for i in range(3000)], 3000) + assert count_nodes(body["root"]) == 2048 + assert body["tree_truncated"] is True + assert len(body["top"]) == 3001 + + +def test_mrg20_deep_stack_chain(client: OpenObserveClient, ref: RefData): + body = merge_ok(client, ref.stream, _catalog(ref)) + node = child(body["root"], "[truncated]") + assert node["total"] == 1_000_000_000 + for i in range(998, -1, -1): + node = child(node, f"deep_{i:04d}") + assert (node["self"], node.get("children")) == (1_000_000_000, None) + assert body["top"][0] == {"name": "deep_0000", "self": 1_000_000_000, "total": 1_000_000_000} + assert "[truncated]" not in {t["name"] for t in body["top"]} + + +def test_mrg21_wide_set(client: OpenObserveClient, ref: RefData): + body = merge_ok(client, ref.stream, _catalog(ref)) + wide = child(child(body["root"], "catalog_main"), "wide_root") + assert len(wide["children"]) == 300 + assert wide["total"] == 4_515_000_000 + assert len(body["top"]) == 1301 + + +def test_mrg22_native_frames_folded(client: OpenObserveClient, prof_stream: str): + ingest_ok(client, prof_stream, native_model()) + body = wait_merge_total(client, prof_stream, recent_window(**cpu()), 9) + folded = child(child(body["root"], "a9_root"), "[libqa.so.1]") + assert (folded["self"], folded.get("children")) == (9, None) + assert body["top"] == [{"name": "[libqa.so.1]", "self": 9, "total": 9}, {"name": "a9_root", "self": 0, "total": 9}] + + +def test_mrg23_native_frames_from_two_binaries(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + lib_a = b.mapping("/opt/libA.so", 0x1000) + lib_b = b.mapping("/opt/libB.so", 0x9000) + locs = [b.loc_raw(mapping_index=lib_a, address=0x1100), b.loc_raw(mapping_index=lib_b, address=0x9100)] + stack = b.stack_locs([*locs, b.loc_fn("root23")]) + b.resource([b.profile([b.sample(stack=stack, values=[4])], time_ns=now_ns() - 60 * NS)], {"service.name": "svc-a"}) + ingest_ok(client, prof_stream, b.model()) + body = wait_merge_total(client, prof_stream, recent_window(**cpu()), 4) + lib_b_node = child(child(body["root"], "root23"), "[libB.so]") + assert child(lib_b_node, "[libA.so]")["self"] == 4 + + +def test_mrg25_zero_value_stacks_skipped(client: OpenObserveClient, prof_stream: str): + body = _ingest_stacks(client, prof_stream, [(["zero_leaf", "zr"], 0), (["five_leaf", "zr"], 5)], 5) + assert _names(child(body["root"], "zr")) == ["five_leaf"] + assert {t["name"] for t in body["top"]} == {"five_leaf", "zr"} + + +def test_mrg26_empty_stack_credited_to_root(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + b.resource( + [b.profile([b.sample(stack=0, values=[5]), b.sample(["es_leaf"], values=[10])], time_ns=now_ns() - 60 * NS)], + {"service.name": "svc-a"}, + ) + ingest_ok(client, prof_stream, b.model()) + body = wait_merge_total(client, prof_stream, recent_window(**cpu()), 15) + assert body["root"]["self"] == 5 + assert _names(body["root"]) == ["es_leaf"] + + +def test_mrg27_inuse_snapshots_averaged(client: OpenObserveClient, prof_stream: str): + minute = (now_ns() - 3 * 60 * NS) // (60 * NS) * 60 * NS + ingest_ok(client, prof_stream, snapshot_model(minute)) + wait_rows(client, prof_stream, 2) + body = merge_ok(client, prof_stream, recent_window(profile_type="inuse_space", profile_unit="bytes")) + assert (body["total"], body["unit"]) == (2 * MIB, "bytes") + + +def test_mrg28_empty_window(client: OpenObserveClient, ref: RefData, empty_window: dict): + body = merge_ok(client, ref.stream, {**empty_window, **cpu(service_name="checkout")}) + assert (body["total"], body["top"]) == (0, []) + assert "children" not in body["root"] + + +MRG29_CASES = [ + ("type", {"profile_type": None}, "profile_type is required"), + ("unit", {"profile_unit": None}, "profile_unit is required"), + ("start-eq-end", {"start_time": "END"}, "start_time must be less than end_time"), + ("no-start", {"start_time": None}, "start_time and end_time are required (microseconds)"), + ("op", {"filters": [{"key": "host_name", "op": "!=", "value": "x"}]}, "only filter op '=' is supported"), + ("key-stack", {"filters": [{"key": "stack", "value": "x"}]}, "invalid filter key: stack"), + ("key-unsafe", {"filters": [{"key": "a;b", "value": "x"}]}, "invalid filter key: a;b"), + ("key-129", {"filters": [{"key": "a" * 129, "value": "x"}]}, f"invalid filter key: {'a' * 129}"), + ("timeout", {"timeout": 0}, "Invalid timeout parameter"), +] + + +@pytest.mark.parametrize( + ("change", "message"), [c[1:] for c in MRG29_CASES], ids=[f"mrg29-{c[0]}" for c in MRG29_CASES] +) +def test_mrg29_validation_errors(client: OpenObserveClient, ref: RefData, change: dict, message: str): + body = _checkout(ref) + for key, value in change.items(): + if value is None: + del body[key] + else: + body[key] = body["end_time"] if value == "END" else value + _assert_400(merge(client, ref.stream, body), message) + + +def test_mrg29_unknown_filter_column(client: OpenObserveClient, ref: RefData): + resp = merge(client, ref.stream, _checkout(ref, filters=[{"key": "no_such_col", "value": "x"}])) + assert resp.status_code == 400, resp.text + assert resp.json()["code"] == 20004 + assert "unknown field 'no_such_col'" in resp.json()["message"] + + +def test_mrg29_malformed_json(client: OpenObserveClient, ref: RefData): + resp = client.post( + f"{ref.stream}/profiles/merge", data='{"start_time":', headers={"Content-Type": "application/json"} + ) + assert resp.status_code == 400, resp.text + assert resp.json()["message"].startswith("Failed to parse the request body as JSON"), resp.text + + +def test_mrg30_negative_max_nodes(client: OpenObserveClient, ref: RefData): + resp = merge(client, ref.stream, _checkout(ref, max_nodes=-1)) + assert resp.status_code == 400, resp.text + assert resp.json()["message"].startswith("Failed to deserialize the JSON body"), resp.text + + +def test_mrg32_unknown_stream(client: OpenObserveClient, ref: RefData): + stream = f"no_such_{uuid.uuid4().hex[:8]}" + resp = merge(client, stream, _checkout(ref)) + assert resp.status_code == 400, resp.text + assert resp.json()["code"] == 20002 + assert resp.json()["message"] == f"Search stream not found: {stream}" + + +def test_mrg33_wrong_method(client: OpenObserveClient, ref: RefData): + resp = query(client, ref.stream, "merge", _checkout(ref), method="PUT") + assert resp.status_code == 405, resp.text + + +def test_mrg34_no_auth(client: OpenObserveClient, ref: RefData): + resp = requests.post(client.url(f"{ref.stream}/profiles/merge"), json=_checkout(ref), timeout=30) + assert resp.status_code == 401, resp.text diff --git a/tests/api-testing/tests/profiles/test_profiles_meta_tag_values.py b/tests/api-testing/tests/profiles/test_profiles_meta_tag_values.py new file mode 100644 index 0000000000..dd277b9257 --- /dev/null +++ b/tests/api-testing/tests/profiles/test_profiles_meta_tag_values.py @@ -0,0 +1,261 @@ +"""Profiles `meta` and `tag_values` query APIs on the reference dataset (feature §3.2, §3.3).""" + +from __future__ import annotations + +import uuid + +import pytest +import requests + +from support.client import OpenObserveClient + +from .profiles_helpers import ( + MIN_US, + TRACE_ID, + US, + ProfilesBuilder, + RefData, + ingest_ok, + now_ns, + query, + recent_window, + wait_rows, +) + +SERVICES = ["catalog", "checkout", "payments"] +ALL_TYPES = [ + {"type": "alloc_space", "unit": "bytes"}, + {"type": "cpu", "unit": "nanoseconds"}, + {"type": "inuse_space", "unit": "bytes"}, +] +CPU_ONLY = [{"type": "cpu", "unit": "nanoseconds"}] +EXPECTED_LABELS = { + "deployment_environment", + "host_name", + "service_instance_id", + "service_version", + "span_id", + "trace_id", +} +NOT_LABELS = { + "stack", + "value", + "service_name", + "profile_type", + "profile_unit", + "frame_count", + "extractor_version", + "profile_duration_nanos", + "duration_nanos", + "_timestamp", +} + + +def _params(w: dict, **extra) -> dict: + return {"start_time": w["start_time"], "end_time": w["end_time"], **extra} + + +def _meta(client: OpenObserveClient, stream: str, params: dict) -> requests.Response: + return query(client, stream, "meta", method="GET", params=params) + + +def _meta_ok(client: OpenObserveClient, stream: str, params: dict) -> dict: + resp = _meta(client, stream, params) + assert resp.status_code == 200, resp.text + return resp.json() + + +def _tags(client: OpenObserveClient, stream: str, params: dict) -> requests.Response: + return query(client, stream, "tag_values", method="GET", params=params) + + +def _tag_values(client: OpenObserveClient, ref: RefData, **params) -> dict: + resp = _tags(client, ref.stream, _params(ref.win(), **params)) + assert resp.status_code == 200, resp.text + return resp.json() + + +def _assert_400(resp: requests.Response, message: str) -> None: + assert resp.status_code == 400, f"{resp.status_code} {resp.text}" + assert resp.json()["message"] == message + + +def test_meta01_lists_services_types_labels(client: OpenObserveClient, ref: RefData): + body = _meta_ok(client, ref.stream, _params(ref.win())) + assert body["services"] == SERVICES + assert body["profile_types"] == ALL_TYPES + assert body["data_sources"] == [] + labels = set(body["label_names"]) + assert EXPECTED_LABELS <= labels, labels + assert not labels & NOT_LABELS, labels & NOT_LABELS + assert body["label_names"] == sorted(body["label_names"]) + + +@pytest.mark.parametrize(("service", "types"), [("checkout", CPU_ONLY), ("payments", ALL_TYPES), ("catalog", CPU_ONLY)]) +def test_meta02_types_scoped_to_service(client: OpenObserveClient, ref: RefData, service: str, types: list): + body = _meta_ok(client, ref.stream, _params(ref.win(), service_name=service)) + assert body["profile_types"] == types + assert body["services"] == SERVICES + + +def test_meta03_unknown_service(client: OpenObserveClient, ref: RefData): + body = _meta_ok(client, ref.stream, _params(ref.win(), service_name="nope")) + assert body["profile_types"] == [] + assert body["services"] == SERVICES + + +def test_meta04_empty_window_keeps_schema_labels(client: OpenObserveClient, ref: RefData, empty_window: dict): + full = _meta_ok(client, ref.stream, _params(ref.win())) + body = _meta_ok(client, ref.stream, _params(empty_window)) + assert body["services"] == [] + assert body["profile_types"] == [] + assert body["label_names"] == full["label_names"] + + +def test_meta05_narrow_window(client: OpenObserveClient, ref: RefData): + c2 = ref.t0_us + 4 * MIN_US + body = _meta_ok(client, ref.stream, {"start_time": c2 - US, "end_time": c2 + 30 * US}) + assert body["services"] == ["checkout"] + assert body["profile_types"] == CPU_ONLY + + +@pytest.mark.parametrize( + ("params", "message"), + [ + ({"end_time": 2_000_000}, "start_time is required"), + ({"start_time": 1_000_000}, "end_time is required"), + ({"start_time": "abc", "end_time": 2_000_000}, "Invalid start_time parameter"), + ({"start_time": 2_000_000, "end_time": 2_000_000}, "start_time must be less than end_time"), + ({"start_time": 3_000_000, "end_time": 2_000_000}, "start_time must be less than end_time"), + ({"start_time": 0, "end_time": 2_000_000}, "start_time and end_time are required (microseconds)"), + ], + ids=["meta06-no-start", "meta07-no-end", "meta08-bad-start", "meta09-equal", "meta10-reversed", "meta11-zero"], +) +def test_meta06_11_time_errors(client: OpenObserveClient, ref: RefData, params: dict, message: str): + _assert_400(_meta(client, ref.stream, params), message) + + +def test_meta12_timeout_zero(client: OpenObserveClient, ref: RefData): + _assert_400(_meta(client, ref.stream, _params(ref.win(), timeout=0)), "Invalid timeout parameter") + + +def test_meta13_unknown_stream_is_empty(client: OpenObserveClient, ref: RefData): + body = _meta_ok(client, f"no_such_{uuid.uuid4().hex[:8]}", _params(ref.win())) + assert (body["services"], body["profile_types"], body["label_names"], body["data_sources"]) == ([], [], [], []) + + +def test_meta15_no_auth(client: OpenObserveClient, ref: RefData): + resp = requests.get(client.url(f"{ref.stream}/profiles/meta"), params=_params(ref.win()), timeout=30) + assert resp.status_code == 401, resp.text + + +def test_tag01_values_of_service_version(client: OpenObserveClient, ref: RefData): + body = _tag_values(client, ref, tag="service_version") + assert body["tag"] == "service_version" + assert body["values"] == ["v1.4.0", "v1.5.0"] + + +def test_tag02_narrowed_by_other_filter(client: OpenObserveClient, ref: RefData): + body = _tag_values(client, ref, tag="deployment_environment", filters="service_version=v1.4.0") + assert body["values"] == ["prod"] + + +def test_tag03_own_tag_filter_ignored(client: OpenObserveClient, ref: RefData): + assert _tag_values(client, ref, tag="service_version", filters="service_version=v1.4.0")["values"] == [ + "v1.4.0", + "v1.5.0", + ] + + +def test_tag04_dotted_tag_normalised(client: OpenObserveClient, ref: RefData): + body = _tag_values(client, ref, tag="service.version") + assert body["tag"] == "service_version" + assert body["values"] == ["v1.4.0", "v1.5.0"] + + +def test_tag05_field_alias(client: OpenObserveClient, ref: RefData): + body = _tag_values(client, ref, field="host_name") + assert body["tag"] == "host_name" + assert body["values"] == [f"host-{c}" for c in "abcdef"] + + +def test_tag06_narrowed_by_service_and_type(client: OpenObserveClient, ref: RefData): + body = _tag_values( + client, ref, tag="host_name", service_name="payments", profile_type="inuse_space", profile_unit="bytes" + ) + assert body["values"] == ["host-d", "host-e"] + + +@pytest.mark.parametrize( + "tag", + ["service_name", "profile_type", "otel_scope_name"], + ids=["tag07-service_name", "tag08-profile_type", "tag08-otel_scope_name"], +) +def test_tag07_08_meta_dimensions_rejected(client: OpenObserveClient, ref: RefData, tag: str): + _assert_400( + _tags(client, ref.stream, _params(ref.win(), tag=tag)), + f"tag {tag} is provided by /profiles/meta; use tag_values only for Tags Filter keys", + ) + + +def test_tag09_missing_tag(client: OpenObserveClient, ref: RefData): + _assert_400(_tags(client, ref.stream, _params(ref.win())), "tag is required") + + +@pytest.mark.parametrize( + "tag", + ["stack", "value", "_timestamp", "a;b", "a" * 129], + ids=["tag10-stack", "tag10-value", "tag10-_timestamp", "tag11-semicolon", "tag11-129-chars"], +) +def test_tag10_11_reserved_or_unsafe_tag(client: OpenObserveClient, ref: RefData, tag: str): + _assert_400( + _tags(client, ref.stream, _params(ref.win(), tag=tag)), + f"tag must be a filterable profile tag column, got: {tag}", + ) + + +def test_tag12_unknown_column(client: OpenObserveClient, ref: RefData): + resp = _tags(client, ref.stream, _params(ref.win(), tag="no_such_col")) + assert resp.status_code == 400, resp.text + body = resp.json() + assert body["code"] == 20004, body + assert "unknown field 'no_such_col'" in body["message"], body + + +def test_tag13_missing_start_time(client: OpenObserveClient, ref: RefData): + _assert_400( + _tags(client, ref.stream, {"tag": "host_name", "end_time": ref.win()["end_time"]}), "start_time is required" + ) + + +def test_tag14_bad_filter_key(client: OpenObserveClient, ref: RefData): + _assert_400( + _tags(client, ref.stream, _params(ref.win(), tag="host_name", filters="stack=x")), "invalid filter key: stack" + ) + + +def test_tag15_unknown_stream(client: OpenObserveClient, ref: RefData): + stream = f"no_such_{uuid.uuid4().hex[:8]}" + resp = _tags(client, stream, _params(ref.win(), tag="host_name")) + assert resp.status_code == 400, resp.text + assert resp.json()["code"] == 20002 + assert resp.json()["message"] == f"Search stream not found: {stream}" + + +def test_tag16_trace_id_values(client: OpenObserveClient, ref: RefData): + assert _tag_values(client, ref, tag="trace_id")["values"] == [TRACE_ID] + + +def test_tag17_percent_encoded_filter_value(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + t = now_ns() - 60_000_000_000 + for owner, host in (("a,b=c%d", "host-enc"), ("plain", "host-plain")): + b.resource( + [b.profile([b.sample([f"{host}_leaf"], values=[1])], time_ns=t)], + {"service.name": "svc-enc", "team.owner": owner, "host.name": host}, + ) + ingest_ok(client, prof_stream, b.model()) + wait_rows(client, prof_stream, 2) + resp = _tags(client, prof_stream, _params(recent_window(), tag="host_name", filters="team_owner=a%2Cb%3Dc%25d")) + assert resp.status_code == 200, resp.text + assert resp.json()["values"] == ["host-enc"] diff --git a/tests/api-testing/tests/profiles/test_profiles_series.py b/tests/api-testing/tests/profiles/test_profiles_series.py new file mode 100644 index 0000000000..52472a3b27 --- /dev/null +++ b/tests/api-testing/tests/profiles/test_profiles_series.py @@ -0,0 +1,361 @@ +"""Profiles `series` API: buckets, per-instance rate, inuse averaging, group_by and step rules (feature §3.4).""" + +from __future__ import annotations + +import uuid + +import pytest +import requests + +from support.client import OpenObserveClient + +from .profiles_helpers import ( + CHECKOUT_TOTAL, + MIB, + NS, + SPAN_ID, + TRACE_ID, + US, + ProfilesBuilder, + RefData, + cpu, + ingest_ok, + merge_ok, + now_ns, + now_us, + query, + recent_window, + snapshot_model, + wait_merge_total, + wait_rows, + window, +) + +INUSE = {"profile_type": "inuse_space", "profile_unit": "bytes"} + + +def _series(client: OpenObserveClient, stream: str, body: dict) -> requests.Response: + return query(client, stream, "series", body) + + +def _series_ok(client: OpenObserveClient, stream: str, body: dict) -> dict: + resp = _series(client, stream, body) + assert resp.status_code == 200, f"series {body} -> {resp.status_code} {resp.text[:500]}" + return resp.json() + + +def _points(body: dict) -> list[tuple[int, float]]: + return [(p["timestamp"], p["value"]) for p in body["series"]] + + +def _assert_400(resp: requests.Response, message: str) -> None: + assert resp.status_code == 400, f"{resp.status_code} {resp.text}" + assert resp.json()["message"] == message + + +def _checkout(ref: RefData, **extra) -> dict: + return ref.win(**cpu(**{"service_name": "checkout", "step": "60", **extra})) + + +def _snapshot_stream(client: OpenObserveClient, stream: str, threads: tuple[str, str] | None = None) -> int: + minute = (now_ns() - 3 * 60 * NS) // (60 * NS) * 60 * NS + ingest_ok(client, stream, snapshot_model(minute, threads)) + return minute // 1000 + + +def test_ser01_checkout_cpu_step_60(client: OpenObserveClient, ref: RefData): + body = _series_ok(client, ref.stream, _checkout(ref)) + assert (body["unit"], body["profile_type"], body["step_secs"]) == ("nanoseconds", "cpu", 60) + assert _points(body) == [(ref.bucket(0), 4.0e9), (ref.bucket(4), 7.85e9), (ref.bucket(6), 0.15e9)] + assert "groups" not in body + merged = merge_ok(client, ref.stream, ref.win(**cpu(service_name="checkout"))) + assert sum(v for _, v in _points(body)) == merged["total"] == CHECKOUT_TOTAL + + +def test_ser02_rate_per_second_of_profiled_time(client: OpenObserveClient, ref: RefData): + rates = [p["rate"] for p in _series_ok(client, ref.stream, _checkout(ref))["series"]] + assert rates == pytest.approx([4.0e8, 7.85e8, 1.5e7], rel=1e-9) + + +def test_ser03_get_equals_post(client: OpenObserveClient, ref: RefData): + post = _series_ok(client, ref.stream, _checkout(ref)) + resp = query(client, ref.stream, "series", method="GET", params=_checkout(ref)) + assert resp.status_code == 200, resp.text + assert resp.json()["series"] == post["series"] + assert resp.json()["step_secs"] == 60 + + +def test_ser04_query_params_fill_post_body(client: OpenObserveClient, ref: RefData): + body = _checkout(ref) + del body["profile_unit"] + resp = query(client, ref.stream, "series", body, params={"profile_unit": "nanoseconds"}) + assert resp.status_code == 200, resp.text + assert resp.json()["unit"] == "nanoseconds" + assert resp.json()["series"] == _series_ok(client, ref.stream, _checkout(ref))["series"] + + +def test_ser05_process_level_filter_keeps_rate(client: OpenObserveClient, ref: RefData): + body = _series_ok(client, ref.stream, _checkout(ref, filters=[{"key": "service_version", "value": "v1.5.0"}])) + assert _points(body) == [(ref.bucket(4), 7.85e9), (ref.bucket(6), 0.15e9)] + assert [p["rate"] for p in body["series"]] == pytest.approx([7.85e8, 1.5e7], rel=1e-9) + + +def test_ser06_trace_filter_drops_rate(client: OpenObserveClient, ref: RefData): + filters = [{"key": "trace_id", "value": TRACE_ID}, {"key": "span_id", "value": SPAN_ID}] + body = _series_ok(client, ref.stream, _checkout(ref, filters=filters)) + assert _points(body) == [(ref.bucket(4), 2.5e8)] + assert "rate" not in body["series"][0] + + +def test_ser07_inuse_per_bucket_no_rate(client: OpenObserveClient, ref: RefData): + body = _series_ok(client, ref.stream, ref.win(service_name="payments", step="60", **INUSE)) + assert body["unit"] == "bytes" + assert _points(body) == [(ref.bucket(2), 2 * MIB), (ref.bucket(5), 4 * MIB)] + assert all("rate" not in p for p in body["series"]) + + +def test_ser08_inuse_two_instances_in_one_bucket_summed(client: OpenObserveClient, ref: RefData): + t2, t5 = ref.t0_us + 2 * 60 * US, ref.t0_us + 5 * 60 * US + # Buckets are epoch-aligned, so pick a step with no bucket edge between the two profiles (e.g. at UTC midnight). + steps = [s for s in range(600, 86_400) if t2 // (s * US) == t5 // (s * US)] + assert steps, f"no step in 600..86399 s keeps {t2} and {t5} in one bucket" + step = steps[0] + body = _series_ok(client, ref.stream, ref.win(service_name="payments", step=str(step), **INUSE)) + assert [v for _, v in _points(body)] == [6 * MIB] + + +def test_ser09_inuse_snapshots_of_one_instance_averaged(client: OpenObserveClient, prof_stream: str): + bucket = _snapshot_stream(client, prof_stream) + wait_rows(client, prof_stream, 2) + body = _series_ok(client, prof_stream, recent_window(step="60", **INUSE)) + assert _points(body) == [(bucket, 2 * MIB)] + + +def test_ser19_inuse_group_by_uses_ungrouped_snapshot_count(client: OpenObserveClient, prof_stream: str): + bucket = _snapshot_stream(client, prof_stream, threads=("A", "B")) + wait_rows(client, prof_stream, 2) + body = _series_ok(client, prof_stream, recent_window(step="60", group_by="thread_name", **INUSE)) + assert _points(body) == [(bucket, 2 * MIB)] + groups = {g["value"]: g for g in body["groups"]} + assert set(groups) == {"A", "B"} + assert groups["A"]["total"] == MIB / 2 + assert groups["B"]["total"] == 3 * MIB / 2 + + +def test_ser10_group_by_service_version(client: OpenObserveClient, ref: RefData): + body = _series_ok(client, ref.stream, _checkout(ref, group_by="service_version")) + assert [(g["value"], g["total"], g["others"]) for g in body["groups"]] == [ + ("v1.5.0", 8.0e9, False), + ("v1.4.0", 4.0e9, False), + ] + for g in body["groups"]: + assert sum(p["value"] for p in g["series"]) == g["total"] + assert sum(v for _, v in _points(body)) == CHECKOUT_TOTAL + + +def test_ser11_group_by_dotted_key(client: OpenObserveClient, ref: RefData): + dotted = _series_ok(client, ref.stream, _checkout(ref, group_by="service.version")) + plain = _series_ok(client, ref.stream, _checkout(ref, group_by="service_version")) + assert dotted["groups"] == plain["groups"] + + +@pytest.mark.parametrize("limit", [1, 0], ids=["ser12-limit-1", "ser13-limit-0-clamps"]) +def test_ser12_13_group_limit_folds_others(client: OpenObserveClient, ref: RefData, limit: int): + body = _series_ok(client, ref.stream, ref.win(**cpu(step="60", group_by="service_name", group_limit=limit))) + assert [(g["value"], g["total"], g["others"]) for g in body["groups"]] == [ + ("checkout", 12.0e9, False), + ("", 9.015e9, True), + ] + + +def test_ser14_group_limit_default_and_max(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + t = now_ns() - 60 * NS + for i in range(52): + b.resource( + [b.profile([b.sample(["gl_leaf"], values=[(i + 1) * 10])], time_ns=t)], + {"service.name": "svc-gl", "service.version": f"v{i:02d}", "service.instance.id": f"gl-{i}"}, + ) + ingest_ok(client, prof_stream, b.model()) + wait_merge_total(client, prof_stream, recent_window(**cpu()), 13_780) + default = _series_ok(client, prof_stream, recent_window(**cpu(group_by="service_version"))) + assert [g["value"] for g in default["groups"]] == [f"v{i:02d}" for i in range(51, 41, -1)] + [""] + assert (default["groups"][-1]["others"], default["groups"][-1]["total"]) == (True, 9030.0) + capped = _series_ok(client, prof_stream, recent_window(**cpu(group_by="service_version", group_limit=51))) + assert [g["value"] for g in capped["groups"]] == [f"v{i:02d}" for i in range(51, 1, -1)] + [""] + assert [g["others"] for g in capped["groups"]] == [False] * 50 + [True] + assert capped["groups"][-1]["total"] == 30.0 + + +def test_ser15_rows_missing_label_form_empty_group(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + t = now_ns() - 60 * NS + b.resource( + [b.profile([b.sample(["pod_leaf"], values=[5])], time_ns=t)], {"service.name": "svc-p", "k8s.pod.name": "pod-1"} + ) + b.resource([b.profile([b.sample(["nopod_leaf"], values=[7])], time_ns=t)], {"service.name": "svc-p"}) + ingest_ok(client, prof_stream, b.model()) + wait_merge_total(client, prof_stream, recent_window(**cpu()), 12) + body = _series_ok(client, prof_stream, recent_window(**cpu(group_by="k8s_pod_name"))) + assert [(g["value"], g["total"], g["others"]) for g in body["groups"]] == [("", 7.0, False), ("pod-1", 5.0, False)] + + +def test_ser16_group_ties_ordered_by_value(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + t = now_ns() - 60 * NS + for version in ("v-b", "v-a"): + b.resource( + [b.profile([b.sample(["tie_leaf"], values=[9])], time_ns=t)], + {"service.name": "svc-t", "service.version": version, "service.instance.id": version}, + ) + ingest_ok(client, prof_stream, b.model()) + wait_merge_total(client, prof_stream, recent_window(**cpu()), 18) + body = _series_ok(client, prof_stream, recent_window(**cpu(group_by="service_version"))) + assert [(g["value"], g["total"]) for g in body["groups"]] == [("v-a", 9.0), ("v-b", 9.0)] + + +def test_ser17_group_by_unknown_column(client: OpenObserveClient, ref: RefData): + _assert_400( + _series(client, ref.stream, _checkout(ref, group_by="nonexistent_col")), + "unknown group_by column: nonexistent_col", + ) + + +@pytest.mark.parametrize("key", ["a;b", "stack"], ids=["ser18-unsafe", "ser18-reserved"]) +def test_ser18_group_by_unsafe_or_reserved(client: OpenObserveClient, ref: RefData, key: str): + _assert_400(_series(client, ref.stream, _checkout(ref, group_by=key)), f"invalid group_by: {key}") + + +@pytest.mark.parametrize(("step", "secs"), [("60", 60), ("1 minute", 60), ("10 seconds", 10)]) +def test_ser20_explicit_step_forms(client: OpenObserveClient, ref: RefData, step: str, secs: int): + body = _series_ok(client, ref.stream, _checkout(ref, step=step)) + assert body["step_secs"] == secs + assert sum(v for _, v in _points(body)) == CHECKOUT_TOTAL + + +def test_ser20_auto_step_forms_agree(client: OpenObserveClient, ref: RefData): + base = ref.win(**cpu(service_name="checkout")) + auto = _series_ok(client, ref.stream, {**base, "step": "auto"}) + assert auto["step_secs"] > 0 + for variant in ({**base, "step": ""}, base): + body = _series_ok(client, ref.stream, variant) + assert (body["step_secs"], body["series"]) == (auto["step_secs"], auto["series"]) + + +@pytest.mark.parametrize( + ("step", "message"), + [("bad", "Invalid step: bad"), ("0", "step must be positive"), ("-5", "Invalid step: -5")], + ids=["ser21-bad", "ser22-zero", "ser23-negative"], +) +def test_ser21_23_invalid_step(client: OpenObserveClient, ref: RefData, step: str, message: str): + _assert_400(_series(client, ref.stream, _checkout(ref, step=step)), message) + + +def _bucket_cap_message(n: int) -> str: + return ( + f"step too fine for time range: estimated {n} histogram buckets exceeds max 99999; " + "increase step or narrow the time range" + ) + + +def test_ser24_bucket_cap(client: OpenObserveClient, ref: RefData): + end = now_us() + resp = _series(client, ref.stream, window(end - 7 * 24 * 3600 * US, end, **cpu(step="1"))) + _assert_400(resp, _bucket_cap_message(604800)) + + +def test_ser25_bucket_cap_boundary(client: OpenObserveClient, ref: RefData): + end = now_us() + ok = _series_ok(client, ref.stream, window(end - 99_999 * US, end, **cpu(step="1", service_name="checkout"))) + assert sum(v for _, v in _points(ok)) == CHECKOUT_TOTAL + _assert_400( + _series(client, ref.stream, window(end - 100_000 * US, end, **cpu(step="1"))), _bucket_cap_message(100000) + ) + + +@pytest.mark.parametrize( + ("drop", "message"), + [ + ("profile_type", "profile_type is required"), + ("profile_unit", "profile_unit is required"), + ("start_time", "start_time and end_time are required (microseconds)"), + ], + ids=["ser26-type", "ser27-unit", "ser28-start-time"], +) +def test_ser26_28_missing_fields(client: OpenObserveClient, ref: RefData, drop: str, message: str): + body = _checkout(ref) + del body[drop] + _assert_400(_series(client, ref.stream, body), message) + + +def test_ser29_start_equals_end(client: OpenObserveClient, ref: RefData): + _assert_400( + _series(client, ref.stream, window(ref.t0_us, ref.t0_us, **cpu())), "start_time must be less than end_time" + ) + + +def test_ser30_filter_op_not_equal(client: OpenObserveClient, ref: RefData): + resp = _series(client, ref.stream, _checkout(ref, filters=[{"key": "host_name", "op": "!=", "value": "x"}])) + _assert_400(resp, "only filter op '=' is supported") + + +@pytest.mark.parametrize("key", ["stack", "a;b", "a" * 129], ids=["ser31-reserved", "ser31-unsafe", "ser31-129-chars"]) +def test_ser31_invalid_filter_key(client: OpenObserveClient, ref: RefData, key: str): + _assert_400( + _series(client, ref.stream, _checkout(ref, filters=[{"key": key, "value": "x"}])), f"invalid filter key: {key}" + ) + + +def test_ser32_filter_unknown_column(client: OpenObserveClient, ref: RefData): + resp = _series(client, ref.stream, _checkout(ref, filters=[{"key": "no_such_col", "value": "x"}])) + assert resp.status_code == 400, resp.text + assert resp.json()["code"] == 20004 + assert "unknown field 'no_such_col'" in resp.json()["message"] + + +@pytest.mark.parametrize("timeout", [0, -5]) +def test_ser33_invalid_timeout(client: OpenObserveClient, ref: RefData, timeout: int): + _assert_400(_series(client, ref.stream, _checkout(ref, timeout=timeout)), "Invalid timeout parameter") + + +def test_ser34_malformed_json_body(client: OpenObserveClient, ref: RefData): + resp = client.post( + f"{ref.stream}/profiles/series", data='{"start_time":', headers={"Content-Type": "application/json"} + ) + assert resp.status_code == 400, resp.text + assert resp.json()["message"].startswith("Failed to parse the request body as JSON"), resp.text + + +def test_ser35_unknown_stream(client: OpenObserveClient, ref: RefData): + stream = f"no_such_{uuid.uuid4().hex[:8]}" + resp = _series(client, stream, _checkout(ref)) + assert resp.status_code == 400, resp.text + assert resp.json()["code"] == 20002 + assert resp.json()["message"] == f"Search stream not found: {stream}" + + +def test_ser36_empty_window(client: OpenObserveClient, ref: RefData, empty_window: dict): + body = _series_ok(client, ref.stream, {**empty_window, **cpu(service_name="checkout", step="60")}) + assert body["series"] == [] + + +def test_ser37_data_source_filter(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + t = now_ns() - 60 * NS + b.resource( + [b.profile([b.sample(["py_leaf"], values=[11])], time_ns=t)], {"service.name": "svc-ds"}, scope_name="pyroscope" + ) + b.resource( + [b.profile([b.sample(["eb_leaf"], values=[22])], time_ns=t)], + {"service.name": "svc-ds"}, + scope_name="ebpf-profiler", + ) + ingest_ok(client, prof_stream, b.model()) + wait_merge_total(client, prof_stream, recent_window(**cpu()), 33) + body = _series_ok(client, prof_stream, recent_window(**cpu(data_source="pyroscope"))) + assert sum(v for _, v in _points(body)) == 11 + + +def test_ser38_no_auth(client: OpenObserveClient, ref: RefData): + resp = requests.post(client.url(f"{ref.stream}/profiles/series"), json=_checkout(ref), timeout=30) + assert resp.status_code == 401, resp.text diff --git a/tests/api-testing/tests/profiles/test_profiles_validation.py b/tests/api-testing/tests/profiles/test_profiles_validation.py new file mode 100644 index 0000000000..d412c20318 --- /dev/null +++ b/tests/api-testing/tests/profiles/test_profiles_validation.py @@ -0,0 +1,490 @@ +"""Profiles ingest validation: Content-Type, decode errors, dictionary rules, partial success, time window (feature §2.1-§2.3).""" + +from __future__ import annotations + +import base64 + +import pytest + +from support.client import OpenObserveClient + +from .profiles_helpers import ( + BAD_CT_BODY, + NS, + PARTIAL_MSG, + STALE_NS, + ProfilesBuilder, + cpu, + decode_fields, + ingest_json, + ingest_proto, + now_ns, + profiles_streams, + recent_window, + simple_request, + to_json, + to_proto, + wait_merge_total, + wait_rows, +) + +A_STACKS = [(["a_leaf", "a_work", "a_main"], 300), (["a_work", "a_main"], 100)] +SAMPLE0 = "resource_profiles[0].scope_profiles[0].profiles[0].samples[0]" +PROFILE0 = "resource_profiles[0].scope_profiles[0].profiles[0]" + + +def _assert_invalid(resp, message: str) -> None: + assert resp.status_code == 400, f"{resp.status_code} {resp.text[:500]}" + assert resp.json() == {"code": 3, "message": message} + + +def _json_error(resp) -> str: + assert resp.status_code == 400, f"{resp.status_code} {resp.text[:500]}" + body = resp.json() + assert body["code"] == 3, body + return body["message"] + + +def _proto_status(resp) -> tuple[int, str]: + assert resp.status_code == 400, f"{resp.status_code} {resp.content[:300]!r}" + assert resp.headers["content-type"].startswith("application/x-protobuf") + fields = decode_fields(resp.content) + return fields[1][0], fields[2][0].decode() + + +def _base() -> dict: + return to_json(simple_request(A_STACKS)) + + +def _compacted() -> dict: + """Collector 0.153 shape: no zero sentinels, index 0 holds real entries.""" + return { + "resourceProfiles": [ + { + "resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "svc-c"}}]}, + "scopeProfiles": [ + { + "profiles": [ + { + "sampleType": {"typeStrindex": 0, "unitStrindex": 1}, + "timeUnixNano": str(now_ns() - 60 * NS), + "durationNano": str(10 * NS), + "samples": [{"stackIndex": 0, "values": ["300"]}, {"stackIndex": 1, "values": ["100"]}], + } + ] + } + ], + } + ], + "dictionary": { + "stringTable": ["cpu", "nanoseconds", "c_leaf", "c_work", "c_main"], + "functionTable": [{"nameStrindex": 2}, {"nameStrindex": 3}, {"nameStrindex": 4}], + "locationTable": [{"lines": [{"functionIndex": i}]} for i in range(3)], + "stackTable": [{"locationIndices": [0, 1, 2]}, {"locationIndices": [1, 2]}], + }, + } + + +def _samples(body: dict) -> list[dict]: + return body["resourceProfiles"][0]["scopeProfiles"][0]["profiles"][0]["samples"] + + +@pytest.mark.parametrize( + "content_type", + ["text/plain", "application/protobuf", None], + ids=["val01-text-plain", "val02-protobuf-without-x", "val03-missing"], +) +def test_val01_03_bad_content_type(client: OpenObserveClient, prof_stream: str, content_type: str | None): + headers = {"stream-name": prof_stream} + if content_type: + headers["Content-Type"] = content_type + resp = client.post("v1/profiles", data=to_proto(simple_request(A_STACKS)), headers=headers) + assert resp.status_code == 400, resp.text + assert resp.json() == BAD_CT_BODY + assert prof_stream not in profiles_streams(client) + + +def test_val04_garbage_protobuf(client: OpenObserveClient, prof_stream: str): + code, message = _proto_status(ingest_proto(client, prof_stream, b"\xff\xff\x07")) + assert code == 3 + assert message.startswith("Invalid proto: "), message + + +def test_val05_truncated_protobuf(client: OpenObserveClient, prof_stream: str): + code, message = _proto_status(ingest_proto(client, prof_stream, to_proto(simple_request(A_STACKS))[:-3])) + assert code == 3 + assert message.startswith("Invalid proto: "), message + assert prof_stream not in profiles_streams(client) + + +@pytest.mark.parametrize( + "raw", + ['{"resourceProfiles":[', '{"resourceProfiles":"x"}', "[]"], + ids=["val06-malformed", "val07-wrong-type", "val08-array"], +) +def test_val06_08_invalid_json(client: OpenObserveClient, prof_stream: str, raw: str): + message = _json_error(ingest_json(client, prof_stream, raw)) + assert message.startswith("Invalid json: "), message + + +SENTINELS = { + "mapping_table": ("mappingTable", {"memoryStart": "4096"}), + "location_table": ("locationTable", {"address": "1"}), + "function_table": ("functionTable", {"nameStrindex": 1}), + "link_table": ("linkTable", {"traceId": "01" * 16, "spanId": "02" * 8}), + "string_table": ("stringTable", "not-empty"), + "attribute_table": ("attributeTable", {"keyStrindex": 1, "value": {"stringValue": "x"}}), + "stack_table": ("stackTable", {"locationIndices": [1]}), +} + + +@pytest.mark.parametrize("table", list(SENTINELS), ids=[f"val09-{t}" for t in SENTINELS]) +def test_val09_canonical_sentinel_per_table(client: OpenObserveClient, prof_stream: str, table: str): + body = _base() + key, entry = SENTINELS[table] + body["dictionary"][key][0] = entry + _assert_invalid(ingest_json(client, prof_stream, body), f"dictionary.{table}[0] must be the zero value") + + +def test_val10_sentinel_order_function_before_stack(client: OpenObserveClient, prof_stream: str): + body = _base() + body["dictionary"]["functionTable"][0] = {"nameStrindex": 1} + body["dictionary"]["stackTable"][0] = {"locationIndices": [1]} + _assert_invalid(ingest_json(client, prof_stream, body), "dictionary.function_table[0] must be the zero value") + + +def test_val11_fully_compacted_dictionary_accepted(client: OpenObserveClient, prof_stream: str): + resp = ingest_json(client, prof_stream, _compacted()) + assert resp.status_code == 200, resp.text + assert resp.json() == {} + rows = wait_rows(client, prof_stream, 2) + assert sorted((r["stack"], r["value"]) for r in rows) == [("c_leaf;c_work;c_main", 300), ("c_work;c_main", 100)] + assert {(r["profile_type"], r["profile_unit"]) for r in rows} == {("cpu", "nanoseconds")} + + +def test_val12_only_string_zero_set_is_not_compacted(client: OpenObserveClient, prof_stream: str): + body = _base() + body["dictionary"]["stringTable"][0] = "cpu" + _assert_invalid(ingest_json(client, prof_stream, body), "dictionary.string_table[0] must be the zero value") + + +def test_val13_string_and_stack_without_location_not_compacted(client: OpenObserveClient, prof_stream: str): + body = _base() + body["dictionary"]["stringTable"][0] = "cpu" + body["dictionary"]["stackTable"][0] = {"locationIndices": [1]} + body["dictionary"]["functionTable"][0] = {"nameStrindex": 1} + _assert_invalid(ingest_json(client, prof_stream, body), "dictionary.function_table[0] must be the zero value") + + +def test_val14_stack_and_location_without_string_not_compacted(client: OpenObserveClient, prof_stream: str): + body = _base() + body["dictionary"]["stackTable"][0] = {"locationIndices": [1]} + body["dictionary"]["locationTable"][0] = {"lines": [{"functionIndex": 1}]} + _assert_invalid(ingest_json(client, prof_stream, body), "dictionary.location_table[0] must be the zero value") + + +def test_val15_compacted_index_checked_before_rebase(client: OpenObserveClient, prof_stream: str): + body = _compacted() + _samples(body)[0]["stackIndex"] = 2 + _assert_invalid( + ingest_json(client, prof_stream, body), f"{SAMPLE0}.stack_index: index 2 is outside dictionary table length 2" + ) + + +@pytest.mark.parametrize("index", [99, -1, 3], ids=["val16-99", "val17-negative", "val18-equal-len"]) +def test_val16_18_stack_index_out_of_range(client: OpenObserveClient, prof_stream: str, index: int): + body = _base() + assert len(body["dictionary"]["stackTable"]) == 3 + _samples(body)[0]["stackIndex"] = index + _assert_invalid( + ingest_json(client, prof_stream, body), + f"{SAMPLE0}.stack_index: index {index} is outside dictionary table length 3", + ) + + +def test_val19_stack_index_last_entry_ok(client: OpenObserveClient, prof_stream: str): + body = _base() + _samples(body)[0]["stackIndex"] = 2 + resp = ingest_json(client, prof_stream, body) + assert resp.status_code == 200, resp.text + assert resp.json() == {} + wait_merge_total(client, prof_stream, recent_window(**cpu()), 400) + + +def test_val20_index_zero_with_empty_table(client: OpenObserveClient, prof_stream: str): + body = _base() + body["dictionary"]["linkTable"] = [] + _samples(body)[0]["linkIndex"] = 0 + resp = ingest_json(client, prof_stream, body) + assert resp.status_code == 200, resp.text + assert resp.json() == {} + wait_merge_total(client, prof_stream, recent_window(**cpu()), 400) + + +def test_val21_function_index_out_of_range(client: OpenObserveClient, prof_stream: str): + body = _base() + assert len(body["dictionary"]["functionTable"]) == 4 + body["dictionary"]["locationTable"][1]["lines"][0]["functionIndex"] = 77 + _assert_invalid( + ingest_json(client, prof_stream, body), + "dictionary.location_table[1].lines[0].function_index: index 77 is outside dictionary table length 4", + ) + + +def test_val22_link_index_out_of_range(client: OpenObserveClient, prof_stream: str): + body = _base() + body["dictionary"]["linkTable"].append({"traceId": "01" * 16, "spanId": "02" * 8}) + _samples(body)[1]["linkIndex"] = 5 + _assert_invalid( + ingest_json(client, prof_stream, body), + f"{PROFILE0}.samples[1].link_index: index 5 is outside dictionary table length 2", + ) + + +def test_val23_location_indices_out_of_range(client: OpenObserveClient, prof_stream: str): + body = _base() + assert len(body["dictionary"]["locationTable"]) == 4 + body["dictionary"]["stackTable"][1]["locationIndices"][0] = 50 + _assert_invalid( + ingest_json(client, prof_stream, body), + "dictionary.stack_table[1].location_indices[0]: index 50 is outside dictionary table length 4", + ) + + +def test_val24_type_strindex_out_of_range(client: OpenObserveClient, prof_stream: str): + body = _base() + assert len(body["dictionary"]["stringTable"]) == 6 + body["resourceProfiles"][0]["scopeProfiles"][0]["profiles"][0]["sampleType"]["typeStrindex"] = 99 + _assert_invalid( + ingest_json(client, prof_stream, body), + f"{PROFILE0}.sample_type.type_strindex: index 99 is outside dictionary table length 6", + ) + + +def test_val25_period_type_unit_strindex_out_of_range(client: OpenObserveClient, prof_stream: str): + body = _base() + body["resourceProfiles"][0]["scopeProfiles"][0]["profiles"][0]["periodType"] = { + "typeStrindex": 4, + "unitStrindex": 99, + } + _assert_invalid( + ingest_json(client, prof_stream, body), + f"{PROFILE0}.period_type.unit_strindex: index 99 is outside dictionary table length 6", + ) + + +def test_val26_mapping_index_out_of_range(client: OpenObserveClient, prof_stream: str): + body = _base() + body["dictionary"]["locationTable"][1]["mappingIndex"] = 7 + _assert_invalid( + ingest_json(client, prof_stream, body), + "dictionary.location_table[1].mapping_index: index 7 is outside dictionary table length 1", + ) + + +def test_val27_mapping_filename_out_of_range(client: OpenObserveClient, prof_stream: str): + body = _base() + body["dictionary"]["mappingTable"].append({"memoryStart": "4096", "filenameStrindex": 999}) + _assert_invalid( + ingest_json(client, prof_stream, body), + "dictionary.mapping_table[1].filename_strindex: index 999 is outside dictionary table length 6", + ) + + +def test_val28_sample_attribute_index_out_of_range(client: OpenObserveClient, prof_stream: str): + body = _base() + _samples(body)[0]["attributeIndices"] = [5] + _assert_invalid( + ingest_json(client, prof_stream, body), + f"{SAMPLE0}.attribute_indices[0]: index 5 is outside dictionary table length 1", + ) + + +def test_val29_nested_array_string_strindex_out_of_range(client: OpenObserveClient, prof_stream: str): + body = _base() + body["dictionary"]["attributeTable"].append( + {"keyStrindex": 1, "value": {"arrayValue": {"values": [{"stringValueStrindex": 999}]}}} + ) + _assert_invalid( + ingest_json(client, prof_stream, body), + "dictionary.attribute_table[1].value.array_value.values[0].string_value_strindex: " + "index 999 is outside dictionary table length 6", + ) + + +def test_val30_resource_key_strindex_out_of_range(client: OpenObserveClient, prof_stream: str): + body = _base() + body["resourceProfiles"][0]["resource"]["attributes"].insert(0, {"keyStrindex": 999, "value": {"stringValue": "x"}}) + _assert_invalid( + ingest_json(client, prof_stream, body), + "resource_profiles[0].resource.attributes[0].key_strindex: index 999 is outside dictionary table length 6", + ) + + +def _partial(client: OpenObserveClient, stream: str, model: dict, rejected: str) -> None: + resp = ingest_json(client, stream, to_json(model)) + assert resp.status_code == 200, resp.text + assert resp.json() == {"partialSuccess": {"rejectedProfiles": rejected, "errorMessage": PARTIAL_MSG}} + + +def test_val31_length_mismatch_rejects_sample_keeps_good(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + t = now_ns() - 60 * NS + bad = b.sample(["bad_leaf"], values=[1, 2], timestamps=[t, t + 1, t + 2]) + b.resource([b.profile([bad, b.sample(["good_leaf"], values=[50])], time_ns=t)], {"service.name": "svc-a"}) + _partial(client, prof_stream, b.model(), "1") + rows = wait_rows(client, prof_stream, 1) + assert (rows[0]["stack"], rows[0]["value"]) == ("good_leaf", 50) + wait_merge_total(client, prof_stream, recent_window(**cpu()), 50) + + +def test_val32_sample_with_no_values_or_timestamps(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + b.resource( + [b.profile([b.sample(["empty_leaf"]), b.sample(["good_leaf"], values=[50])], time_ns=now_ns() - 60 * NS)], + {"service.name": "svc-a"}, + ) + _partial(client, prof_stream, b.model(), "1") + rows = wait_rows(client, prof_stream, 1) + assert (rows[0]["stack"], rows[0]["value"]) == ("good_leaf", 50) + + +def test_val33_rejected_counts_profiles_not_samples(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + t = now_ns() - 60 * NS + x = b.profile([b.sample(["x1"], values=[1], timestamps=[t, t]), b.sample(["x2"])], time_ns=t) + y = b.profile([b.sample(["y_leaf"], values=[8])], time_ns=t) + b.resource([x, y], {"service.name": "svc-a"}) + _partial(client, prof_stream, b.model(), "1") + rows = wait_rows(client, prof_stream, 1) + assert (rows[0]["stack"], rows[0]["value"]) == ("y_leaf", 8) + + +def test_val34_two_bad_profiles(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + t = now_ns() - 60 * NS + profiles = [b.profile([b.sample([f"bad_{i}"]), b.sample([f"ok_{i}"], values=[i + 1])], time_ns=t) for i in range(2)] + b.resource(profiles, {"service.name": "svc-a"}) + _partial(client, prof_stream, b.model(), "2") + rows = wait_rows(client, prof_stream, 2) + assert sorted((r["stack"], r["value"]) for r in rows) == [("ok_0", 1), ("ok_1", 2)] + + +def _paired(ts: list[int], values: list[int]) -> dict: + b = ProfilesBuilder() + b.resource( + [b.profile([b.sample(["win_leaf"], values=values, timestamps=ts)], time_ns=now_ns())], {"service.name": "svc-a"} + ) + return b.model() + + +def test_val35_all_samples_far_future(client: OpenObserveClient, prof_stream: str): + _partial(client, prof_stream, _paired([now_ns() + 48 * 3600 * NS], [5]), "1") + assert prof_stream not in profiles_streams(client) + + +def test_val36_future_inside_window(client: OpenObserveClient, prof_stream: str): + ts = now_ns() + 23 * 3600 * NS + resp = ingest_json(client, prof_stream, to_json(_paired([ts], [5]))) + assert resp.status_code == 200, resp.text + assert resp.json() == {} + rows = wait_rows(client, prof_stream, 1) + assert (rows[0]["_timestamp"], rows[0]["value"]) == (ts // 1000, 5) + + +def test_val37_past_out_of_window(client: OpenObserveClient, prof_stream: str): + _partial(client, prof_stream, _paired([now_ns() - STALE_NS], [5]), "1") + assert prof_stream not in profiles_streams(client) + + +def test_val38_past_inside_window(client: OpenObserveClient, prof_stream: str): + ts = now_ns() - 4 * 3600 * NS + resp = ingest_json(client, prof_stream, to_json(_paired([ts], [5]))) + assert resp.status_code == 200, resp.text + assert resp.json() == {} + rows = wait_rows(client, prof_stream, 1) + assert (rows[0]["_timestamp"], rows[0]["value"]) == (ts // 1000, 5) + + +def test_val39_mixed_observations_keep_in_window(client: OpenObserveClient, prof_stream: str): + now = now_ns() + _partial(client, prof_stream, _paired([now, now + 48 * 3600 * NS], [1, 2]), "1") + rows = wait_rows(client, prof_stream, 1) + assert (rows[0]["_timestamp"], rows[0]["value"]) == (now // 1000, 1) + assert rows[0]["profile_duration_nanos"] == 10 * NS + + +def test_val40_proto_partial_success_shape(client: OpenObserveClient, prof_stream: str): + b = ProfilesBuilder() + t = now_ns() - 60 * NS + bad = b.sample(["bad_leaf"], values=[1, 2], timestamps=[t]) + b.resource([b.profile([bad, b.sample(["good_leaf"], values=[50])], time_ns=t)], {"service.name": "svc-a"}) + resp = ingest_proto(client, prof_stream, b.model()) + assert resp.status_code == 200, resp.text + assert resp.headers["content-type"].startswith("application/x-protobuf") + partial = decode_fields(decode_fields(resp.content)[1][0]) + assert partial[1] == [1] + assert partial[2] == [PARTIAL_MSG.encode()] + wait_merge_total(client, prof_stream, recent_window(**cpu()), 50) + + +def _linked_body(trace_id: str) -> dict: + b = ProfilesBuilder() + link = b.link("01" * 16, "02" * 8) + b.resource( + [b.profile([b.sample(["link_leaf"], values=[3], link=link)], time_ns=now_ns() - 60 * NS)], + {"service.name": "svc-a"}, + ) + body = to_json(b.model()) + body["dictionary"]["linkTable"][link]["traceId"] = trace_id + return body + + +@pytest.mark.parametrize( + "trace_id", ["0af7651916cd43dd8448eb211c80319", "zz" * 16], ids=["val41-odd-length", "val42-non-hex"] +) +def test_val41_42_bad_trace_id(client: OpenObserveClient, prof_stream: str, trace_id: str): + message = _json_error(ingest_json(client, prof_stream, _linked_body(trace_id))) + assert message.startswith("Invalid json: "), message + assert prof_stream not in profiles_streams(client) + + +def test_val43_short_trace_id_stored_as_is(client: OpenObserveClient, prof_stream: str): + resp = ingest_json(client, prof_stream, _linked_body("abcd1234")) + assert resp.status_code == 200, resp.text + assert wait_rows(client, prof_stream, 1)[0]["trace_id"] == "abcd1234" + + +def test_val44_stringified_integers_coerced(client: OpenObserveClient, prof_stream: str): + body = _base() + for sample in _samples(body): + sample["stackIndex"] = str(sample["stackIndex"]) + profile = body["resourceProfiles"][0]["scopeProfiles"][0]["profiles"][0] + assert isinstance(profile["timeUnixNano"], str) + _samples(body)[0]["values"] = ["5"] + resp = ingest_json(client, prof_stream, body) + assert resp.status_code == 200, resp.text + assert resp.json() == {} + wait_merge_total(client, prof_stream, recent_window(**cpu()), 105) + + +def test_val45_omitted_defaults(client: OpenObserveClient, prof_stream: str): + body = _base() + del body["resourceProfiles"][0]["scopeProfiles"][0]["profiles"][0]["durationNano"] + assert all("mappingIndex" not in loc for loc in body["dictionary"]["locationTable"]) + resp = ingest_json(client, prof_stream, body) + assert resp.status_code == 200, resp.text + rows = wait_rows(client, prof_stream, 2) + assert {r["duration_nanos"] for r in rows} == {0} + assert all("profile_duration_nanos" not in r for r in rows) + + +def test_val46_original_payload_ignored(client: OpenObserveClient, prof_stream: str): + body = _base() + profile = body["resourceProfiles"][0]["scopeProfiles"][0]["profiles"][0] + profile["originalPayloadFormat"] = "pprofext" + profile["originalPayload"] = base64.b64encode(b"\x1f\x8b raw pprof bytes").decode() + resp = ingest_json(client, prof_stream, body) + assert resp.status_code == 200, resp.text + assert resp.json() == {} + wait_merge_total(client, prof_stream, recent_window(**cpu()), 400) diff --git a/tests/ui-testing/ci-matrix/ci_matrix.json b/tests/ui-testing/ci-matrix/ci_matrix.json index 952648374e..a341065134 100644 --- a/tests/ui-testing/ci-matrix/ci_matrix.json +++ b/tests/ui-testing/ci-matrix/ci_matrix.json @@ -637,5 +637,20 @@ "slo-alerts.spec.js" ], "disabled": [] + }, + { + "testfolder": "Profiles", + "actual_folder": "Profiles", + "browser": "chrome", + "run_files": [ + "profiles-navigation-entry.spec.js", + "profiles-query-controls.spec.js", + "profiles-chart-breakdown.spec.js", + "profiles-views.spec.js", + "profiles-flame-graph.spec.js", + "profiles-compare.spec.js", + "profiles-states-edge.spec.js" + ], + "disabled": [] } ] diff --git a/tests/ui-testing/ci-matrix/smoke_config.json b/tests/ui-testing/ci-matrix/smoke_config.json index c3310ebe61..a242959537 100644 --- a/tests/ui-testing/ci-matrix/smoke_config.json +++ b/tests/ui-testing/ci-matrix/smoke_config.json @@ -173,6 +173,12 @@ "tests/ui-testing/pages/reportsPages/**", "tests/ui-testing/pages/dashboardPages/**" ], + "Profiles": [ + "web/src/plugins/profiles/**", + "web/src/services/profiles.*", + "web/src/components/common/FlameGraphView.vue", + "tests/ui-testing/pages/profilesPages/**" + ], "RUM": [ "web/src/components/rum/**", "web/src/views/RUM/**", diff --git a/tests/ui-testing/pages/alertsPages/alertCreationWizard.js b/tests/ui-testing/pages/alertsPages/alertCreationWizard.js index b94b425e49..189bbfc9f2 100644 --- a/tests/ui-testing/pages/alertsPages/alertCreationWizard.js +++ b/tests/ui-testing/pages/alertsPages/alertCreationWizard.js @@ -5,7 +5,6 @@ * - Real-time alerts with conditions * - Scheduled alerts with SQL/PromQL queries * - Multi-condition alerts with AND/OR logic - * - Alerts with deduplication configuration */ import { expect } from '@playwright/test'; @@ -180,7 +179,7 @@ export class AlertCreationWizard { /** * Create a real-time alert using Alerts 2.0 wizard UI - * Wizard steps for real-time: Step 1 (Setup) -> Step 2 (Conditions) -> Step 4 (Settings) -> Step 6 (Advanced) -> Submit + * Wizard steps for real-time: Step 1 (Setup) -> Step 2 (Conditions) -> Step 4 (Settings) -> Step 5 (Advanced) -> Submit * @param {string} streamName - Name of the stream * @param {string} column - Column name for condition * @param {string} value - Value for condition @@ -414,7 +413,7 @@ export class AlertCreationWizard { /** * Create a scheduled alert with SQL query using Alerts 2.0 wizard UI - * Wizard steps for scheduled: Step 1 (Setup) -> Step 2 (Conditions/SQL) -> Step 3 (Compare) -> Step 4 (Settings) -> Step 5 (Dedup) -> Step 6 (Advanced) -> Submit + * Wizard steps for scheduled: Step 1 (Setup) -> Step 2 (Conditions/SQL) -> Step 3 (Compare) -> Step 4 (Settings) -> Step 5 (Advanced) -> Submit * @param {string} streamName - Name of the stream * @param {string} destinationName - Name of the destination * @param {string} randomValue - Random string for unique naming @@ -775,208 +774,6 @@ export class AlertCreationWizard { return randomAlertName; } - /** - * Create a scheduled alert with deduplication configuration - * @param {string} streamName - Name of the stream - * @param {string} destinationName - Name of the destination - * @param {string} randomValue - Random string for unique naming - * @param {Object} dedupConfig - Deduplication configuration - * @param {number} [dedupConfig.timeWindowMinutes] - Time window in minutes - * @param {string[]} [dedupConfig.fingerprintFields] - Fields for fingerprinting - */ - async createScheduledAlertWithDeduplication(streamName, destinationName, randomValue, dedupConfig = {}) { - const randomAlertName = 'auto_dedup_alert_' + randomValue; - this.currentAlertName = randomAlertName; - - testLogger.info('Creating scheduled alert with deduplication', { - streamName, - destinationName, - alertName: randomAlertName, - dedupConfig - }); - - // ==================== ALERT SETUP ==================== - await this.page.locator(this.locators.addAlertButton).click(); - await this.page.waitForLoadState('networkidle', { timeout: 10000 }).catch(() => {}); - testLogger.info('Add alert dialog opened'); - - await this.fillAlertNameField(randomAlertName); - testLogger.info('Filled alert name', { alertName: randomAlertName }); - - // Select stream type (logs) via OSelect popover pattern (§4) - await this.page.locator(this.locators.streamTypeDropdown).click(); - await expect(this.page.locator(this.locators.streamTypePopover)).toBeVisible({ timeout: 10000 }); - await this.page.locator(`${this.locators.streamTypeOption}[data-test-value="logs"]`).click(); - // Deterministic wait: stream-name dropdown becomes enabled once `formData.stream_type` is set (§10) - await expect(this.page.locator(this.locators.streamNameDropdown)).toBeEnabled({ timeout: 10000 }); - testLogger.info('Selected stream type: logs'); - - // Select stream name - await this.selectStreamByName(streamName); - testLogger.info('Selected stream name', { streamName }); - - // Select Scheduled alert type via dropdown (v3 UI) - await this._selectAlertType('Scheduled'); - testLogger.info('Selected scheduled alert type'); - - // ==================== CONDITIONS (SQL) ==================== - const sqlTab = this.page.locator('[data-test="step2-query-tabs"] button:has-text("SQL")'); - await sqlTab.waitFor({ state: 'visible', timeout: 10000 }); - await sqlTab.click(); - await this.page.waitForTimeout(1000); - testLogger.info('Switched to SQL tab'); - - const viewEditorBtn = this.page.locator(this.locators.viewEditorButton); - await viewEditorBtn.waitFor({ state: 'visible', timeout: 10000 }); - await viewEditorBtn.click(); - await this.page.waitForLoadState('networkidle', { timeout: 10000 }).catch(() => {}); - await this.page.waitForTimeout(1000); - testLogger.info('Opened SQL Editor dialog'); - - await this._replaceSqlEditorQuery(`SELECT kubernetes_labels_name FROM "${streamName}" where kubernetes_labels_name = 'ziox-querier'`); - - await this.page.locator(this.locators.runQueryButton).click(); - await this.page.waitForLoadState('networkidle', { timeout: 10000 }).catch(() => {}); - await this.page.waitForTimeout(2000); - testLogger.info('Ran SQL query'); - - // Close the query-editor drawer via ITS back button — scope to the - // query-editor-dialog container, since an unscoped add-alert-back-btn matches - // the wizard's own back arrow and leaves the drawer open to intercept clicks - // on the threshold row below. Assert the drawer detaches before proceeding. - const editorDialog = this.page.locator('[data-test="query-editor-dialog"]').first(); - const dialogBackBtn = editorDialog.locator('[data-test="add-alert-back-btn"]').first(); - await dialogBackBtn.waitFor({ state: 'visible', timeout: 10000 }); - await dialogBackBtn.click(); - await expect(editorDialog).not.toBeAttached({ timeout: 10000 }); - await this.page.waitForLoadState('networkidle', { timeout: 10000 }).catch(() => {}); - await this.page.waitForTimeout(500); - - testLogger.info('Closed SQL Editor dialog — portal cleaned up'); - - // ==================== ALERT SETTINGS ==================== - // Threshold row via stable data-test hooks (same as createScheduledAlertWithSQL) — - // the old `.alert-condition-row`/`.alert-v3-select` class chain no longer exists - // on current builds (removed with the alert wizard revamp). - // OSelect forwards data-test to an inner -trigger button; the wrapper div itself - // is not clickable. Target the trigger so the popover actually opens. - const thresholdOperatorSelect = this.page.locator('[data-test="alert-trigger-operator-select"]').first(); - await thresholdOperatorSelect.waitFor({ state: 'visible', timeout: 10000 }); - testLogger.info('Threshold section visible'); - - // Open via the OSelect helper (clicks the inner -trigger until aria-expanded), - // not a raw click — the reka-ui trigger can open-then-close on a single click. - await openOSelectDropdown(this.page, thresholdOperatorSelect); - await this.page.locator('[data-test="alert-trigger-operator-select-option"][data-test-value=">="]').first().click(); - testLogger.info('Set threshold operator: >='); - - const thresholdInput = this.page.locator('[data-test="alert-trigger-threshold-input-field"]').first(); - await thresholdInput.waitFor({ state: 'visible', timeout: 5000 }); - await thresholdInput.fill('1'); - testLogger.info('Set threshold value: 1'); - - const periodInput = this.page.locator('[data-test="alert-settings-period-input-field"]').first(); - await this.page.waitForTimeout(500); - if (await periodInput.isVisible({ timeout: 3000 })) { - await periodInput.fill('15'); - testLogger.info('Set period: 15 minutes'); - } - - const destinationDropdown = this.page.locator('[data-test="alert-destinations-select"]'); - await destinationDropdown.waitFor({ state: 'visible', timeout: 5000 }); - await openOSelectDropdown(this.page, destinationDropdown); - await this.page.waitForTimeout(1000); - - // Use popover selector for destination - const visibleDestMenuDedupItems = this.page.locator('[data-test$="-popover"] [data-test$="-option"]'); - await expect(visibleDestMenuDedupItems.first()).toBeVisible({ timeout: 5000 }); - - let destFoundDedup = false; - const destOptionDedup = visibleDestMenuDedupItems.filter({ hasText: destinationName }).first(); - if (await destOptionDedup.isVisible({ timeout: 3000 }).catch(() => false)) { - await destOptionDedup.click(); - destFoundDedup = true; - testLogger.info('Selected destination via popover', { destinationName }); - } - - if (!destFoundDedup) { - // Fallback: select first available destination - await visibleDestMenuDedupItems.first().click(); - testLogger.warn('Selected first available destination (fallback)', { requestedDestination: destinationName }); - } - await this.page.waitForTimeout(500); - await this.page.locator('body').click({ position: { x: 10, y: 10 } }); - testLogger.info('Selected destination', { destinationName }); - - // ==================== DEDUPLICATION (Advanced tab) ==================== - await this._switchToAdvancedTab(); - - const dedupContainer = this.page.locator(this.locators.stepDeduplication); - await expect(dedupContainer).toBeVisible({ timeout: 5000 }); - testLogger.info('Deduplication container visible in Advanced tab'); - - // Configure fingerprint fields if provided - if (dedupConfig.fingerprintFields && dedupConfig.fingerprintFields.length > 0) { - const fingerprintSelect = this.page.locator(this.locators.stepDeduplicationFingerprintSelect); - await fingerprintSelect.waitFor({ state: 'visible', timeout: 5000 }); - - for (const field of dedupConfig.fingerprintFields) { - await fingerprintSelect.click(); - await this.page.waitForTimeout(500); - await fingerprintSelect.locator('input').fill(field); - await this.page.waitForTimeout(500); - - const fieldOption = this.page.getByRole('option', { name: field }).first(); - if (await fieldOption.isVisible({ timeout: 2000 })) { - await fieldOption.click(); - } else { - await this.page.keyboard.press('Enter'); - } - await this.page.waitForTimeout(300); - } - await this.page.locator('body').click({ position: { x: 10, y: 10 } }); - testLogger.info('Configured fingerprint fields', { fields: dedupConfig.fingerprintFields }); - } else { - testLogger.info('Skipping fingerprint fields (auto-detect)'); - } - - // Configure time window if provided - if (dedupConfig.timeWindowMinutes !== undefined) { - const timeWindowInput = this.page.locator(this.locators.stepDeduplicationTimeWindowInput); - await timeWindowInput.waitFor({ state: 'visible', timeout: 5000 }); - await timeWindowInput.clear(); - await timeWindowInput.fill(String(dedupConfig.timeWindowMinutes)); - testLogger.info('Set time window', { minutes: dedupConfig.timeWindowMinutes }); - } else { - testLogger.info('Skipping time window (auto-detect)'); - } - - // ==================== SUBMIT ALERT ==================== - // The submit button sits inside a scroll container that clips it from the viewport - // — worse here because the Advanced/deduplication tab lengthens the form. Playwright's - // .click() (even force:true) doesn't bypass scroll-container clipping, so the click - // silently misses, the alert never saves, and the success toast never appears (the - // exact alpha1 failure). Use a native DOM click via evaluate() — the same reliable - // submit the non-dedup scheduled-alert path (createScheduledAlertWithSQL) uses. - await this.page.locator(this.locators.alertSubmitButton).waitFor({ state: 'attached', timeout: 10000 }); - await this.page.evaluate((selector) => { - const btn = document.querySelector(selector); - if (btn) btn.click(); - }, this.locators.alertSubmitButton); - // OToast renders the message in 3 elements (sr-only ARIA span, sr-only title div, - // visible message div) — scope to the visible `o-toast-message` data-test to avoid - // strict-mode violation per AGENT_RULES §2. - await expect( - this.page - .locator('[data-test="o-toast-message"]') - .filter({ hasText: this.locators.alertSuccessMessage }) - .first() - ).toBeVisible({ timeout: 30000 }); - testLogger.info('Successfully created scheduled alert with deduplication', { alertName: randomAlertName }); - - return randomAlertName; - } - /** * Test condition operator toggle (AND to OR) during alert creation * @param {string} streamName - Name of the stream @@ -1159,39 +956,6 @@ export class AlertCreationWizard { testLogger.info('Deleted condition', { index }); } - /** - * Verify deduplication step is visible (only for scheduled alerts) - */ - async verifyDeduplicationStepVisible() { - const dedupContainer = this.page.locator(this.locators.stepDeduplication); - await expect(dedupContainer).toBeVisible({ timeout: 5000 }); - - const fingerprintSelect = this.page.locator(this.locators.stepDeduplicationFingerprintSelect); - await expect(fingerprintSelect).toBeVisible({ timeout: 3000 }); - - const timeWindowInput = this.page.locator(this.locators.stepDeduplicationTimeWindowInput); - await expect(timeWindowInput).toBeVisible({ timeout: 3000 }); - - testLogger.info('Deduplication step verified - all elements visible'); - return true; - } - - /** - * Get current deduplication configuration from the UI - */ - async getDeduplicationConfig() { - const timeWindowInput = this.page.locator(this.locators.stepDeduplicationTimeWindowInput); - const timeWindowValue = await timeWindowInput.inputValue(); - - const fingerprintSelect = this.page.locator(this.locators.stepDeduplicationFingerprintSelect); - const chips = await fingerprintSelect.locator('[data-test*="chip"], [data-test*="badge"]').allTextContents(); - - return { - timeWindowMinutes: timeWindowValue ? parseInt(timeWindowValue, 10) : null, - fingerprintFields: chips - }; - } - getCurrentAlertName() { return this.currentAlertName; } @@ -1264,262 +1028,11 @@ export class AlertCreationWizard { testLogger.info('Entered SQL query', { query }); } - /** - * Create a scheduled alert with deduplication for validation testing - * Uses custom columns (city) and configures for quick evaluation cycles - * @param {string} streamName - Name of the stream (with custom columns) - * @param {string} column - Column name for condition (e.g., 'city') - * @param {string} value - Value for condition (e.g., 'bangalore') - * @param {string} destinationName - Name of the destination (validation destination) - * @param {string} randomValue - Random string for unique naming - * @param {Object} dedupConfig - Deduplication configuration - * @param {number} [dedupConfig.timeWindowMinutes] - Time window in minutes (default: 5) - * @param {string[]} [dedupConfig.fingerprintFields] - Fields for fingerprinting (default: [column]) - */ - async createScheduledAlertWithDedupForValidation(streamName, column, value, destinationName, randomValue, dedupConfig = {}) { - const randomAlertName = 'auto_dedup_val_' + randomValue; - this.currentAlertName = randomAlertName; - - // Default dedup config for validation testing - const timeWindowMinutes = dedupConfig.timeWindowMinutes || 5; - const fingerprintFields = dedupConfig.fingerprintFields || [column]; - - testLogger.info('Creating scheduled alert with deduplication for validation', { - streamName, - column, - value, - destinationName, - alertName: randomAlertName, - dedupConfig: { timeWindowMinutes, fingerprintFields } - }); - - // ==================== ALERT SETUP ==================== - await this.page.locator(this.locators.addAlertButton).click(); - await this.page.waitForLoadState('networkidle', { timeout: 10000 }).catch(() => {}); - testLogger.info('Add alert dialog opened'); - - await this.fillAlertNameField(randomAlertName); - testLogger.info('Filled alert name', { alertName: randomAlertName }); - - // Select stream type (logs) via OSelect popover pattern (§4) - await this.page.locator(this.locators.streamTypeDropdown).click(); - await expect(this.page.locator(this.locators.streamTypePopover)).toBeVisible({ timeout: 10000 }); - await this.page.locator(`${this.locators.streamTypeOption}[data-test-value="logs"]`).click(); - // Deterministic wait: stream-name dropdown becomes enabled once `formData.stream_type` is set (§10) - await expect(this.page.locator(this.locators.streamNameDropdown)).toBeEnabled({ timeout: 10000 }); - testLogger.info('Selected stream type: logs'); - - // Select stream name - await this.selectStreamByName(streamName); - testLogger.info('Selected stream name', { streamName }); - - // Select Scheduled alert type via dropdown (v3 UI) - await this._selectAlertType('Scheduled'); - testLogger.info('Selected scheduled alert type'); - - // ==================== CONDITIONS (SQL) ==================== - const sqlTab = this.page.locator('[data-test="step2-query-tabs"] button:has-text("SQL")'); - await sqlTab.waitFor({ state: 'visible', timeout: 10000 }); - await sqlTab.click(); - await this.page.waitForTimeout(1000); - testLogger.info('Switched to SQL tab'); - - const viewEditorBtn = this.page.locator(this.locators.viewEditorButton); - await viewEditorBtn.waitFor({ state: 'visible', timeout: 10000 }); - await viewEditorBtn.click(); - await this.page.waitForLoadState('networkidle', { timeout: 10000 }).catch(() => {}); - await this.page.waitForTimeout(1000); - testLogger.info('Opened SQL Editor dialog'); - - // Use SQL query that selects data matching our condition - const sqlQuery = `SELECT ${column} FROM "${streamName}" WHERE ${column} = '${value}'`; - await this._replaceSqlEditorQuery(sqlQuery); - - await this.page.locator(this.locators.runQueryButton).click(); - await this.page.waitForLoadState('networkidle', { timeout: 10000 }).catch(() => {}); - await this.page.waitForTimeout(2000); - testLogger.info('Ran SQL query'); - - // Close dialog — try back button, then close button - try { - const closeButton = this.page.locator('[data-test="add-alert-back-btn"]').first(); - await closeButton.click({ force: true, timeout: 10000 }); - } catch (error) { - testLogger.warn('Close button force-click failed, trying dialog close button', { error: error.message }); - const dialogCloseBtn = this.page.locator('[data-test="o-dialog-close-btn"]').first(); - if (await dialogCloseBtn.isVisible({ timeout: 2000 }).catch(() => false)) { - await dialogCloseBtn.click(); - } - await this.page.waitForTimeout(500); - } - await this.page.waitForLoadState('networkidle', { timeout: 10000 }).catch(() => {}); - await this.page.waitForTimeout(1000); - - // Verify SQL editor dialog is closed by waiting for it to be hidden - const sqlEditorDialog = this.page.locator(this.locators.sqlEditorDialog); - const dialogHidden = await sqlEditorDialog.isHidden({ timeout: 5000 }).catch(() => true); - if (!dialogHidden) { - testLogger.warn('SQL Editor dialog still visible, clicking close button'); - await this.page.locator('[data-test="o-dialog-close-btn"]').first().click().catch(() => {}); - await this.page.waitForTimeout(1000); - } - testLogger.info('Closed SQL Editor dialog'); - - // ==================== COMPARE WITH PAST (Advanced tab) ==================== - await this._switchToAdvancedTab(); - - // Add time range comparison - const addTimeRangeBtn = this.page.locator(this.locators.multiTimeRangeAddButton); - if (await addTimeRangeBtn.isVisible({ timeout: 3000 })) { - await addTimeRangeBtn.click(); - await this.page.waitForTimeout(1000); - await this.page.locator('[data-test="date-time-btn"]').click(); - await this.page.locator('[data-test="date-time-relative-30-m-btn"]').click(); - // The auto-apply picker stays open after a pick, and its wider popover can cover the tab bar below. - const dateTimeMenu = this.page.locator('#date-time-menu'); - await this.page.keyboard.press('Escape'); - await dateTimeMenu.waitFor({ state: 'hidden', timeout: 3000 }).catch(() => {}); - testLogger.info('Added 30-minute time range comparison'); - } else { - testLogger.info('Time range button not visible, skipping'); - } - - // ==================== ALERT SETTINGS (back in Alert Rules tab) ==================== - await this._switchToAlertRulesTab(); - - // The SQL query-editor drawer often doesn't close via its close button, and while it stays - // open reka disables pointer events on the rest of the page — so the threshold-operator - // click below is intercepted (by the drawer, its overlay, or ultimately ). Dismiss it - // with Escape until it is no longer open, which lets reka restore page interaction. - for (let i = 0; i < 5; i++) { - if ((await this.page.locator('[data-test="query-editor-dialog"][data-state="open"]').count()) === 0) break; - await this.page.keyboard.press('Escape').catch(() => {}); - await this.page.waitForTimeout(400); - } - // Fallback: if it still won't close, re-enable page pointer events and hide any residue so - // the operator is reachable anyway. - await this.page.evaluate(() => { - document.body.style.pointerEvents = 'auto'; - document.querySelectorAll( - '[data-test="query-editor-dialog"], [data-test="o-drawer-overlay"], div[data-reka-dialog-overlay], div[data-reka-portalled]' - ).forEach((el) => { el.style.display = 'none'; }); - }).catch(() => {}); - await this.page.waitForTimeout(300); - - // Threshold row via stable data-test hooks — the old `.alert-condition-row`/ - // `.alert-v3-select` class chain no longer exists on current builds. - // OSelect forwards data-test to an inner -trigger button; the wrapper div itself - // is not clickable. Target the trigger so the popover actually opens. - const thresholdOperatorSelect = this.page.locator('[data-test="alert-trigger-operator-select"]').first(); - await thresholdOperatorSelect.waitFor({ state: 'visible', timeout: 10000 }); - // Open via the OSelect helper (clicks the inner -trigger until aria-expanded), - // not a raw click — the reka-ui trigger can open-then-close on a single click. - await openOSelectDropdown(this.page, thresholdOperatorSelect); - await this.page.locator('[data-test="alert-trigger-operator-select-option"][data-test-value=">="]').first().click(); - testLogger.info('Set threshold operator: >='); - - const thresholdInput = this.page.locator('[data-test="alert-trigger-threshold-input-field"]').first(); - await thresholdInput.waitFor({ state: 'visible', timeout: 10000 }); - await thresholdInput.fill('1'); - testLogger.info('Set threshold value: 1'); - - // Set shortest period (1 minute) for faster testing - const periodInput = this.page.locator('[data-test="alert-settings-period-input-field"]').first(); - await this.page.waitForTimeout(500); - if (await periodInput.isVisible({ timeout: 5000 })) { - await periodInput.fill('1'); - testLogger.info('Set period: 1 minute (shortest)'); - } - - const destinationDropdown = this.page.locator('[data-test="alert-destinations-select"]'); - await destinationDropdown.waitFor({ state: 'visible', timeout: 10000 }); - await openOSelectDropdown(this.page, destinationDropdown); - await this.page.waitForTimeout(1000); - - // Use popover selector for destination - const visibleDestMenuItems = this.page.locator('[data-test$="-popover"] [data-test$="-option"]'); - await expect(visibleDestMenuItems.first()).toBeVisible({ timeout: 5000 }); - - let destFound = false; - const destOption = visibleDestMenuItems.filter({ hasText: destinationName }).first(); - if (await destOption.isVisible({ timeout: 3000 }).catch(() => false)) { - await destOption.click(); - destFound = true; - testLogger.info('Selected destination via popover', { destinationName }); - } - - if (!destFound) { - // Try scrolling to find the destination - try { - await this.commonActions.scrollAndFindOption(destinationName, 'template'); - testLogger.info('Found destination by scrolling', { destinationName }); - } catch (scrollError) { - await visibleDestMenuItems.first().click(); - testLogger.warn('Selected first available destination (fallback)', { requestedDestination: destinationName }); - } - } - await this.page.waitForTimeout(500); - await this.page.locator('body').click({ position: { x: 10, y: 10 } }); - testLogger.info('Selected destination', { destinationName }); - - // ==================== DEDUPLICATION (Advanced tab) ==================== - await this._switchToAdvancedTab(); - - const dedupContainer = this.page.locator(this.locators.stepDeduplication); - await expect(dedupContainer).toBeVisible({ timeout: 5000 }); - testLogger.info('Deduplication container visible in Advanced tab'); - - // Configure fingerprint fields - if (fingerprintFields && fingerprintFields.length > 0) { - const fingerprintSelect = this.page.locator(this.locators.stepDeduplicationFingerprintSelect); - await fingerprintSelect.waitFor({ state: 'visible', timeout: 5000 }); - - for (const field of fingerprintFields) { - await fingerprintSelect.click(); - await this.page.waitForTimeout(500); - await fingerprintSelect.locator('input').fill(field); - await this.page.waitForTimeout(500); - - const fieldOption = this.page.getByRole('option', { name: field }).first(); - if (await fieldOption.isVisible({ timeout: 2000 })) { - await fieldOption.click(); - } else { - await this.page.keyboard.press('Enter'); - } - await this.page.waitForTimeout(300); - } - await this.page.locator('body').click({ position: { x: 10, y: 10 } }); - testLogger.info('Configured fingerprint fields', { fields: fingerprintFields }); - } - - // Configure time window - const timeWindowInput = this.page.locator(this.locators.stepDeduplicationTimeWindowInput); - await timeWindowInput.waitFor({ state: 'visible', timeout: 5000 }); - await timeWindowInput.clear(); - await timeWindowInput.fill(String(timeWindowMinutes)); - testLogger.info('Set dedup time window', { minutes: timeWindowMinutes }); - - // ==================== SUBMIT ALERT ==================== - await this.page.locator(this.locators.alertSubmitButton).click(); - // OToast renders the message in 3 elements (sr-only ARIA span, sr-only title div, - // visible message div) — scope to the visible `o-toast-message` data-test to avoid - // strict-mode violation per AGENT_RULES §2. - await expect( - this.page - .locator('[data-test="o-toast-message"]') - .filter({ hasText: this.locators.alertSuccessMessage }) - .first() - ).toBeVisible({ timeout: 30000 }); - testLogger.info('Successfully created scheduled alert with deduplication for validation', { alertName: randomAlertName }); - - return randomAlertName; - } - /** * Create a scheduled alert with aggregation enabled in Builder mode * PR #10470 moved aggregation from Step 4 (AlertSettings) to Step 2 (QueryConfig) * - * Wizard flow: Step 1 (Setup) -> Step 2 (Conditions + Aggregation) -> Step 3 (Compare) -> Step 4 (Settings) -> Step 5 (Dedup) -> Step 6 (Advanced) -> Submit + * Wizard flow: Step 1 (Setup) -> Step 2 (Conditions + Aggregation) -> Step 3 (Compare) -> Step 4 (Settings) -> Step 5 (Advanced) -> Submit * * @param {string} streamName - Name of the log stream * @param {string} destinationName - Name of the destination @@ -1703,7 +1216,7 @@ export class AlertCreationWizard { await this.page.locator('body').click({ position: { x: 10, y: 10 } }); testLogger.info('Configured alert settings'); - // Remaining advanced features (CompareWithPast, Dedup) are in Advanced tab + // Remaining advanced features (CompareWithPast) are in Advanced tab // but we skip them for this test // ==================== SUBMIT ==================== @@ -1923,7 +1436,7 @@ export class AlertCreationWizard { await this.page.locator('body').click({ position: { x: 10, y: 10 } }); testLogger.info('Selected destination', { destinationName }); - // Remaining advanced features (CompareWithPast, Dedup) are in Advanced tab + // Remaining advanced features (CompareWithPast) are in Advanced tab // but we skip them for this test // ==================== SUBMIT ALERT ==================== diff --git a/tests/ui-testing/pages/alertsPages/alertManagement.js b/tests/ui-testing/pages/alertsPages/alertManagement.js index 4e6803410f..d25864b91c 100644 --- a/tests/ui-testing/pages/alertsPages/alertManagement.js +++ b/tests/ui-testing/pages/alertsPages/alertManagement.js @@ -792,111 +792,17 @@ export class AlertManagement { } // ========================================================================= - // ALERT EDIT — DEDUPLICATION REGRESSION HELPERS + // ALERT EDIT HELPERS // ========================================================================= - /** - * Wait for the alert list page to be ready (canonical landmark). - * @param {number} timeout - */ - async waitForAlertListReady(timeout = 30000) { - await this.page.locator(this.locators.alertListPage).waitFor({ state: 'visible', timeout }); - } - - /** - * Open the alert edit screen by clicking the alert's update (pencil) button. - * The list row exposes `[data-test="alert-list-{name}-update-alert"]` (AlertList.vue:312). - * @param {string} alertName - */ - async openAlertEditByName(alertName) { - const updateBtn = this.page.locator(`[data-test="alert-list-${alertName}-update-alert"]`).first(); - await expect(updateBtn).toBeVisible({ timeout: 10000 }); - await updateBtn.click(); - // Wait for the AddAlert form to render — `add-alert-submit-btn` is the canonical landmark. - await this.page.locator(this.locators.alertSubmitButton).waitFor({ state: 'visible', timeout: 15000 }); - testLogger.info('Opened alert edit view', { alertName }); - } - /** * Activate the Advanced tab in the AddAlert v3 wizard. - * Deduplication lives in this tab (AddAlert.vue line ~273). */ async openAdvancedTab() { const advancedTab = this.page.locator(this.locators.addAlertTabAdvanced); await expect(advancedTab).toBeVisible({ timeout: 10000 }); await advancedTab.click(); - // Wait for the dedup fingerprint OSelect wrapper to mount in the Advanced tab. - await this.page.locator(this.locators.dedupFingerprintFieldsSelect).waitFor({ state: 'visible', timeout: 10000 }); - testLogger.info('Switched to Advanced tab — deduplication panel visible'); - } - - /** - * Read currently selected fingerprint field values from the OSelect trigger. - * OSelect exposes `data-test-selected-value` (comma-joined for multiple). - * @returns {Promise} - */ - async getSelectedFingerprintFields() { - const trigger = this.page.locator(this.locators.dedupFingerprintFieldsTrigger); - await expect(trigger).toBeVisible({ timeout: 5000 }); - const raw = await trigger.getAttribute('data-test-selected-value'); - if (!raw) return []; - return raw.split(',').map(v => v.trim()).filter(Boolean); - } - - /** - * Remove all currently selected fingerprint fields by opening the OSelect popover - * and clicking each previously selected option to toggle it OFF (multi-select). - * Returns the number of fields removed. - */ - async removeAllFingerprintFields() { - const trigger = this.page.locator(this.locators.dedupFingerprintFieldsTrigger); - await expect(trigger).toBeVisible({ timeout: 5000 }); - - const selectedBefore = await this.getSelectedFingerprintFields(); - testLogger.info('Removing fingerprint fields — initial selection', { selected: selectedBefore }); - - if (selectedBefore.length === 0) { - testLogger.warn('No fingerprint fields selected to remove'); - return 0; - } - - await trigger.click(); - const popover = this.page.locator(this.locators.dedupFingerprintFieldsPopover); - await expect(popover).toBeVisible({ timeout: 5000 }); - - let removed = 0; - for (const value of selectedBefore) { - // Per-value option uses `data-test-value=""` - const option = popover.locator(`[data-test="alert-dedup-fingerprint-fields-option"][data-test-value="${value}"]`).first(); - await expect(option).toBeVisible({ timeout: 5000 }); - await option.click(); - removed += 1; - // Wait for the trigger's selected-value attribute to no longer include this value. - await expect.poll(async () => { - const current = await trigger.getAttribute('data-test-selected-value'); - if (!current) return []; - return current.split(',').map(v => v.trim()).filter(Boolean); - }, { timeout: 5000 }).not.toContain(value); - testLogger.info('Deselected fingerprint option', { value, removed }); - } - - // Close the popover via keyboard — Escape is the canonical dismissal. - await this.page.keyboard.press('Escape'); - await popover.waitFor({ state: 'hidden', timeout: 5000 }); - - return removed; - } - - /** - * Submit the AddAlert form (Save button) and wait for the success toast. - * @returns {Promise} - */ - async submitAlertEdit() { - const submitBtn = this.page.locator(this.locators.alertSubmitButton); - await expect(submitBtn).toBeEnabled({ timeout: 10000 }); - await submitBtn.click(); - // OToast variant=success carries data-test="o-toast-success"; data-test-message holds the text. - await this.page.locator(this.locators.oToastSuccess).first().waitFor({ state: 'visible', timeout: 15000 }); - testLogger.info('Alert update submitted — success toast visible'); + await this.page.locator(this.locators.advancedTemplateOverrideSelect).waitFor({ state: 'visible', timeout: 10000 }); + testLogger.info('Switched to Advanced tab'); } } diff --git a/tests/ui-testing/pages/alertsPages/alertsPage.js b/tests/ui-testing/pages/alertsPages/alertsPage.js index 070645fcc5..7610c740ab 100644 --- a/tests/ui-testing/pages/alertsPages/alertsPage.js +++ b/tests/ui-testing/pages/alertsPages/alertsPage.js @@ -106,14 +106,7 @@ export class AlertsPage { multiTimeRangeDeleteButton: '[data-test="multi-time-range-alerts-delete-btn"]', goToViewEditorButton: '[data-test="go-to-view-editor-btn"]', - // Step 5: Deduplication (Scheduled only — in Advanced tab). Use data-test - // hooks from Deduplication.vue; the old `.step-deduplication .alert-v3-select` - // class chain no longer exists on newer builds (e.g. alpha pre-cloud). - stepDeduplication: '[data-test="alert-dedup-fingerprint-fields"]', - stepDeduplicationFingerprintSelect: '[data-test="alert-dedup-fingerprint-fields"]', - stepDeduplicationTimeWindowInput: '[data-test="alert-dedup-time-window-field"]', - - // Step 6: Advanced settings + // Step 5: Advanced settings contextAttributesAddButton: '[data-test="alert-variables-add-btn"]', contextAttributeKeyInput: '[data-test="alert-variables-key-input"]', contextAttributeValueInput: '[data-test="alert-variables-value-input"]', @@ -334,16 +327,6 @@ export class AlertsPage { addAlertTabCondition: '[data-test="add-alert-tab-condition"]', addAlertTabAdvanced: '[data-test="add-alert-tab-advanced"]', - // v3 Deduplication step (Advanced tab) - // OSelect wrapper has data-test, popover auto-derived suffix `-popover`, - // options auto-derived suffix `-option` with `data-test-value=""` per item. - dedupFingerprintFieldsSelect: '[data-test="alert-dedup-fingerprint-fields"]', - dedupFingerprintFieldsTrigger: '[data-test="alert-dedup-fingerprint-fields-trigger"]', - dedupFingerprintFieldsPopover: '[data-test="alert-dedup-fingerprint-fields-popover"]', - dedupFingerprintFieldsOption: '[data-test="alert-dedup-fingerprint-fields-option"]', - dedupTimeWindowInput: '[data-test="alert-dedup-time-window"]', - dedupTimeWindowInputField: '[data-test="alert-dedup-time-window-field"]', - // OToast (alerts module - success / error variants) oToastSuccess: '[data-test-variant="success"]', oToastError: '[data-test-variant="error"]', @@ -758,18 +741,6 @@ export class AlertsPage { return result; } - async createScheduledAlertWithDeduplication(streamName, destinationName, randomValue, dedupConfig = {}) { - const result = await this.creationWizard.createScheduledAlertWithDeduplication(streamName, destinationName, randomValue, dedupConfig); - this.currentAlertName = result; - return result; - } - - async createScheduledAlertWithDedupForValidation(streamName, column, value, destinationName, randomValue, dedupConfig = {}) { - const result = await this.creationWizard.createScheduledAlertWithDedupForValidation(streamName, column, value, destinationName, randomValue, dedupConfig); - this.currentAlertName = result; - return result; - } - /** * Create a scheduled alert with PromQL query for metrics streams * Tests fix for bug #9967 - cannot save alert when selecting PromQL mode @@ -806,14 +777,6 @@ export class AlertsPage { return this.creationWizard.deleteCondition(index); } - async verifyDeduplicationStepVisible() { - return this.creationWizard.verifyDeduplicationStepVisible(); - } - - async getDeduplicationConfig() { - return this.creationWizard.getDeduplicationConfig(); - } - // ==================== DELEGATE TO MANAGEMENT ==================== async updateAlert(alertName) { @@ -908,32 +871,12 @@ export class AlertsPage { return this.management.verifyAlertTriggerInValidationStream(alertName, validationStreamName, logsPage, waitTimeMs); } - // ==================== ALERT EDIT — DEDUP REGRESSION ==================== - - async waitForAlertListReady(timeout) { - return this.management.waitForAlertListReady(timeout); - } - - async openAlertEditByName(alertName) { - return this.management.openAlertEditByName(alertName); - } + // ==================== ALERT EDIT ==================== async openAdvancedTab() { return this.management.openAdvancedTab(); } - async getSelectedFingerprintFields() { - return this.management.getSelectedFingerprintFields(); - } - - async removeAllFingerprintFields() { - return this.management.removeAllFingerprintFields(); - } - - async submitAlertEdit() { - return this.management.submitAlertEdit(); - } - // ==================== ALERT DETAIL PAGE ==================== /** @@ -4321,7 +4264,7 @@ export class AlertsPage { * Navigate through wizard steps by clicking Continue button multiple times * @param {number} times - Number of times to click Continue */ - async navigateThroughWizardSteps(times = 5) { + async navigateThroughWizardSteps(times = 4) { for (let i = 0; i < times; i++) { const clicked = await this.clickContinueButton(); if (!clicked) break; diff --git a/tests/ui-testing/pages/apiCleanup.js b/tests/ui-testing/pages/apiCleanup.js index 38a90e388d..0da1f27f20 100644 --- a/tests/ui-testing/pages/apiCleanup.js +++ b/tests/ui-testing/pages/apiCleanup.js @@ -3650,93 +3650,6 @@ class APICleanup { } } - /** - * ========================================== - * CORRELATION SETTINGS CLEANUP METHODS - * ========================================== - */ - - /** - * Clean up test semantic groups created by correlation settings tests - * Uses the alerts deduplication config API (same as frontend) - * Test groups created by ensureSemanticGroupsExist(): k8s-cluster, k8s-namespace, k8s-deployment, service - * @param {Array} groupIds - Array of semantic group IDs to delete (default: test group IDs) - */ - async cleanupCorrelationSettings(groupIds = ['k8s-cluster', 'k8s-namespace', 'k8s-deployment', 'service']) { - testLogger.info('Starting correlation settings cleanup', { groupIds }); - - try { - // First, fetch the current deduplication config - const getResponse = await this._fetch(`${this.baseUrl}/api/${this.org}/alerts/deduplication/config`, { - method: 'GET', - headers: { - 'Authorization': this.authHeader, - 'Content-Type': 'application/json' - } - }); - - if (!getResponse.ok) { - if (getResponse.status === 404) { - testLogger.info('Deduplication config not found (nothing to clean up)'); - return; - } - testLogger.warn('Failed to fetch deduplication config', { status: getResponse.status }); - return; - } - - const config = await getResponse.json(); - const currentGroups = config.semantic_field_groups || []; - testLogger.info('Current semantic groups in dedup config', { count: currentGroups.length }); - - if (currentGroups.length === 0) { - testLogger.info('No semantic groups to clean up'); - return; - } - - // Filter out the test groups - const remainingGroups = currentGroups.filter(g => !groupIds.includes(g.id)); - const removedCount = currentGroups.length - remainingGroups.length; - - if (removedCount === 0) { - testLogger.info('No test semantic groups found to clean up'); - return; - } - - testLogger.info('Removing test semantic groups', { - total: currentGroups.length, - removing: removedCount, - remaining: remainingGroups.length - }); - - // Update the config with remaining groups - const updatePayload = { - ...config, - semantic_field_groups: remainingGroups, - // Also clean up fingerprint groups that reference removed semantic groups - alert_fingerprint_groups: (config.alert_fingerprint_groups || []) - .filter(id => !groupIds.includes(id)) - }; - - const updateResponse = await this._fetch(`${this.baseUrl}/api/${this.org}/alerts/deduplication/config`, { - method: 'POST', - headers: { - 'Authorization': this.authHeader, - 'Content-Type': 'application/json' - }, - body: JSON.stringify(updatePayload) - }); - - if (updateResponse.ok) { - testLogger.info('Correlation settings cleanup completed', { removedGroups: removedCount }); - } else { - testLogger.warn('Failed to update deduplication config', { status: updateResponse.status }); - } - - } catch (error) { - testLogger.error('Correlation settings cleanup failed', { error: error.message }); - } - } - /** * IAM leftovers a CRASHED run orphaned: roles, groups, users and service accounts * under `prefix`. diff --git a/tests/ui-testing/pages/commonActions.js b/tests/ui-testing/pages/commonActions.js index 84ebe331db..0d0fa2dcf2 100644 --- a/tests/ui-testing/pages/commonActions.js +++ b/tests/ui-testing/pages/commonActions.js @@ -657,92 +657,6 @@ export class CommonActions { return `Basic ${credentials}`; } - /** - * Count alert notifications in validation stream - * Used for deduplication testing to verify suppression of duplicate alerts - * @param {string} validationStreamName - Name of the validation stream to query - * @param {string} alertName - Alert name to search for (partial match) - * @param {number} timeRangeMinutes - How far back to search (default: 15 minutes) - * @returns {Promise<{success: boolean, count: number, data: any[]}>} - */ - async countAlertNotificationsInStream(validationStreamName, alertName, timeRangeMinutes = 15) { - const headers = getAuthHeaders(); - const baseUrl = process.env["ZO_BASE_URL"]; - const orgName = getOrgIdentifier(); - - const query = `SELECT * FROM "${validationStreamName}" WHERE str_match_ignore_case(text, '${alertName}')`; - - const endTime = Date.now() * 1000; // microseconds - const startTime = endTime - (timeRangeMinutes * 60 * 1000 * 1000); // microseconds - - const searchPayload = { - query: { - sql: query, - start_time: startTime, - end_time: endTime, - from: 0, - size: 1000 - } - }; - - try { - const response = await this.page.request.post(`${baseUrl}/api/${orgName}/_search?type=logs`, { - headers, - data: searchPayload - }); - const responseData = await response.json().catch(() => null); - const hits = responseData?.hits || []; - testLogger.debug(`Count notifications for "${alertName}" in ${validationStreamName}: found ${hits.length} notifications`); - return { - success: true, - count: hits.length, - data: hits - }; - } catch (error) { - testLogger.error('Error counting notifications in stream', { error: error.message }); - return { success: false, count: 0, data: [] }; - } - } - - /** - * Wait for exact notification count in validation stream - * Used for deduplication testing to verify the expected number of notifications - * @param {string} validationStreamName - Name of the validation stream - * @param {string} alertName - Alert name to search for - * @param {number} expectedCount - Expected number of notifications - * @param {number} maxWaitSeconds - Maximum time to wait (default: 120 seconds) - * @param {number} pollIntervalSeconds - How often to poll (default: 10 seconds) - * @returns {Promise<{match: boolean, actualCount: number, attempts: number}>} - */ - async waitForExactNotificationCount(validationStreamName, alertName, expectedCount, maxWaitSeconds = 120, pollIntervalSeconds = 10) { - const maxAttempts = Math.ceil(maxWaitSeconds / pollIntervalSeconds); - - testLogger.info(`Waiting for exactly ${expectedCount} notifications for "${alertName}" in "${validationStreamName}" (max ${maxWaitSeconds}s)...`); - - for (let attempt = 1; attempt <= maxAttempts; attempt++) { - const result = await this.countAlertNotificationsInStream(validationStreamName, alertName, 15); - - if (result.success && result.count === expectedCount) { - testLogger.info(`Found expected ${expectedCount} notifications after ${attempt * pollIntervalSeconds}s`); - return { match: true, actualCount: result.count, attempts: attempt }; - } - - if (result.count > expectedCount) { - testLogger.warn(`Found more notifications than expected: ${result.count} > ${expectedCount}`); - return { match: false, actualCount: result.count, attempts: attempt }; - } - - if (attempt < maxAttempts) { - testLogger.debug(`Attempt ${attempt}/${maxAttempts}: Found ${result.count} notifications, expected ${expectedCount}, waiting ${pollIntervalSeconds}s...`); - await new Promise(resolve => setTimeout(resolve, pollIntervalSeconds * 1000)); - } - } - - const finalResult = await this.countAlertNotificationsInStream(validationStreamName, alertName, 15); - testLogger.info(`Final count: ${finalResult.count} notifications (expected ${expectedCount})`); - return { match: finalResult.count === expectedCount, actualCount: finalResult.count, attempts: maxAttempts }; - } - /** * Find elements containing specific text (case-sensitive) * @param {string} text - Text to search for diff --git a/tests/ui-testing/pages/dashboardPages/dashboard-chart.js b/tests/ui-testing/pages/dashboardPages/dashboard-chart.js index c69662880a..bcd8235066 100644 --- a/tests/ui-testing/pages/dashboardPages/dashboard-chart.js +++ b/tests/ui-testing/pages/dashboardPages/dashboard-chart.js @@ -89,6 +89,50 @@ export default class ChartTypeSelector { return this.htmlRenderer.locator("script"); } + getHtmlRendererElement(selector) { + return this.htmlRenderer.locator(selector); + } + + // Page-wide, since an escaped element would land outside the renderer + getInjectedOnErrorImages() { + return this.page.locator("img[onerror]"); + } + + async getHtmlPanelSentinel() { + return this.page.evaluate(() => window.__o2HtmlPanelSentinel); + } + + async addHtmlPanelContent(html) { + await this.selectChartType("html"); + await this.fillHtmlEditor(html); + } + + // A null target asserts the sanitizer dropped it and added no rel + async expectHtmlLinkTarget(selector, target) { + const element = this.getHtmlRendererElement(selector); + if (target === null) { + await expect(element).toHaveCount(1); + await expect(element).not.toHaveAttribute("target"); + await expect(element).not.toHaveAttribute("rel"); + return; + } + await expect(element).toHaveAttribute("target", target); + await expect(element).toHaveAttribute("rel", "noopener noreferrer"); + } + + async clickHtmlRendererElementForPopup(selector) { + const [popup] = await Promise.all([ + this.page.context().waitForEvent("page"), + this.getHtmlRendererElement(selector).click(), + ]); + await popup.waitForLoadState("domcontentloaded"); + return popup; + } + + async isPopupOpenerNull(popup) { + return popup.evaluate(() => window.opener === null); + } + // Returns the field-list row for a given field name getFieldListRow(fieldName) { return this.page.locator(`[data-test="o-field-list-row-${fieldName}"]`); diff --git a/tests/ui-testing/pages/generalPages/correlationSettingsPage.js b/tests/ui-testing/pages/generalPages/correlationSettingsPage.js index 4cb9822845..e26034f940 100644 --- a/tests/ui-testing/pages/generalPages/correlationSettingsPage.js +++ b/tests/ui-testing/pages/generalPages/correlationSettingsPage.js @@ -7,9 +7,9 @@ const testLogger = require('../../playwright-tests/utils/test-logger.js'); * Route: /web/settings/correlation?org_identifier={orgId} * * Tabs: - * - Service Identity (identity) * - Discovered Services (services) - * - Alert Correlation (alert-correlation) + * - Detection Rules (discovery) + * - Field Mappings (field-aliases) */ export class CorrelationSettingsPage { constructor(page) { @@ -20,7 +20,7 @@ export class CorrelationSettingsPage { this.pageSubtitle = '.general-page-subtitle'; // Tab selectors (OTabs - render as [role="tablist"] / [role="tab"]) - // Note: tabs use name attribute: services, discovery, alert-correlation, field-aliases + // Note: tabs use name attribute: services, discovery, field-aliases this.tabsContainer = '[role="tablist"]'; // The Correlation Settings tabs wrapper (data-test container). this.correlationSettingsTabs = '[data-test="correlation-settings-tabs"]'; @@ -29,11 +29,10 @@ export class CorrelationSettingsPage { // Placeholder text of the "+ Add field" OSelect trigger. this.selectAFieldText = 'Select a field'; // Visible tab labels (post-revamp): the tab `name` attrs are - // services / discovery / alert-correlation / field-aliases, rendered as + // services / discovery / field-aliases, rendered as // these i18n labels. this.servicesTabName = 'Discovered Services'; this.serviceDiscoveryTabName = 'Detection Rules'; - this.alertCorrelationTabName = 'Alert Grouping'; this.fieldAliasesTabName = 'Field Mappings'; // Detection Rules view (the "Distinguish Services By" configuration tab) this.detectionRulesTabName = 'Detection Rules'; @@ -51,12 +50,6 @@ export class CorrelationSettingsPage { this.retryDiscoveredServicesBtn = '[data-test="retry-discovered-services-btn"]'; this.refreshDiscoveredServicesBtn = '[data-test="refresh-discovered-services-btn"]'; - // ==================== Alert Correlation Tab Selectors ==================== - this.dedupSettingsRefreshBtn = '[data-test="dedup-settings-refresh-btn"]'; - this.organizationDedupEnableCheckbox = '[data-test="organization-deduplication-enable-checkbox"]'; - this.crossAlertEnableCheckbox = '[data-test="organizationdeduplication-enable-cross-alert-checkbox"]'; - this.defaultWindowInput = '[data-test="organizationdeduplication-default-window-input"]'; - // ==================== Semantic Field Groups Selectors ==================== this.semanticGroupCategorySelect = '[data-test="semantic-group-category-select"]'; this.importJsonBtn = '[data-test="correlation-semanticfieldgroup-import-json-btn"]'; @@ -79,101 +72,6 @@ export class CorrelationSettingsPage { this.notification = '[role="alert"]'; } - // ==================== Test Data Setup ==================== - - /** - * Ensure semantic groups exist in the backend for testing fingerprint checkboxes - * Adds any missing test groups to the org's semantic groups (GET, merge, PUT), keeping every existing group - * @param {string} orgId - Organization identifier - * @returns {Promise} - True if successful - */ - async ensureSemanticGroupsExist(orgId) { - const baseUrl = process.env["ZO_BASE_URL"]; - const username = process.env["ZO_ROOT_USER_EMAIL"]; - const password = process.env["ZO_ROOT_USER_PASSWORD"]; - const authHeader = 'Basic ' + Buffer.from(`${username}:${password}`).toString('base64'); - - // Define test semantic groups that cover common use cases. - // Post-revamp shape: {id, display, group, fields} only — the per-group - // flags (is_scope / is_stable / normalize) were removed. - const testSemanticGroups = [ - { - id: 'k8s-cluster', - display: 'K8s Cluster', - group: 'Kubernetes', - fields: ['k8s.cluster.name', 'k8s_cluster_name'], - }, - { - id: 'k8s-namespace', - display: 'K8s Namespace', - group: 'Kubernetes', - fields: ['k8s.namespace.name', 'k8s_namespace_name'], - }, - { - id: 'k8s-deployment', - display: 'K8s Deployment', - group: 'Kubernetes', - fields: ['k8s.deployment.name', 'k8s_deployment_name'], - }, - { - id: 'service', - display: 'Service', - group: 'Common', - fields: ['service.name', 'service_name', 'service'], - } - ]; - - testLogger.info('Setting up test semantic groups via API', { orgId, groupCount: testSemanticGroups.length }); - - try { - const url = `${baseUrl}/api/${orgId}/alerts/deduplication/semantic-groups`; - const headers = { - 'Authorization': authHeader, - 'Content-Type': 'application/json' - }; - - // PUT replaces the whole list, so writing only the test groups would delete the org's defaults (e.g. `host`) for every other spec on the server. - const getResponse = await fetch(url, { headers }); - if (!getResponse.ok) { - testLogger.error('GET semantic-groups failed', { status: getResponse.status }); - return false; - } - const current = await getResponse.json(); - const currentGroups = Array.isArray(current) ? current : current.groups || []; - const currentIds = new Set(currentGroups.map((g) => g.id)); - const missing = testSemanticGroups.filter((g) => !currentIds.has(g.id)); - if (missing.length === 0) { - testLogger.info('Test semantic groups already present', { orgId, total: currentGroups.length }); - return true; - } - - testLogger.info('PUT semantic groups', { url, adding: missing.map((g) => g.id) }); - const putResponse = await fetch(url, { - method: 'PUT', - headers, - body: JSON.stringify([...currentGroups, ...missing]) - }); - - const putText = await putResponse.text(); - testLogger.info('PUT semantic-groups response', { - status: putResponse.status, - statusText: putResponse.statusText, - body: putText.substring(0, 500) - }); - - // Note: We intentionally do NOT POST to /config here. - // The config endpoint doesn't persist semantic_field_groups anyway (GET returns 0), - // and concurrent config POSTs in parallel tests cause race conditions where - // loadConfig() overwrites UI checkbox state with stale API values. - // The PUT above is the single source of truth for semantic groups. - - return putResponse.ok; - } catch (error) { - testLogger.error('Failed to setup semantic groups', { error: error.message, stack: error.stack }); - return false; - } - } - // ==================== Navigation ==================== async navigateToCorrelationSettings(orgId) { @@ -195,12 +93,10 @@ export class CorrelationSettingsPage { // Use getByRole for tabs or filter by text const servicesTab = this.page.getByRole('tab', { name: this.servicesTabName }); const serviceDiscoveryTab = this.page.getByRole('tab', { name: this.serviceDiscoveryTabName }); - const alertCorrelationTab = this.page.getByRole('tab', { name: this.alertCorrelationTabName }); const fieldAliasesTab = this.page.getByRole('tab', { name: this.fieldAliasesTabName }); await expect(servicesTab).toBeVisible({ timeout: 15000 }); await expect(serviceDiscoveryTab).toBeVisible(); - await expect(alertCorrelationTab).toBeVisible(); await expect(fieldAliasesTab).toBeVisible(); } @@ -223,22 +119,6 @@ export class CorrelationSettingsPage { await this.clickServicesTab(); } - async clickAlertCorrelationTab() { - const tab = this.page.getByRole('tab', { name: this.alertCorrelationTabName }); - // Wait for semantic-groups API response that loadConfig() triggers on component mount. - // This prevents a race condition where the test interacts with checkboxes before - // loadConfig() completes and overwrites localConfig with API values. - const semanticGroupsPromise = this.page.waitForResponse( - resp => resp.url().includes('/alerts/deduplication/semantic-groups') && resp.status() === 200, - { timeout: 15000 } - ).catch(() => { - testLogger.info('Semantic groups API response not intercepted (may have already fired)'); - }); - await tab.click(); - await semanticGroupsPromise; - await this.page.waitForLoadState('networkidle', { timeout: 10000 }).catch(() => {}); - } - async expectServiceDiscoveryTabActive() { const tab = this.page.getByRole('tab', { name: this.serviceDiscoveryTabName }); await expect(tab).toHaveAttribute('aria-selected', 'true', { timeout: 5000 }); @@ -249,11 +129,6 @@ export class CorrelationSettingsPage { await expect(tab).toHaveAttribute('aria-selected', 'true', { timeout: 5000 }); } - async expectAlertCorrelationTabActive() { - const tab = this.page.getByRole('tab', { name: this.alertCorrelationTabName }); - await expect(tab).toHaveAttribute('aria-selected', 'true', { timeout: 5000 }); - } - // ==================== Tab / Button Getters (raw-selector relocation) ==================== /** @@ -484,93 +359,6 @@ export class CorrelationSettingsPage { await expect(refreshBtn).toBeVisible({ timeout: 15000 }); } - // ==================== Alert Correlation Tab Actions ==================== - - async expectAlertCorrelationContentVisible() { - await expect(this.page.locator(this.organizationDedupEnableCheckbox)).toBeVisible({ timeout: 15000 }); - } - - async clickEnableDeduplicationCheckbox() { - await this.page.locator(this.organizationDedupEnableCheckbox).click(); - } - - async clickEnableCrossAlertCheckbox() { - await this.page.locator(this.crossAlertEnableCheckbox).click(); - } - - async expectCrossAlertCheckboxVisible() { - await expect(this.page.locator(this.crossAlertEnableCheckbox)).toBeVisible({ timeout: 5000 }); - } - - async expectCrossAlertCheckboxHidden() { - await expect(this.page.locator(this.crossAlertEnableCheckbox)).not.toBeVisible({ timeout: 5000 }); - } - - /** - * Check if the cross-alert deduplication checkbox is currently checked - * @returns {Promise} True if checkbox is checked - */ - async isCrossAlertCheckboxChecked() { - const checkbox = this.page.locator(this.crossAlertEnableCheckbox); - // The checkbox uses aria-checked attribute - const ariaChecked = await checkbox.getAttribute('aria-checked'); - if (ariaChecked !== null) { - return ariaChecked === 'true'; - } - // Fallback to checking the inner input if visible - const input = checkbox.locator('input[type="checkbox"]'); - if (await input.count() > 0) { - return await input.isChecked(); - } - // Last resort - check for checked class - const hasCheckedClass = await checkbox.evaluate(el => { - return el.classList.contains('checked') || - el.closest('[role="checkbox"]')?.getAttribute('aria-checked') === 'true'; - }).catch(() => false); - return hasCheckedClass; - } - - async fillTimeWindowInput(minutes) { - // OFormInput forwards data-test to a wrapper
; the real control is - // the inner . - const wrap = this.page.locator(this.defaultWindowInput); - await expect(wrap).toBeVisible({ timeout: 5000 }); - await wrap.locator('input').first().fill(String(minutes)); - } - - async getTimeWindowValue() { - return await this.page.locator(this.defaultWindowInput).locator('input').first().inputValue(); - } - - async clickRefreshDedupSettings() { - await this.page.locator(this.dedupSettingsRefreshBtn).click(); - await this.page.waitForLoadState('networkidle', { timeout: 10000 }).catch(() => {}); - } - - async expectEnableDedupCheckboxVisible() { - await expect(this.page.locator(this.organizationDedupEnableCheckbox)).toBeVisible({ timeout: 10000 }); - } - - async expectTimeWindowInputVisible() { - await expect(this.page.locator(this.defaultWindowInput)).toBeVisible({ timeout: 10000 }); - } - - async expectDedupRefreshButtonVisible() { - await expect(this.page.locator(this.dedupSettingsRefreshBtn)).toBeVisible({ timeout: 10000 }); - } - - // ==================== Fingerprint Checkbox Actions ==================== - - async clickFingerprintCheckbox(groupId) { - const selector = `[data-test="organizationdeduplication-fingerprint-${groupId}-checkbox"]`; - await this.page.locator(selector).click(); - } - - async expectFingerprintCheckboxVisible(groupId) { - const selector = `[data-test="organizationdeduplication-fingerprint-${groupId}-checkbox"]`; - await expect(this.page.locator(selector)).toBeVisible({ timeout: 5000 }); - } - // ==================== Semantic Field Groups Actions ==================== async clickCategorySelect() { @@ -707,35 +495,6 @@ export class CorrelationSettingsPage { await this.page.waitForLoadState('networkidle', { timeout: 30000 }).catch(() => {}); } - async isDedupCheckboxChecked() { - const checkbox = this.page.locator(this.organizationDedupEnableCheckbox); - // Checkboxes use aria-checked or have a specific class when checked - const isChecked = await checkbox.getAttribute('aria-checked'); - if (isChecked !== null) { - return isChecked === 'true'; - } - // Fallback: check for checked class - const classList = await checkbox.getAttribute('class'); - return classList?.includes('checked'); - } - - // ==================== Alert Correlation Save Button ==================== - - async clickSaveAlertCorrelation() { - // The save button in Alert Correlation tab - const saveBtn = this.page.getByRole('button', { name: /save/i }).last(); - await saveBtn.click(); - await this.page.waitForLoadState('networkidle', { timeout: 10000 }).catch(() => {}); - } - - async expectAlertCorrelationSaveSuccess() { - // Wait for positive notification about saved settings - const notification = this.page.locator('[role="alert"]'); - await expect(notification).toBeVisible({ timeout: 10000 }); - const text = await notification.textContent(); - return text?.toLowerCase().includes('saved') || text?.toLowerCase().includes('success'); - } - // ==================== Phase 1 High Priority - Dimension Management ==================== /** @@ -816,56 +575,7 @@ export class CorrelationSettingsPage { await expect(item).not.toBeVisible({ timeout: 5000 }); } - // ==================== Phase 1 High Priority - Fingerprint Groups ==================== - - /** - * Get all fingerprint group checkboxes - * @returns {import('@playwright/test').Locator} - */ - getFingerprintGroupCheckboxes() { - return this.page.locator('[data-test^="organizationdeduplication-fingerprint-"]'); - } - - /** - * Expect fingerprint groups section to be visible - */ - async expectFingerprintGroupsVisible() { - const checkboxes = this.getFingerprintGroupCheckboxes(); - await expect(checkboxes.first()).toBeVisible({ timeout: 10000 }); - } - - /** - * Expect fingerprint groups section to be hidden - */ - async expectFingerprintGroupsHidden() { - const checkboxes = this.page.locator('[data-test^="organizationdeduplication-fingerprint-"]'); - await expect(checkboxes.first()).not.toBeVisible({ timeout: 5000 }); - } - - /** - * Get count of fingerprint group checkboxes - * @returns {Promise} - */ - async getFingerprintGroupsCount() { - const checkboxes = this.getFingerprintGroupCheckboxes(); - return await checkboxes.count(); - } - - /** - * Check if a specific fingerprint group is checked - * @param {string} groupId - The group ID - * @returns {Promise} - */ - async isFingerprintGroupChecked(groupId) { - const selector = `[data-test="organizationdeduplication-fingerprint-${groupId}-checkbox"]`; - const checkbox = this.page.locator(selector); - const isChecked = await checkbox.getAttribute('aria-checked'); - if (isChecked !== null) { - return isChecked === 'true'; - } - const classList = await checkbox.getAttribute('class'); - return classList?.includes('checked'); - } + // ==================== Phase 1 High Priority - Validation Notifications ==================== /** * Expect validation error notification to be visible @@ -880,14 +590,6 @@ export class CorrelationSettingsPage { } } - /** - * Click the Cancel button in Alert Correlation tab - */ - async clickCancelAlertCorrelation() { - const cancelBtn = this.page.getByRole('button', { name: /cancel/i }); - await cancelBtn.click(); - } - // ==================== Phase 1 High Priority - Discovered Services ==================== /** diff --git a/tests/ui-testing/pages/iamPages/rolesPage.js b/tests/ui-testing/pages/iamPages/rolesPage.js index 782459ff9f..b7c3496554 100644 --- a/tests/ui-testing/pages/iamPages/rolesPage.js +++ b/tests/ui-testing/pages/iamPages/rolesPage.js @@ -93,6 +93,16 @@ export class RolesPage { this.summaryEmpty = page.locator('[data-test="edit-role-summary-empty"]'); this.summaryLoading = page.locator('[data-test="edit-role-summary-loading"]'); + // ---------- presets panel (Role Overview, every role) ---------- + // The panel shows either the preset list or one preset's detail, never both. + this.presetsPanel = page.locator('[data-test="edit-role-presets-panel"]'); + this.presetDetail = page.locator('[data-test="edit-role-preset-detail"]'); + this.presetBack = page.locator('[data-test="edit-role-preset-back"]'); + this.presetApply = page.locator('[data-test="edit-role-preset-apply"]'); + // A viewer can close the panel; the floating button reopens it, and each page visit starts open. + this.presetsClose = page.locator('[data-test="edit-role-presets-close"]'); + this.presetsOpen = page.locator('[data-test="edit-role-presets-open"]'); + // ---------- pane pagination (OTable's own controls, scoped to the pane) ---------- // MUST be scoped. GroupUsers and GroupServiceAccounts stay MOUNTED under // v-show in EditRole.vue, so an unscoped `o2-table-*` slug also matches the @@ -120,6 +130,10 @@ export class RolesPage { // ---------- unsaved drawer ---------- this.drawer = page.locator('[data-test="edit-role-unsaved-drawer"]'); + // The drawer's own Save; the page Save underneath is unclickable while it is open. + this.drawerSave = page.locator('[data-test="edit-role-unsaved-save"]'); + // "Not added yet": grants the role needs to work that are not staged. + this.drawerCompanions = page.locator('[data-test="edit-role-companions"]'); this.toastMessages = page.locator('[data-test="o-toast-message"]'); } @@ -274,9 +288,46 @@ export class RolesPage { return this.page.locator('[data-test^="edit-role-unsaved-undo-"]'); } - /** Preset cards in the empty state carry no slug of their own; match their label. */ - presetCard(label) { - return this.summaryEmpty.getByText(label, { exact: true }); + /** Companion row actions: Add stages the module-wide grant; Open (Streams Get only) opens the module instead. */ + companionAdd(moduleKey) { + return this.page.locator(`[data-test="edit-role-companion-add-${moduleKey}"]`); + } + companionOpen(moduleKey) { + return this.page.locator(`[data-test="edit-role-companion-open-${moduleKey}"]`); + } + + /** A row in the Presets panel, by preset id (readonly, dbm, k8s, rum_viewer, rum_editor, alert_manager, …). */ + presetRow(id) { + return this.page.locator(`[data-test="edit-role-preset-${id}"]`); + } + /** The row's "+N" or "In Role" badge. */ + presetStatus(id) { + return this.page.locator(`[data-test="edit-role-preset-${id}-status"]`); + } + + /** Reopens the Presets panel if a test closed it earlier on the same page visit. */ + async ensurePresetsOpen() { + await expect(this.presetsPanel.or(this.presetsOpen)).toBeVisible({ timeout: 15000 }); + if (await this.presetsOpen.isVisible()) await this.presetsOpen.click(); + await expect(this.presetsPanel).toBeVisible(); + } + + /** Opens a preset's detail in the panel. */ + async openPreset(id) { + await this.ensurePresetsOpen(); + await this.presetRow(id).click(); + await expect(this.presetDetail).toBeVisible({ timeout: 15000 }); + } + + /** + * Opens a preset and clicks Add. Add stays disabled until the preview is ready + * (stream presets load their lists first), so wait for it rather than click early. + */ + async applyPreset(id) { + await this.openPreset(id); + await expect(this.presetApply).toBeEnabled({ timeout: 30000 }); + await this.presetApply.click(); + await this.waitForPresetStaged(); } // ================= capability probes ================= @@ -448,8 +499,11 @@ export class RolesPage { // ================= save ================= - /** Resolves with the PUT body the save sent, or null when no request fired. */ - async saveAndCapture({ expectRequest = true } = {}) { + /** + * Resolves with the PUT body the save sent, or null when no request fired. + * `via` picks the Save button: the page footer's (default) or the drawer's `drawerSave`. + */ + async saveAndCapture({ expectRequest = true, via = this.saveButton } = {}) { let captured = null; const onRequest = (req) => { if (req.method() === 'PUT' && /\/roles\/[^/]+$/.test(new URL(req.url()).pathname)) { @@ -457,7 +511,7 @@ export class RolesPage { } }; this.page.on('request', onRequest); - await this.saveButton.click(); + await via.click(); // A save that legitimately sends nothing must not be papered over by a wait. await this.page.waitForTimeout(expectRequest ? 3000 : 2500); this.page.off('request', onRequest); @@ -496,8 +550,9 @@ export class RolesPage { * * It is an ODrawer — a modal dismissable layer — so while it is open its subtree * intercepts pointer events and Save is visible and enabled but UNCLICKABLE. It - * self-closes only once nothing is staged (S-05), so any test that visits the - * drawer and then saves has to close it first. + * self-closes only once nothing is staged and no companion grant is missing (S-05), + * so any test that visits the drawer and then saves has to close it first, or save + * through `drawerSave`. */ async closeDrawer() { if (!(await this.drawer.isVisible().catch(() => false))) return; diff --git a/tests/ui-testing/pages/metricsPages/metricsPage.js b/tests/ui-testing/pages/metricsPages/metricsPage.js index 1927111a6f..a4df477510 100644 --- a/tests/ui-testing/pages/metricsPages/metricsPage.js +++ b/tests/ui-testing/pages/metricsPages/metricsPage.js @@ -157,11 +157,12 @@ export class MetricsPage { this.shareErrorToast = page.locator('[data-test-variant="error"][data-test-message*="Error shortening link"]'); this.toastMessage = page.locator('[data-test="o-toast-message"]'); - // Query history drawer (QueryHistoryDrawer.vue) this.historyButton = page.locator('[data-test="metrics-history-btn"]'); - this.historyTable = page.locator('[data-test="metrics-history-table"]'); - this.historySearch = page.locator('[data-test="metrics-history-search"] input'); - this.historyStarredOnly = page.locator('[data-test="metrics-history-starred-only"]'); + this.historyPopover = page.locator('[data-test="recent-searches-popover"]'); + this.historyViewAll = this.historyPopover.locator('[data-test="recent-searches-view-all"]'); + this.historyDrawer = page.locator('[data-test="recent-searches-drawer"]'); + this.historySearch = this.historyDrawer.locator('[data-test="recent-searches-search"] input'); + this.historyStarredOnly = this.historyDrawer.locator('[data-test="recent-searches-starred-only"]'); } /** @@ -2793,7 +2794,8 @@ export class MetricsPage { async openQueryHistory() { await this.historyButton.click(); - await this.historyTable.waitFor({ state: 'visible', timeout: 15000 }); + await this.historyViewAll.click(); + await this.historyDrawer.waitFor({ state: 'visible', timeout: 15000 }); } /** Resolves once the filtered list answers: a click landing mid-re-render is lost. */ @@ -2805,24 +2807,33 @@ export class MetricsPage { await listed; } - /** The history row whose query cell is exactly `query`. */ + /** The drawer row whose query text contains `query`. */ historyRow(query) { - return this.historyTable.locator('[data-test^="o2-table-row-"]').filter({ - has: this.page.locator('[data-test="o2-table-cell-query"]', { hasText: query }), + return this.historyDrawer.locator('[data-test^="recent-searches-drawer-row-"]').filter({ + has: this.page.locator('[data-test="recent-searches-drawer-query"]', { hasText: query }), }); } /** A row's star toggle; its data-test carries the entry id, so match the prefix. */ historyStarButton(query) { - return this.historyRow(query).locator('[data-test^="metrics-history-star-"]'); + return this.historyRow(query).locator('[data-test^="recent-searches-star-"]'); } + /** Picks `action` (load, delete, copy-link, copy-query) from a row's "More" menu. */ + async historyRowAction(query, action) { + const rowTest = await this.historyRow(query).getAttribute('data-test'); + const id = rowTest.replace('recent-searches-drawer-row-', ''); + await this.historyRow(query).locator(`[data-test="recent-searches-more-${id}"]`).click(); + await this.page.locator(`[data-test="recent-searches-${action}-${id}"]`).click(); + } + + /** The DELETE is sent once the row's Undo toast closes, so callers wait for it. */ async deleteHistoryEntry(query) { - await this.historyRow(query).locator('[data-test^="metrics-history-delete-"]').click(); + await this.historyRowAction(query, 'delete'); } - /** Clicking a row's query cell loads the entry into the open editor. */ + /** Loads the entry into the editor without running it. */ async loadHistoryEntry(query) { - await this.historyRow(query).locator('[data-test="o2-table-cell-query"]').click(); + await this.historyRowAction(query, 'load'); } } \ No newline at end of file diff --git a/tests/ui-testing/pages/page-manager.js b/tests/ui-testing/pages/page-manager.js index 35bef1dd6c..f349553a97 100644 --- a/tests/ui-testing/pages/page-manager.js +++ b/tests/ui-testing/pages/page-manager.js @@ -87,6 +87,7 @@ import { MetricsExplorerPage } from "./metricsPages/metricsExplorerPage.js"; import { TracesPage } from "./tracesPages/tracesPage.js"; import { ServiceGraphPage } from "./tracesPages/serviceGraphPage.js"; import { ServicesCatalogPage } from "./tracesPages/servicesCatalogPage.js"; +import { ProfilesPage } from "./profilesPages/profilesPage.js"; import { DatabaseMonitoringPage } from "./dbmPages/databaseMonitoringPage.js"; import { RumPage } from "./rumPages/rumPage.js"; import { RumSessionsPage } from "./rumPages/rumSessionsPage.js"; @@ -285,6 +286,7 @@ class PageManager { this.tracesPage = new TracesPage(page); this.serviceGraphPage = new ServiceGraphPage(page); this.servicesCatalogPage = new ServicesCatalogPage(page); + this.profilesPage = new ProfilesPage(page); this.databaseMonitoringPage = new DatabaseMonitoringPage(page); this.rumPage = new RumPage(page); this.reportsPage = new ReportsPage(page); diff --git a/tests/ui-testing/pages/profilesPages/profilesPage.js b/tests/ui-testing/pages/profilesPages/profilesPage.js new file mode 100644 index 0000000000..b0928a4b63 --- /dev/null +++ b/tests/ui-testing/pages/profilesPages/profilesPage.js @@ -0,0 +1,1247 @@ +// Page object for Continuous Profiling (/web/profiles); selectors from web/src/plugins/profiles and FlameGraphView.vue. + +import { expect } from '@playwright/test'; +import { PNG } from 'pngjs'; + +const { getOrgIdentifier } = require('../../playwright-tests/utils/cloud-auth.js'); + +const FLAME_ROW_PX = 26; +const FLAME_PLOT_INSET_PX = 10; +const FLAME_TOOLTIP_Z = 'z-index: 9999999'; + +// Hand-computed frame geometry (percent of width) for the reference checkout cpu tree (total 12 s). +export const CHECKOUT_FRAMES = { + root: { depth: 0, left: 0, width: 100, path: 'root' }, + main: { depth: 1, left: 0, width: 98.75, path: 'root/0' }, + runtime_main: { depth: 1, left: 98.75, width: 1.25, path: 'root/1' }, + handle_order: { depth: 2, left: 0, width: 98.75, path: 'root/0/0' }, + gc_sweep: { depth: 2, left: 98.75, width: 1.25, path: 'root/1/0' }, + db_query: { depth: 3, left: 0, width: 50, path: 'root/0/0/0' }, + json_encode: { depth: 3, left: 50, width: 33.333, path: 'root/0/0/1' }, + process_payment: { depth: 3, left: 83.333, width: 2.083, path: 'root/0/0/2' }, +}; + +// Reference catalog cpu tree (total 5.515 s): catalog_main 4.515 s, [truncated] 1 s, 55 values below the 0.1 % draw floor. +export const CATALOG_FRAMES = { + root: { depth: 0, left: 0, width: 100 }, + catalog_main: { depth: 1, left: 0, width: 81.868 }, + truncated: { depth: 1, left: 81.868, width: 18.132 }, + wide_root: { depth: 2, left: 0, width: 81.868 }, + smallFrames: { depth: 3, left: 79.076, width: 2.792 }, +}; + +/** A frame spanning the whole plot at `depth` (zoomed frames and their dimmed callers). */ +export const fullWidthFrame = (depth) => ({ depth, left: 0, width: 100 }); + +export class ProfilesPage { + constructor(page) { + this.page = page; + + this.pageRoot = '[data-test="profiles-page"]'; + this.pageTitle = '[data-test="profiles-page-title"]'; + this.betaBadge = '[data-test="beta-badge"]'; + this.datePrevBtn = '[data-test="profiles-date-picker"] [data-test="date-time-prev-btn"]'; + this.dateNextBtn = '[data-test="profiles-date-picker"] [data-test="date-time-next-btn"]'; + this.dateBtn = '[data-test="profiles-date-picker"] [data-test="date-time-btn"]'; + this.shareBtn = '[data-test="profiles-share-btn"]'; + this.runBtn = '[data-test="profiles-run-query"]'; + this.cancelBtn = '[data-test="profiles-cancel-query"]'; + this.runOptionsBtn = '[data-test="profiles-run-options"]'; + this.autoRunToggle = '[data-test="profiles-auto-run-toggle"]'; + this.streamSelect = 'profiles-stream-select'; + this.serviceSelect = 'profiles-service-select'; + this.typeSelect = 'profiles-type-select'; + + this.addFilterBtn = '[data-test="profiles-add-filter"]'; + this.tagKeySelect = 'profiles-tag-key-select'; + this.tagValueSelect = 'profiles-tag-value-select'; + this.filterChips = '[data-test^="profiles-filter-chip-"]'; + + this.errorState = '[data-test="profiles-error"]'; + this.noStreamState = '[data-test="profiles-no-stream"]'; + this.searchCancelledState = '[data-test="profiles-search-cancelled"]'; + this.noResultsState = '[data-test="profiles-no-results"]'; + this.errorBanner = '[data-test="profiles-error-banner"]'; + this.errorRetryBtn = '[data-test="profiles-error-retry"]'; + this.partialResultBanner = '[data-test="profiles-partial-result"]'; + + this.seriesCard = '[data-test="profiles-series-card"]'; + this.seriesChart = '[data-test="profiles-series-card"] [data-test="chart-renderer"]'; + this.breakdownSelect = 'profiles-breakdown-select'; + this.breakdownLegend = '[data-test="profiles-breakdown-legend"]'; + this.breakdownChipOthers = '[data-test="profiles-breakdown-chip-others"]'; + this.seriesSkeleton = '[data-test="profiles-series-skeleton"]'; + this.noSeries = '[data-test="profiles-no-series"]'; + + this.searchInput = '[data-test="profiles-search-functions-field"]'; + this.searchSummary = '[data-test="profiles-search-summary"]'; + this.resultsExpandBtn = '[data-test="profiles-results-expand"]'; + this.topTable = '[data-test="profiles-top-table"]'; + this.stackTable = '[data-test="profiles-stack-table"]'; + this.flameSkeleton = '[data-test="profiles-flame-skeleton"]'; + this.sandwichFunction = 'profiles-sandwich-function'; + this.sandwichCoverage = '[data-test="profiles-sandwich-coverage"]'; + this.sandwichSkeleton = '[data-test="profiles-sandwich-skeleton"]'; + this.loadingIndicators = [ + '[data-test="profiles-series-skeleton"]', + '[data-test="profiles-flame-skeleton"]', + '[data-test="profiles-sandwich-skeleton"]', + '[data-test="profiles-top-table"] [data-test="o2-table"][data-test-loading="true"]', + '[data-test="profiles-stack-table"] [data-test="o2-table"][data-test-loading="true"]', + ].join(', '); + + this.details = '[data-test="profiles-function-details"]'; + this.detailsCopy = '[data-test="profiles-function-details-copy"]'; + this.detailsClose = '[data-test="profiles-function-details-close"]'; + this.detailsPackage = '[data-test="profiles-function-details-package"]'; + this.detailsSelf = '[data-test="profiles-function-details-self"]'; + this.detailsTotal = '[data-test="profiles-function-details-total"]'; + this.detailsCalledBy = '[data-test="profiles-function-details-called-by"]'; + this.detailsCalls = '[data-test="profiles-function-details-calls"]'; + this.detailsSandwich = '[data-test="profiles-function-details-sandwich"]'; + this.detailsShow = '[data-test="profiles-function-details-show"]'; + + this.compare = '[data-test="profiles-compare"]'; + this.compareLabel = 'profiles-compare-label'; + this.compareBaseline = 'profiles-compare-baseline'; + this.compareComparison = 'profiles-compare-comparison'; + this.compareSwap = '[data-test="profiles-compare-swap"]'; + this.compareSkeleton = '[data-test="profiles-compare-skeleton"]'; + this.compareMessage = '[data-test="profiles-compare-message"]'; + this.compareFlame = '[data-test="profiles-compare-flame"]'; + this.compareTable = '[data-test="profiles-compare-table"]'; + this.compareSideBaseline = '[data-test="profiles-compare-side-baseline"]'; + this.compareSideComparison = '[data-test="profiles-compare-side-comparison"]'; + + this.flameViewOptions = '[data-test="flame-graph-view-options"]'; + this.flameAlignLeft = '[data-test="flame-graph-align-left"]'; + this.flameAlignRight = '[data-test="flame-graph-align-right"]'; + this.flameRootAtBottom = '[data-test="flame-graph-root-at-bottom-btn"]'; + this.flamePlot = '[data-test="flame-graph-plot"]'; + this.flameFrameMore = '[data-test="flame-graph-frame-more"]'; + this.flameZoomHint = '[data-test="flame-graph-zoom-hint"]'; + this.flameZoomHintNever = '[data-test="flame-graph-zoom-hint-never"]'; + this.flameZoomHintClose = '[data-test="flame-graph-zoom-hint-close"]'; + this.flameMenu = '[data-test="flame-graph-frame-menu"]'; + + this.successToast = '[data-test-variant="success"]'; + this.menuProfiles = '[data-test="menu-link-/profiles-item"]'; + this.menuTraces = '[data-test="menu-link-/traces-item"]'; + this.menuLinks = '[data-test^="menu-link-"]'; + this.navRail = '[data-test="navbar-main-nav"]'; + + this.streamTypeFilter = '[data-test="log-stream-type-filter"]'; + this.streamSearchInput = '[data-test="streams-search-stream-input-field"]'; + this.streamExploreBtn = '[data-test="log-stream-explore-btn"]'; + this.ingestionProfilesTab = '[data-test="ingestion-custom-tab-ingestProfiles"]'; + this.ingestionOtelTab = '[data-test="ingestion-profiles-tab-otelCollector"]'; + this.ingestionOtelBlock = '[data-test="ingestion-profiles-otel-code-block-pre"]'; + this.ingestionOtelStartBlock = '[data-test="ingestion-profiles-otel-start-code-block-pre"]'; + this.ingestionOtelGrpcBlock = '[data-test="ingestion-profiles-otel-grpc-code-block-pre"]'; + this.traceViewProfileBtn = '[data-test="trace-details-sidebar-header-toolbar-view-profile-btn"]'; + } + + orgName() { + return getOrgIdentifier(); + } + + baseUrl() { + return (process.env['ZO_BASE_URL'] || '').replace(/\/+$/, ''); + } + + /** Patches /config and /api/{org}/config so Auto Run and the profiling flag do not depend on how the server was started. */ + async mockServerConfig({ autoQuery = false, profiling = true } = {}) { + await this.page.route(/\/config(\?.*)?$/, async (route) => { + const url = new URL(route.request().url()); + if (url.pathname !== '/config' && !/^\/api\/[^/]+\/config$/.test(url.pathname)) return route.fallback(); + const response = await route.fetch(); + const body = await response.json(); + await route.fulfill({ + response, + json: { ...body, auto_query_enabled: autoQuery, profiling_enabled: profiling }, + }); + }); + } + + /** Drops this page's routes so an in-flight mock cannot fail the test after it ends. */ + async teardown() { + await this.page.unrouteAll({ behavior: 'ignoreErrors' }); + } + + /** Hides the one-time flame zoom hint so it never covers a frame the test clicks. */ + async hideFlameZoomHint() { + await this.page.addInitScript(() => localStorage.setItem('o2_flame_zoom_hint_hidden', '1')); + } + + seededUrl({ stream, from, to, period, service, type = 'cpu', unit = 'nanoseconds', view = 'top', filters, extra = {} }) { + const params = new URLSearchParams({ org_identifier: this.orgName(), stream }); + if (from && to) { + params.set('from', String(from)); + params.set('to', String(to)); + } + if (period) params.set('period', period); + if (service) params.set('service_name', service); + if (type) { + params.set('profile_type', type); + params.set('profile_unit', unit); + } + if (filters) params.set('filters', filters); + if (view) params.set('view', view); + for (const [key, value] of Object.entries(extra)) params.set(key, value); + return `${this.baseUrl()}/web/profiles?${params.toString()}`; + } + + /** Opens the page on a seeded reference stream (`ref` from seedReferenceStream) and waits for the run. */ + async gotoRef(ref, { service = 'checkout', view = 'top', type, unit, filters, extra = {} } = {}) { + await this.gotoSeeded({ stream: ref.stream, ...ref.window, service, view, filters, extra, ...(type ? { type, unit } : {}) }); + } + + async gotoSeeded(options) { + await this.page.goto(this.seededUrl(options)); + await this.waitForPageReady(); + } + + /** Opens a seeded URL and returns once the page shell is up, without waiting for the run. */ + async openSeededNoWait(options) { + await this.page.goto(this.seededUrl(options)); + await expect(this.page.locator(this.pageRoot)).toBeVisible({ timeout: 30000 }); + } + + async gotoProfiles() { + await this.page.goto(`${this.baseUrl()}/web/profiles?org_identifier=${this.orgName()}`); + await this.waitForPageReady(); + } + + async gotoUrl(url) { + await this.page.goto(url); + await this.waitForPageReady(); + } + + async reload() { + await this.page.reload(); + await this.waitForPageReady(); + } + + async waitForPageReady() { + await expect(this.page.locator(this.pageRoot)).toBeVisible({ timeout: 30000 }); + await this.waitForRunIdle(); + } + + /** Waits until the query AND the page init (meta) are done: skeletons render while either runs, and init ignores input. */ + async waitForRunIdle() { + await expect(this.page.locator(this.cancelBtn)).toHaveCount(0, { timeout: 30000 }); + await expect(this.page.locator(this.runBtn)).toBeVisible({ timeout: 30000 }); + // Before the org is known nothing has started yet; a picked stream (or the no-stream state) proves init ran. + const initStarted = this.page.locator(`[data-test="${this.streamSelect}-trigger"]:not([data-test-selected-value=""])`); + await expect(initStarted.or(this.page.locator(this.noStreamState)).first()).toBeVisible({ timeout: 30000 }); + await expect(this.page.locator(this.loadingIndicators)).toHaveCount(0, { timeout: 30000 }); + } + + /** In-app navigation through the SPA router, as a menu click would do. */ + async routerPush(location) { + await this.page.evaluate((to) => { + const app = document.querySelector('#app')?.__vue_app__; + return app.config.globalProperties.$router.push(to); + }, location); + } + + currentQuery() { + return Object.fromEntries(new URL(this.page.url()).searchParams.entries()); + } + + async expectUrlParam(key, value) { + await expect.poll(() => this.currentQuery()[key], { timeout: 10000, message: `URL ${key}` }).toBe(value); + } + + async expectUrlParamAbsent(key) { + await expect.poll(() => key in this.currentQuery(), { timeout: 10000, message: `URL ${key} absent` }).toBe(false); + } + + async historyLength() { + return this.page.evaluate(() => window.history.length); + } + + trackRequests() { + const calls = { series: [], merge: [], meta: [], tag_values: [], diff: [] }; + this.page.on('request', (request) => { + const match = request.url().match(/\/profiles\/(series|merge|meta|tag_values|diff)(\?|$)/); + if (!match) return; + const body = request.method() === 'POST' ? request.postDataJSON() : null; + calls[match[1]].push({ url: request.url(), body }); + }); + return calls; + } + + /** Holds matching profiles API calls until the returned release() is called, so loading and cancel states stay on screen. */ + async holdApi(kind) { + let release; + const gate = new Promise((resolve) => { + release = resolve; + }); + await this.page.route(new RegExp(`/profiles/${kind}(\\?|$)`), async (route) => { + await gate; + await route.fallback().catch((error) => { + if (!/already handled|closed/i.test(error.message)) throw error; + }); + }); + return release; + } + + async mockApi(kind, { status = 200, json }) { + await this.page.route(new RegExp(`/profiles/${kind}(\\?|$)`), (route) => route.fulfill({ status, json })); + } + + /** Rewrites the real `kind` response with the fields `patch(body)` returns (e.g. truncated flags). */ + async patchApiResponse(kind, patch) { + await this.page.route(new RegExp(`/profiles/${kind}(\\?|$)`), async (route) => { + const response = await route.fetch(); + const body = await response.json(); + await route.fulfill({ response, json: { ...body, ...patch(body) } }); + }); + } + + /** Rewrites the merge request's max_nodes before it reaches the server. */ + async forceMergeMaxNodes(maxNodes) { + await this.page.route(/\/profiles\/merge(\?|$)/, async (route) => { + const body = route.request().postDataJSON(); + await route.fallback({ postData: JSON.stringify({ ...body, max_nodes: maxNodes }) }); + }); + } + + /** Answers the profiles stream-name list with exactly `names`, so "first stream" does not depend on other streams in the org. */ + async mockStreamList(names) { + await this.page.route(/\/api\/[^/]+\/streams\?.*type=profiles/, (route) => + route.fulfill({ status: 200, json: { list: names.map((name) => ({ name, stream_type: 'profiles' })) } }), + ); + } + + async unrouteApi(kind) { + await this.page.unroute(new RegExp(`/profiles/${kind}(\\?|$)`)); + } + + selectTrigger(name, scope) { + return (scope ?? this.page).locator(`[data-test="${name}-trigger"]`).first(); + } + + async openSelect(name, scope) { + const trigger = this.selectTrigger(name, scope); + await expect(trigger).toBeVisible(); + await expect(async () => { + if ((await trigger.getAttribute('aria-expanded')) !== 'true') await trigger.click(); + await expect(trigger).toHaveAttribute('aria-expanded', 'true', { timeout: 1500 }); + }).toPass({ timeout: 15000 }); + } + + async closeSelect(name, scope) { + const trigger = this.selectTrigger(name, scope); + if ((await trigger.getAttribute('aria-expanded')) === 'true') await this.page.keyboard.press('Escape'); + await expect(trigger).toHaveAttribute('aria-expanded', 'false'); + } + + options(name) { + return this.page.locator(`[data-test="${name}-popover"] [data-test="${name}-option"]`); + } + + async selectOptionLabels(name, scope) { + await this.openSelect(name, scope); + await expect(this.options(name).first()).toBeVisible(); + const labels = await this.options(name).evaluateAll((els) => els.map((el) => el.getAttribute('data-test-label'))); + await this.closeSelect(name, scope); + return labels; + } + + async pickOption(name, label, scope) { + await this.openSelect(name, scope); + const search = this.page.locator(`[data-test="${name}-popover"] [data-test="${name}-search"]`); + // Lists over 50 entries are virtualised, so an option off screen only renders once the search narrows to it. + if (await search.count()) await search.fill(label); + const option = this.page.locator(`[data-test="${name}-popover"] [data-test="${name}-option"][data-test-label="${label}"]`); + await expect(option).toBeVisible(); + await option.click(); + await expect(this.selectTrigger(name, scope)).toHaveAttribute('data-test-selected-label', label); + } + + async expectSelected(name, label, scope) { + await expect(this.selectTrigger(name, scope)).toHaveAttribute('data-test-selected-label', label, { timeout: 15000 }); + } + + async selectStream(stream) { + await this.pickOption(this.streamSelect, stream); + } + + async selectService(service) { + await this.pickOption(this.serviceSelect, service); + } + + async clickRun() { + await expect(this.page.locator(this.runBtn)).toBeEnabled(); + await this.page.locator(this.runBtn).click(); + await this.waitForRunIdle(); + } + + /** Clicks Run without waiting for the query to answer (for cancel tests). */ + async clickRunNoWait() { + await this.page.locator(this.runBtn).click(); + } + + async clickCancel() { + await expect(this.page.locator(this.cancelBtn)).toBeVisible({ timeout: 15000 }); + await this.page.locator(this.cancelBtn).click(); + await this.waitForRunIdle(); + } + + async expectRunPending(pending) { + const run = this.page.locator(this.runBtn); + if (pending) await expect(run).toHaveAttribute('data-run-pending', 'true'); + else await expect(run).not.toHaveAttribute('data-run-pending', 'true'); + } + + async expectRunDisabled() { + await expect(this.page.locator(this.runBtn)).toBeDisabled(); + } + + async hoverRunAndExpectTooltip(text) { + await this.page.locator(this.runBtn).hover({ force: true }); + await expect(this.page.locator('[data-test="o-tooltip-content"]').filter({ hasText: text })).toBeVisible(); + } + + async expectRunOptionsHidden() { + await expect(this.page.locator(this.runBtn)).toBeVisible(); + await expect(this.page.locator(this.runOptionsBtn)).toHaveCount(0); + } + + async openRunOptions() { + const trigger = this.page.locator(this.runOptionsBtn); + await expect(async () => { + if ((await trigger.getAttribute('aria-expanded')) !== 'true') await trigger.click(); + await expect(this.page.locator(this.autoRunToggle)).toBeVisible({ timeout: 1500 }); + }).toPass({ timeout: 15000 }); + } + + async expectAutoRunToggleText(text) { + await expect(this.page.locator(this.autoRunToggle)).toContainText(text); + } + + async clickAutoRunToggle() { + await this.page.locator(this.autoRunToggle).click(); + await expect(this.page.locator(this.autoRunToggle)).toHaveCount(0); + } + + async autoRunStorageValue() { + return this.page.evaluate(() => localStorage.getItem('oo_toggle_auto_run')); + } + + async clickDatePrev() { + await this.page.locator(this.datePrevBtn).click(); + } + + async clickDateNext() { + await this.page.locator(this.dateNextBtn).click(); + } + + async expectDateButtonText(pattern) { + await expect(this.page.locator(this.dateBtn)).toContainText(pattern); + } + + async openAddFilter() { + const add = this.page.locator(this.addFilterBtn); + await expect(async () => { + if ((await add.getAttribute('aria-expanded')) !== 'true') await add.click(); + await expect(this.selectTrigger(this.tagKeySelect)).toBeVisible({ timeout: 1500 }); + }).toPass({ timeout: 15000 }); + } + + async openFilterEditor(key) { + const edit = this.page.locator(`[data-test="profiles-filter-edit-${key}"]`); + await expect(async () => { + if ((await edit.getAttribute('aria-expanded')) !== 'true') await edit.click(); + await expect(this.selectTrigger(this.tagValueSelect)).toBeVisible({ timeout: 1500 }); + }).toPass({ timeout: 15000 }); + } + + /** Opens the add-filter menu, picks the label, then the value (which applies the filter). */ + async addFilter(key, value) { + await this.openAddFilter(); + await this.pickOption(this.tagKeySelect, key); + await this.pickFilterValue(value); + } + + async pickFilterValue(value) { + const trigger = this.selectTrigger(this.tagValueSelect); + await expect(trigger).toBeEnabled(); + await this.openSelect(this.tagValueSelect); + const option = this.page.locator( + `[data-test="${this.tagValueSelect}-popover"] [data-test="${this.tagValueSelect}-option"][data-test-label="${value}"]`, + ); + await expect(option).toBeVisible(); + await option.click(); + await expect(this.selectTrigger(this.tagValueSelect)).toHaveCount(0); + } + + async filterValueOptionLabels(key) { + await this.openAddFilter(); + await this.pickOption(this.tagKeySelect, key); + await expect(this.selectTrigger(this.tagValueSelect)).toBeEnabled(); + const labels = await this.selectOptionLabels(this.tagValueSelect); + await this.page.keyboard.press('Escape'); + return labels; + } + + async editFilter(key, value) { + await this.openFilterEditor(key); + await this.pickFilterValue(value); + } + + async removeFilter(key) { + await this.page.locator(`[data-test="profiles-filter-remove-${key}"]`).click(); + await expect(this.page.locator(`[data-test="profiles-filter-chip-${key}"]`)).toHaveCount(0); + } + + async expectFilterChip(key, value) { + const chip = this.page.locator(`[data-test="profiles-filter-chip-${key}"]`); + await expect(chip).toHaveCount(1); + await expect(chip).toContainText(key); + await expect(chip).toContainText(value); + } + + async expectFilterChipCount(count) { + await expect(this.page.locator(this.filterChips)).toHaveCount(count); + } + + async expectValueSelectDisabled() { + await this.openAddFilter(); + await expect(this.selectTrigger(this.tagValueSelect)).toBeDisabled(); + await this.page.keyboard.press('Escape'); + } + + async switchView(view) { + await this.page.locator(`[data-test="profiles-view-${view}"]`).click(); + await this.expectViewActive(view); + } + + async expectViewActive(view) { + await expect(this.page.locator(`[data-test="profiles-view-${view}"]`)).toHaveAttribute('data-state', 'on', { timeout: 15000 }); + } + + async search(text) { + await this.page.locator(this.searchInput).fill(text); + } + + async expectSearchValue(text) { + await expect(this.page.locator(this.searchInput)).toHaveValue(text); + } + + async expectSearchSummary(text) { + await expect(this.page.locator(this.searchSummary)).toHaveText(text); + } + + async toggleResultsExpand() { + await this.page.locator(this.resultsExpandBtn).click(); + } + + /** Rendered row at `position` (data-test row ids keep the unsorted data index, so position is by DOM order). */ + topRow(position) { + return this.topRowsLocator().nth(position); + } + + topRowsLocator() { + return this.page.locator(`${this.topTable} [data-test^="o2-table-row-"]`); + } + + /** Asserts a Top Table row: values are display strings, any omitted key is not checked. */ + async expectTopRow(index, { name, self, selfPercent, total, totalPercent }) { + const row = this.topRow(index); + if (name !== undefined) await expect(row.locator('[data-test="o2-table-cell-name"]')).toHaveText(name, { timeout: 15000 }); + if (self !== undefined) await expect(row.locator('[data-test="o2-table-cell-self"]')).toHaveText(self); + if (selfPercent !== undefined) await expect(row.locator('[data-test="o2-table-cell-selfPercent"]')).toHaveText(selfPercent); + if (total !== undefined) await expect(row.locator('[data-test="o2-table-cell-total"]')).toHaveText(total); + if (totalPercent !== undefined) await expect(row.locator('[data-test="o2-table-cell-totalPercent"]')).toHaveText(totalPercent); + } + + async expectTopRowNames(names) { + await expect(this.topRowsLocator().locator('[data-test="o2-table-cell-name"]')).toHaveText(names, { timeout: 15000 }); + } + + async expectTopRowCount(count) { + await expect(this.topRowsLocator()).toHaveCount(count, { timeout: 15000 }); + } + + async expectTopTableVisible() { + await expect(this.page.locator(this.topTable)).toBeVisible({ timeout: 15000 }); + } + + async sortTopBy(columnId) { + await this.page.locator(`th[data-test="o2-table-th-${columnId}"] [data-test="o2-table-th-sort-trigger"]`).click(); + } + + async expectTopSort(columnId, direction) { + await expect(this.page.locator(`th[data-test="o2-table-th-${columnId}"]`)).toHaveAttribute('aria-sort', direction); + } + + async expectMarkedText(text) { + await expect(this.topRowsLocator().locator('mark').first()).toHaveText(text); + } + + /** Opens the ⋮ menu of the row rendered at `position`; returns the menu's data-test prefix. */ + async openRowMenu(kind, position) { + const row = kind === 'tree' ? this.treeRow(position) : this.topRow(position); + const trigger = row.locator(`[data-test^="profiles-${kind}-row-"][data-test$="-menu"]`); + const prefix = (await trigger.getAttribute('data-test')).replace(/-menu$/, ''); + await expect(async () => { + if ((await trigger.getAttribute('aria-expanded')) !== 'true') await trigger.click(); + await expect(this.page.locator(`[data-test="${prefix}-copy"]`)).toBeVisible({ timeout: 1500 }); + }).toPass({ timeout: 15000 }); + return prefix; + } + + async rowMenuAction(kind, position, action) { + const prefix = await this.openRowMenu(kind, position); + await this.page.locator(`[data-test="${prefix}-${action}"]`).click(); + } + + treeRow(position) { + return this.page.locator(`${this.stackTable} [data-test^="o2-table-row-"]`).nth(position); + } + + async expectTreeRow(index, { name, counts, self, total }) { + const row = this.treeRow(index); + if (name !== undefined) await expect(row.locator('[data-test="o2-table-cell-name"]')).toContainText(name, { timeout: 15000 }); + const countCell = row.locator('[data-test^="profiles-tree-row-"][data-test$="-counts"]'); + if (counts === null) await expect(countCell).toHaveCount(0); + else if (counts !== undefined) await expect(countCell).toHaveText(counts); + if (self !== undefined) await expect(row.locator('[data-test="o2-table-cell-self"]')).toHaveText(self); + if (total !== undefined) await expect(row.locator('[data-test="o2-table-cell-total"]')).toHaveText(total); + } + + async expectTreeRowCount(count) { + await expect(this.page.locator(`${this.stackTable} [data-test^="o2-table-row-"]`)).toHaveCount(count); + } + + async toggleTreeRow(index) { + await this.treeRow(index).locator('[data-test="o2-table-tree-toggle-name"]').click(); + } + + async expectTreeRowExpanded(index, expanded) { + await expect(this.treeRow(index).locator('[data-test="o2-table-tree-toggle-name"]')).toHaveAttribute('aria-expanded', String(expanded)); + } + + async expectDetails({ name, self, total, calledBy, calls, hasPackage }) { + const panel = this.page.locator(this.details); + await expect(panel).toBeVisible({ timeout: 15000 }); + if (name !== undefined) await expect(panel).toContainText(name); + if (self !== undefined) await expect(panel.locator(this.detailsSelf)).toHaveText(self); + if (total !== undefined) await expect(panel.locator(this.detailsTotal)).toHaveText(total); + if (calledBy !== undefined) { + for (const caller of calledBy) await expect(panel.locator(this.detailsCalledBy)).toContainText(caller); + } + if (calls !== undefined) { + for (const callee of calls) await expect(panel.locator(this.detailsCalls)).toContainText(callee); + } + if (hasPackage !== undefined) await expect(panel.locator(this.detailsPackage)).toHaveCount(hasPackage ? 1 : 0); + } + + async detailsListRowCount(list) { + const section = this.page.locator(list === 'calls' ? this.detailsCalls : this.detailsCalledBy); + return section.locator(':scope > div').count(); + } + + async expectDetailsHidden() { + await expect(this.page.locator(this.details)).toHaveCount(0); + } + + async clickDetails(action) { + const map = { copy: this.detailsCopy, close: this.detailsClose, sandwich: this.detailsSandwich, show: this.detailsShow }; + await this.page.locator(map[action]).click(); + } + + async expectSandwichFunction(name) { + await this.expectSelected(this.sandwichFunction, name); + } + + async pickSandwichFunction(name) { + await this.pickOption(this.sandwichFunction, name); + } + + async expectSandwichCoverage(visible) { + await expect(this.page.locator(this.sandwichCoverage)).toHaveCount(visible ? 1 : 0, { timeout: 15000 }); + } + + async expectSandwichCoverageText(text) { + await expect(this.page.locator(this.sandwichCoverage)).toContainText(text, { timeout: 15000 }); + } + + flameGraph(scope) { + return (scope ? this.page.locator(scope) : this.page).locator(this.flamePlot).first(); + } + + /** Viewport point at the centre of a frame from its hand-computed geometry (rows 26 px, plot inset 10 px). */ + async framePoint(frame, { scope, rootAtBottom = false, maxDepth = 0, at = 0.5 } = {}) { + const box = await this.flameGraph(scope).boundingBox(); + if (!box) throw new Error('Flame graph plot is not on screen'); + const depth = rootAtBottom ? maxDepth - frame.depth : frame.depth; + const share = (frame.left + frame.width * at) / 100; + return { + x: box.x + FLAME_PLOT_INSET_PX + share * (box.width - 2 * FLAME_PLOT_INSET_PX), + y: box.y + 1 + depth * FLAME_ROW_PX + FLAME_ROW_PX / 2 - 1, + }; + } + + flameTooltip(text) { + return this.page.locator(`body > div[style*="${FLAME_TOOLTIP_Z}"]`).filter({ hasText: text }); + } + + /** Hovers a frame and proves the tooltip names it, so later clicks land on the intended frame. */ + async hoverFrame(frame, name, options = {}) { + const point = await this.framePoint(frame, options); + await expect(async () => { + await this.page.mouse.move(point.x, point.y - 3); + await this.page.mouse.move(point.x, point.y); + await expect(this.flameTooltip(name).first()).toBeVisible({ timeout: 1500 }); + }).toPass({ timeout: 15000 }); + return point; + } + + async expectFlameTooltipContains(name, texts) { + const tooltip = this.flameTooltip(name).first(); + for (const text of texts) await expect(tooltip).toContainText(text); + } + + async expectFlameTooltipLacks(name, text) { + await expect(this.flameTooltip(name).first()).not.toContainText(text); + } + + async expectFlameViewOptions() { + await expect(this.page.locator(this.flameViewOptions).first()).toBeVisible(); + } + + async clickFrame(frame, name, options = {}) { + const point = await this.hoverFrame(frame, name, options); + await this.page.mouse.click(point.x, point.y); + } + + async rightClickFrame(frame, name, options = {}) { + const point = await this.hoverFrame(frame, name, options); + await this.page.mouse.click(point.x, point.y, { button: 'right' }); + } + + async expectFlameMenu(items, absent = []) { + const menu = this.page.locator(this.flameMenu); + await expect(menu).toBeVisible(); + for (const item of items) await expect(menu.locator(`[data-test="flame-graph-${item}"]`)).toBeVisible(); + for (const item of absent) await expect(menu.locator(`[data-test="flame-graph-${item}"]`)).toHaveCount(0); + } + + /** Hovering a wide frame shows its ⋮ button only while no frame menu is open, so this proves the menu is closed. */ + async expectNoFrameMenuVia(frame, name, options = {}) { + await this.hoverFrame(frame, name, options); + await expect(this.page.locator(this.flameFrameMore)).toBeVisible(); + await expect(this.page.locator(this.flameMenu)).toHaveCount(0); + } + + /** Opens a frame's menu, which offers Reset zoom only while a frame is focused, then closes it again. */ + async expectNotZoomedVia(frame, name, options = {}) { + await this.rightClickFrame(frame, name, options); + await this.expectFlameMenu(['copy-name'], ['zoom-reset', 'zoom-out', 'zoom-out-here']); + await this.pressEscape(); + await expect(this.page.locator(this.flameMenu)).toHaveCount(0); + } + + async expectFlameMenuHidden() { + await expect(this.page.locator(this.flameMenu)).toHaveCount(0); + } + + async clickFlameMenuItem(item) { + await this.page.locator(`${this.flameMenu} [data-test="flame-graph-${item}"]`).click(); + } + + async expectFlameRootSummary(text, scope) { + await this.hoverFrame(CHECKOUT_FRAMES.root, text, { scope }); + } + + /** Screenshot of the label end of a frame (first 40 px), where search highlighting draws. */ + async frameLabelPng(frame, options = {}) { + const box = await this.flameGraph(options.scope).boundingBox(); + const x = box.x + FLAME_PLOT_INSET_PX + (frame.left / 100) * (box.width - 2 * FLAME_PLOT_INSET_PX); + const y = box.y + 1 + frame.depth * FLAME_ROW_PX; + return this.page.screenshot({ clip: { x: Math.round(x), y: Math.round(y), width: 40, height: FLAME_ROW_PX - 2 } }); + } + + async flameCanvasPng(scope) { + return this.flameGraph(scope).screenshot(); + } + + async toggleRootAtBottom() { + await this.page.locator(this.flameRootAtBottom).first().click(); + } + + async expectRootAtBottom(checked) { + await expect(this.page.locator(this.flameRootAtBottom).first()).toHaveAttribute('data-state', checked ? 'checked' : 'unchecked'); + } + + async clickAlign(side) { + await this.page.locator(side === 'right' ? this.flameAlignRight : this.flameAlignLeft).first().click(); + } + + async expectAlignActive(side) { + const active = this.page.locator(side === 'right' ? this.flameAlignRight : this.flameAlignLeft).first(); + await expect(active).toHaveAttribute('data-o2-variant', 'primary'); + } + + async flamePlotWidth(scope) { + const box = await this.flameGraph(scope).boundingBox(); + return box ? box.width : 0; + } + + async flameCanvasWidth(scope) { + return this.flameGraph(scope).locator('canvas').first().evaluate((canvas) => canvas.getBoundingClientRect().width); + } + + async ctrlWheelOnFrame(frame, name, deltaY) { + const point = await this.hoverFrame(frame, name); + await this.page.keyboard.down('Control'); + await this.page.mouse.wheel(0, deltaY); + await this.page.keyboard.up('Control'); + return point; + } + + async dragOnFlame(from, dx) { + await this.page.mouse.move(from.x, from.y); + await this.page.mouse.down(); + await this.page.mouse.move(from.x + dx / 2, from.y, { steps: 5 }); + await this.page.mouse.move(from.x + dx, from.y, { steps: 5 }); + await this.page.mouse.up(); + } + + async expectFlameGrabCursor(zoomed) { + const plot = this.flameGraph(); + if (zoomed) await expect(plot).toHaveClass(/cursor-grab/); + else await expect(plot).not.toHaveClass(/cursor-grab/); + } + + async expectZoomHint(visible) { + await expect(this.page.locator(this.flameZoomHint)).toHaveCount(visible ? 1 : 0, { timeout: 15000 }); + } + + async expectZoomHintText(pattern) { + await expect(this.page.locator(this.flameZoomHint)).toContainText(pattern); + } + + async clickZoomHint(action) { + await this.page.locator(action === 'never' ? this.flameZoomHintNever : this.flameZoomHintClose).click(); + } + + async zoomHintStorageValue() { + return this.page.evaluate(() => localStorage.getItem('o2_flame_zoom_hint_hidden')); + } + + async expectFrameMoreVisible() { + await expect(this.page.locator(this.flameFrameMore)).toBeVisible(); + } + + async clickFrameMore() { + await this.page.locator(this.flameFrameMore).click(); + } + + async pressEscape() { + await this.page.keyboard.press('Escape'); + } + + pixelAt(png, fx, fy) { + const image = PNG.sync.read(png); + const x = Math.min(image.width - 1, Math.max(0, Math.round(fx * image.width))); + const y = Math.min(image.height - 1, Math.max(0, Math.round(fy * image.height))); + const i = (image.width * y + x) * 4; + return { r: image.data[i], g: image.data[i + 1], b: image.data[i + 2] }; + } + + /** RGB at a viewport point, from a 1×1 page screenshot. */ + async viewportPixel(x, y) { + const png = await this.page.screenshot({ clip: { x: Math.round(x), y: Math.round(y), width: 1, height: 1 } }); + return this.pixelAt(png, 0, 0); + } + + /** Horizontal centre (0–1 of the chart width) of the strongly blue pixels the series draws. */ + async seriesBlueCentre() { + const png = await this.page.locator(this.seriesChart).screenshot(); + const image = PNG.sync.read(png); + let sum = 0; + let count = 0; + for (let y = 0; y < image.height; y++) { + for (let x = 0; x < image.width; x++) { + const i = (image.width * y + x) * 4; + const [r, g, b] = [image.data[i], image.data[i + 1], image.data[i + 2]]; + if (b > 150 && b - r > 60 && b - g > 30) { + sum += x; + count += 1; + } + } + } + return { count, centre: count ? sum / count / image.width : -1 }; + } + + async expectSeriesCardVisible(visible) { + if (visible) await expect(this.page.locator(this.seriesCard)).toBeVisible(); + else await expect(this.page.locator(this.seriesCard)).toBeHidden(); + } + + /** Sweeps the pointer across the series chart and returns every distinct tooltip text. */ + async collectSeriesTooltips(step = 5) { + const chart = this.page.locator(this.seriesChart); + await expect(chart.locator('canvas').first()).toBeVisible(); + const box = await chart.boundingBox(); + const texts = new Set(); + const tooltip = chart.locator(':scope > div').filter({ hasText: /\d/ }); + for (let x = box.x + 2; x < box.x + box.width - 2; x += step) { + await this.page.mouse.move(x, box.y + box.height / 2); + const visible = tooltip.last(); + if (await visible.isVisible()) { + const text = (await visible.innerText()).replace(/\s+/g, ' ').trim(); + if (text) texts.add(text); + } + } + await this.page.mouse.move(box.x - 20, box.y - 20); + return [...texts]; + } + + async dragAcrossChart(fromShare, toShare) { + const box = await this.page.locator(this.seriesChart).boundingBox(); + const y = box.y + box.height / 2; + await this.page.mouse.move(box.x + box.width * fromShare, y); + await this.page.mouse.down(); + await this.page.mouse.move(box.x + box.width * ((fromShare + toShare) / 2), y, { steps: 5 }); + await this.page.mouse.move(box.x + box.width * toShare, y, { steps: 5 }); + await this.page.mouse.up(); + } + + async pickBreakdown(label) { + await this.pickOption(this.breakdownSelect, label); + } + + async breakdownOptionLabels() { + return this.selectOptionLabels(this.breakdownSelect); + } + + async expectBreakdownDisabled() { + await expect(this.selectTrigger(this.breakdownSelect)).toBeDisabled(); + } + + breakdownChip(key) { + return this.page.locator(`[data-test="profiles-breakdown-chip-${key}"]`); + } + + async expectBreakdownChips(labels) { + await expect(this.page.locator(`${this.breakdownLegend} [data-test^="profiles-breakdown-chip-"]`)).toHaveText(labels, { timeout: 15000 }); + } + + async clickBreakdownChip(key) { + await this.breakdownChip(key).click(); + } + + async expectOthersChipDisabled() { + await expect(this.page.locator(this.breakdownChipOthers)).toBeDisabled(); + await expect(this.page.locator(this.breakdownChipOthers)).toHaveText('Others'); + } + + async expectState(name, { title, description } = {}) { + const map = { + error: this.errorState, + noStream: this.noStreamState, + cancelled: this.searchCancelledState, + noResults: this.noResultsState, + errorBanner: this.errorBanner, + partial: this.partialResultBanner, + noSeries: this.noSeries, + seriesSkeleton: this.seriesSkeleton, + sandwichSkeleton: this.sandwichSkeleton, + }; + const state = this.page.locator(map[name]); + await expect(state).toBeVisible({ timeout: 15000 }); + if (title) await expect(state).toContainText(title); + if (description) await expect(state).toContainText(description); + } + + async expectStateHidden(name) { + const map = { error: this.errorState, noResults: this.noResultsState, errorBanner: this.errorBanner, partial: this.partialResultBanner, cancelled: this.searchCancelledState }; + await expect(this.page.locator(map[name])).toHaveCount(0, { timeout: 15000 }); + } + + /** OEmptyState renders its action buttons without a data-test hook, so the button is found by its label inside the state. */ + async clickStateAction(name) { + const map = { + error: [this.errorState, 'Retry'], + noStream: [this.noStreamState, 'Set up profiles ingestion'], + cancelled: [this.searchCancelledState, 'Run query'], + }; + const [state, label] = map[name]; + await this.page.locator(state).getByRole('button', { name: label }).click(); + } + + async expectTopTableLoading() { + await expect(this.page.locator(`${this.topTable} [data-test="o2-table"]`)).toHaveAttribute('data-test-loading', 'true'); + } + + async expectSkeleton(kind) { + const map = { series: this.seriesSkeleton, flame: this.flameSkeleton, sandwich: this.sandwichSkeleton }; + await expect(this.page.locator(map[kind])).toBeVisible({ timeout: 15000 }); + } + + /** Switches the results view while a query is still loading (the toggle stays usable). */ + async switchViewWhileLoading(view) { + await this.page.locator(`[data-test="profiles-view-${view}"]`).click(); + await expect(this.page.locator(`[data-test="profiles-view-${view}"]`)).toHaveAttribute('data-state', 'on'); + } + + async expectErrorBannerText(text) { + await expect(this.page.locator(this.errorBanner)).toContainText(text, { timeout: 15000 }); + } + + async clickErrorBannerRetry() { + await this.page.locator(this.errorRetryBtn).click(); + } + + async expectShareDisabled(disabled) { + const share = this.page.locator(this.shareBtn); + if (disabled) await expect(share).toBeDisabled(); + else await expect(share).toBeEnabled(); + } + + async clickShareAndReadClipboard() { + await this.page.locator(this.shareBtn).click(); + await expect(this.page.locator(this.successToast).filter({ hasText: 'Link Copied Successfully!' })).toBeVisible({ timeout: 15000 }); + return this.readClipboard(); + } + + async readClipboard() { + return this.page.evaluate(() => navigator.clipboard.readText()); + } + + async expectCopiedToast() { + await expect(this.page.locator(this.successToast).filter({ hasText: 'Copied to clipboard' }).first()).toBeVisible({ timeout: 15000 }); + } + + /** RGB of the frame's fill at `at` across its width (away from the label, which sits at the left). */ + async framePixel(frame, { at = 0.8, scope } = {}) { + const point = await this.framePoint(frame, { at, scope }); + await this.page.mouse.move(0, 0); + return this.viewportPixel(point.x, point.y); + } + + /** Chart-canvas pixel near its top-right corner (no axis or line there) and the card pixel just above the canvas. */ + async chartAndCardBackground() { + const box = await this.page.locator(this.seriesChart).boundingBox(); + const chart = await this.viewportPixel(box.x + box.width - 6, box.y + 3); + const card = await this.viewportPixel(box.x + box.width - 6, box.y - 6); + return { chart, card }; + } + + async setViewport(width, height) { + await this.page.setViewportSize({ width, height }); + } + + async openMobileViewMenu() { + const trigger = this.page.locator('[data-test="profiles-view-dropdown-btn"]'); + await expect(async () => { + if ((await trigger.getAttribute('aria-expanded')) !== 'true') await trigger.click(); + await expect(this.page.locator('[data-test="profiles-view-top-item"]')).toBeVisible({ timeout: 1500 }); + }).toPass({ timeout: 15000 }); + } + + async mobileViewMenuItems() { + return this.page.locator('[data-test^="profiles-view-"][data-test$="-item"]').evaluateAll((els) => els.map((el) => el.getAttribute('data-test'))); + } + + async pickMobileView(view) { + await this.openMobileViewMenu(); + await this.page.locator(`[data-test="profiles-view-${view}-item"]`).click(); + } + + async hasHorizontalPageScroll() { + return this.page.evaluate(() => document.documentElement.scrollWidth > document.documentElement.clientWidth); + } + + async detailsSitsBelowTable() { + const table = await this.page.locator(this.topTable).boundingBox(); + const details = await this.page.locator(this.details).boundingBox(); + return details.y >= table.y + table.height - 1; + } + + async expectCompareSides(label, baseline, comparison) { + await this.expectSelected(this.compareLabel, label); + await this.expectSelected(this.compareBaseline, baseline); + await this.expectSelected(this.compareComparison, comparison); + } + + async waitForCompareIdle() { + await expect(this.page.locator(this.compare)).toBeVisible({ timeout: 15000 }); + await expect(this.page.locator(this.compareSkeleton)).toHaveCount(0, { timeout: 30000 }); + } + + async pickCompareDisplay(display) { + await this.page.locator(`[data-test="profiles-compare-display-${display}"]`).click(); + await expect(this.page.locator(`[data-test="profiles-compare-display-${display}"]`)).toHaveAttribute('data-state', 'on'); + await this.waitForCompareIdle(); + } + + async clickCompareSwap() { + await this.page.locator(this.compareSwap).click(); + await this.waitForCompareIdle(); + } + + async pickCompareLabel(label) { + await this.pickOption(this.compareLabel, label); + await this.waitForCompareIdle(); + } + + async compareBaselineOptions() { + return this.selectOptionLabels(this.compareBaseline); + } + + compareTableRow(name) { + return this.page.locator(`${this.compareTable} [data-test^="o2-table-row-"]`).filter({ + has: this.page.locator('[data-test="o2-table-cell-name"]', { hasText: new RegExp(`^${name.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}$`) }), + }); + } + + async expectCompareTableRow(name, { before, after, beforePercent, afterPercent, change }) { + const row = this.compareTableRow(name); + await expect(row).toHaveCount(1, { timeout: 15000 }); + if (before !== undefined) await expect(row.locator('[data-test="o2-table-cell-before"]')).toHaveText(before); + if (after !== undefined) await expect(row.locator('[data-test="o2-table-cell-after"]')).toHaveText(after); + if (beforePercent !== undefined) await expect(row.locator('[data-test="o2-table-cell-beforePercent"]')).toHaveText(beforePercent); + if (afterPercent !== undefined) await expect(row.locator('[data-test="o2-table-cell-afterPercent"]')).toHaveText(afterPercent); + if (change !== undefined) await expect(row.locator('[data-test="o2-table-cell-change"]')).toHaveText(change); + } + + async expectCompareTableRowNames(names) { + await expect( + this.page.locator(`${this.compareTable} [data-test^="o2-table-row-"] [data-test="o2-table-cell-name"]`), + ).toHaveText(names, { timeout: 15000 }); + } + + async expectCompareMessage(text) { + await expect(this.page.locator(this.compareMessage)).toContainText(text, { timeout: 15000 }); + } + + async expectMenuProfilesVisible(visible) { + await expect(this.page.locator(this.navRail)).toBeVisible(); + await expect(this.page.locator(this.menuTraces)).toBeVisible(); + await expect(this.page.locator(this.menuProfiles)).toHaveCount(visible ? 1 : 0); + } + + async clickMenuProfiles() { + await this.page.locator(this.menuProfiles).click(); + await this.waitForPageReady(); + } + + async expectPageHeader() { + await expect(this.page.locator(this.pageRoot)).toBeVisible(); + await expect(this.page.locator(this.betaBadge).first()).toBeVisible(); + await expect(this.page.locator(this.pageTitle)).toContainText('Profiles'); + } + + async gotoDataPageProfiles(stream) { + await this.page.goto(`${this.baseUrl()}/web/streams?org_identifier=${this.orgName()}`); + const filter = this.page.locator(this.streamTypeFilter); + await expect(filter).toBeVisible({ timeout: 30000 }); + // The stream-type toggle items carry no data-test of their own (OToggleGroupItem only tags its mobile menu rows). + const profilesToggle = filter.getByRole('button', { name: 'Profiles' }); + await profilesToggle.click(); + await expect(profilesToggle).toHaveAttribute('data-state', 'on'); + await expect(this.page.locator('[data-test^="log-stream-name-cell-"]').first()).toBeVisible({ timeout: 30000 }); + await expect(async () => { + await this.page.locator(this.streamSearchInput).fill(stream); + await expect(this.page.locator('[data-test^="log-stream-name-cell-"]')).toHaveCount(1, { timeout: 5000 }); + }).toPass({ timeout: 30000 }); + await expect(this.page.locator(`[data-test="log-stream-name-cell-${stream}"]`)).toBeVisible(); + } + + /** Clicks the row's explore button until the app leaves the Data page (the filtered list re-renders under the pointer). */ + async clickDataPageExplore(stream) { + const row = this.page + .locator('[data-test^="o2-table-row-"]') + .filter({ has: this.page.locator(`[data-test="log-stream-name-cell-${stream}"]`) }); + await expect(async () => { + if (new URL(this.page.url()).pathname === '/web/streams') await row.locator(this.streamExploreBtn).click({ timeout: 2000 }); + await expect.poll(() => new URL(this.page.url()).pathname, { timeout: 3000 }).toBe('/web/logs'); + }).toPass({ timeout: 20000 }); + } + + /** Data-test ids of the visible left-rail items, top to bottom. */ + async visibleMenuOrder() { + await expect(this.page.locator(this.menuProfiles)).toBeVisible(); + return this.page.locator(this.menuLinks).evaluateAll((els) => + els + .map((el) => ({ id: el.getAttribute('data-test'), box: el.getBoundingClientRect() })) + .filter(({ box }) => box.width > 0 && box.height > 0) + .sort((a, b) => a.box.top - b.box.top) + .map(({ id }) => id), + ); + } + + async gotoProfilesIngestion() { + await this.page.goto(`${this.baseUrl()}/web/ingestion/custom/profiles/otelcollector?org_identifier=${this.orgName()}`); + await expect(this.page.locator(this.ingestionOtelBlock)).toBeVisible({ timeout: 30000 }); + } + + async ingestionSnippetText(which) { + const map = { otel: this.ingestionOtelBlock, start: this.ingestionOtelStartBlock, grpc: this.ingestionOtelGrpcBlock }; + await expect(this.page.locator(map[which])).toBeVisible(); + return this.page.locator(map[which]).innerText(); + } + + async expectIngestionProfilesTabsActive() { + await expect(this.page.locator(this.ingestionProfilesTab)).toBeVisible(); + await expect(this.page.locator(this.ingestionOtelTab)).toBeVisible(); + } + + async expectLogsExplorerRows() { + await expect(this.page.locator('[data-test="logs-search-result-logs-table"] tbody tr[data-test^="o2-table-row-"]').first()).toBeVisible({ timeout: 30000 }); + } + + /** Resolves once the trace sidebar's profile probe has queried `stream` for `spanId`. */ + waitForSpanProfileProbe(stream, spanId) { + return this.page.waitForResponse( + (response) => { + const body = response.request().postData() || ''; + return response.url().includes('/_search') && body.includes(spanId) && body.includes(stream); + }, + { timeout: 30000 }, + ); + } + + async expectTraceSidebarSpan(spanId) { + await expect(this.page.locator('[data-test="trace-details-sidebar-header-operation-name"]').first()).toHaveText(`op_${spanId}`); + } + + async clickTraceSpanAndAwaitProbe(stream, spanId) { + const probe = this.waitForSpanProfileProbe(stream, spanId); + await this.openTraceSpan(spanId); + await (await probe).finished(); + } + + async gotoTraceDetails({ traceId, stream, fromUs, toUs }) { + await this.page.goto( + `${this.baseUrl()}/web/traces/trace-details?trace_id=${traceId}&stream=${stream}&from=${fromUs}&to=${toUs}&org_identifier=${this.orgName()}`, + ); + } + + async openTraceSpan(spanId) { + const span = this.page.locator(`[data-test="trace-tree-span-operation-name-${spanId}"]`); + await expect(span).toBeVisible({ timeout: 30000 }); + await span.click(); + } + + async expectViewProfileButton(visible) { + if (visible) await expect(this.page.locator(this.traceViewProfileBtn)).toBeVisible({ timeout: 30000 }); + else await expect(this.page.locator(this.traceViewProfileBtn)).toHaveCount(0); + } + + async clickViewProfile() { + await this.page.locator(this.traceViewProfileBtn).click(); + await this.waitForPageReady(); + } + + async isXssExecuted() { + return this.page.evaluate(() => window.__xss !== undefined || !!document.querySelector('img[src="x"]')); + } +} diff --git a/tests/ui-testing/playwright-tests/Alerts/alerts-advanced.spec.js b/tests/ui-testing/playwright-tests/Alerts/alerts-advanced.spec.js index a97ed578b5..dcaa5edb36 100644 --- a/tests/ui-testing/playwright-tests/Alerts/alerts-advanced.spec.js +++ b/tests/ui-testing/playwright-tests/Alerts/alerts-advanced.spec.js @@ -4,7 +4,6 @@ * Tests for: * 1. Advanced Conditions - Multiple conditions with AND/OR logic * 2. Bulk Operations - Bulk pause/unpause alerts - * 3. Deduplication Configuration - Scheduled alerts with dedup settings * * Note: Each test is independent and creates its own test data. * Cleanup is handled by cleanup.spec.js via 'auto_' prefix patterns. @@ -14,7 +13,7 @@ const { test, expect, navigateToBase } = require('../utils/enhanced-baseFixtures const testLogger = require('../utils/test-logger.js'); const PageManager = require('../../pages/page-manager.js'); const logData = require('../../fixtures/log.json'); -const { getOrgIdentifier, isCloudEnvironment } = require('../utils/cloud-auth.js'); +const { getOrgIdentifier } = require('../utils/cloud-auth.js'); const TEST_STREAM = 'e2e_automate'; @@ -161,208 +160,4 @@ test.describe("Alerts Advanced Coverage Tests", () => { testLogger.info('Bulk operations test completed successfully'); }); - - // ==================== DEDUPLICATION TESTS ==================== - - test("Create scheduled alert with deduplication configuration", { - tag: ['@alertsAdvanced', '@all', '@alerts', '@deduplication'] - }, async ({ page }) => { - const uniqueSuffix = Math.random().toString(36).substring(2, 8); - testLogger.info('Testing scheduled alert with deduplication configuration', { uniqueSuffix }); - - // This test focuses on UI validation of scheduled alert with deduplication settings - // Scheduled alert trigger verification is covered by alerts-ui-operations.spec.js - const templateName = 'auto_template_dedup_' + uniqueSuffix; - const destinationName = 'auto_dest_dedup_' + uniqueSuffix; - - await pm.alertTemplatesPage.navigateToTemplates(); - await pm.alertTemplatesPage.waitForTemplateListReady(); - await pm.alertTemplatesPage.createTemplate(templateName); - testLogger.info('Template created', { templateName }); - - await pm.alertDestinationsPage.navigateToDestinations(); - await pm.alertDestinationsPage.waitForDestinationListReady(); - const destinationUrl = notificationSinkUrl(); - await pm.alertDestinationsPage.createDestination(destinationName, destinationUrl, templateName); - testLogger.info('Destination created', { destinationName }); - - const alertsUrl = `${logData.alertUrl}?org_identifier=${getOrgIdentifier()}`; - await page.goto(alertsUrl); - await pm.alertsPage.waitForAddAlertButton(); - - const dedupConfig = { - timeWindowMinutes: 30 - // fingerprintFields left empty to test auto-detection - }; - - const alertName = await pm.alertsPage.createScheduledAlertWithDeduplication( - TEST_STREAM, - destinationName, - uniqueSuffix, - dedupConfig - ); - - await pm.alertsPage.verifyAlertCreated(alertName); - testLogger.info('Scheduled alert with deduplication created and verified successfully', { - alertName, - dedupConfig - }); - }); - - /** - * Deduplication Validation Test with Trigger Verification - * - * NOTE: This test is currently skipped because: - * 1. Scheduled alert trigger validation has timing issues (known limitation) - * 2. Deduplication is an enterprise-only feature (requires enterprise build) - * 3. UI configuration testing is covered by "Create scheduled alert with deduplication configuration" - * - * To enable this test: - * - Ensure enterprise features are enabled in the build - * - Consider using longer timeouts or API-based validation - * - May need to adjust scheduled alert query time ranges - * - * Test design: - * 1. Creates unique source stream with custom columns - * 2. Creates scheduled alert with deduplication configured - * 3. Ingests data to trigger first alert - verifies notification appears - * 4. Ingests more data (same fingerprint, within time window) - * 5. Verifies NO NEW notification - dedup suppressed the duplicate - */ - test("Deduplication validation - verify duplicate alerts are suppressed", { - tag: ['@alertsAdvanced', '@alerts', '@deduplication', '@dedupValidation', '@enterprise', '@skip'] - }, async ({ page }) => { - // Round-trip verification needs a self-referencing destination (alert notification - // ingested back into a validation stream). On cloud the SSRF guard blocks the - // cluster's own (private-IP) URL at destination-create time, so this cannot run. - test.skip(isCloudEnvironment(), 'Self-referencing validation destination is blocked by the SSRF guard on cloud'); - - test.slow(); // Mark as slow test due to scheduled alert evaluation cycles - - // Generate unique suffix (lowercase for API compatibility) - const uniqueSuffix = Math.random().toString(36).substring(2, 8).toLowerCase(); - testLogger.info('Starting deduplication validation test', { uniqueSuffix }); - - // Create unique source stream with custom columns (same approach as e2e-flow) - const sourceStreamName = `dedup_src_${uniqueSuffix}`.toLowerCase(); - const column = 'city'; // Custom column from our test stream - const value = 'bangalore'; // Value that triggers the alert condition - - // Initialize the source stream with custom columns - await pm.commonActions.initializeAlertTestStream(sourceStreamName); - testLogger.info('Initialized source stream with custom columns', { - sourceStreamName, - columns: ['city', 'country', 'status', 'age', 'test_run_id', 'test_timestamp', 'message'] - }); - - // Create validation infrastructure (template + destination pointing to validation stream) - const validationInfra = await pm.alertsPage.ensureValidationInfrastructure(pm, uniqueSuffix); - testLogger.info('Validation infrastructure ready', validationInfra); - - // Navigate to alerts page - refresh to ensure new stream appears - await pm.commonActions.navigateToAlerts(); - await page.reload(); - await page.waitForLoadState('networkidle', { timeout: 10000 }).catch(() => {}); - await pm.alertsPage.waitForAddAlertButton(); - - // Create scheduled alert with deduplication for validation - const dedupConfig = { - timeWindowMinutes: 5, // 5-minute dedup window - fingerprintFields: [column] // Fingerprint based on city column - }; - - const alertName = await pm.alertsPage.createScheduledAlertWithDedupForValidation( - sourceStreamName, - column, - value, - validationInfra.destinationName, - uniqueSuffix, - dedupConfig - ); - await pm.alertsPage.verifyAlertCreated(alertName); - testLogger.info('Created scheduled alert with deduplication', { - alertName, - dedupConfig, - stream: sourceStreamName - }); - - // Wait for alert to register (handled implicitly by the validation-stream poll below) - testLogger.info('Waiting for alert registration...'); - - // ===== TRIGGER #1: Ingest data to trigger first alert ===== - testLogger.info('Ingesting first batch of data to trigger alert...'); - await pm.commonActions.ingestTestDataWithUniqueId(sourceStreamName, `trigger1_${uniqueSuffix}`, column, value); - testLogger.info('First data ingestion complete'); - - // Wait for scheduled alert evaluation cycle (1 min period + buffer) - testLogger.info('Waiting for first scheduled alert evaluation cycle...'); - - // Verify first notification appears in validation stream - const firstCheck = await pm.commonActions.waitForAlertInValidationStream( - validationInfra.streamName, - alertName, - 150, // max 150 seconds (2.5 min for scheduled alert to fire) - 10 // poll every 10 seconds - ); - - expect(firstCheck.found, `First alert notification should appear in validation stream for ${alertName}`).toBe(true); - testLogger.info('First notification verified in validation stream', { - alertName, - found: firstCheck.found, - attempts: firstCheck.attempts - }); - - // Get initial notification count - const initialCount = await pm.commonActions.countAlertNotificationsInStream( - validationInfra.streamName, - alertName, - 15 - ); - testLogger.info('Initial notification count', { - count: initialCount.count, - alertName - }); - - // ===== TRIGGER #2: Additional data ingestion (same fingerprint, within window) ===== - testLogger.info('Ingesting second batch of data (should be suppressed by dedup)...'); - const trigger2Start = Date.now(); - await pm.commonActions.ingestTestDataWithUniqueId(sourceStreamName, `trigger2_${uniqueSuffix}`, column, value); - testLogger.info('Second data ingestion complete'); - - // Wait for another evaluation cycle: poll until the scheduled-alert window has elapsed - // since trigger2 ingestion. This is the deterministic equivalent of the previous time wait. - testLogger.info('Waiting for second scheduled alert evaluation cycle...'); - await page.waitForFunction( - (startedAt) => Date.now() - startedAt >= 100000, - trigger2Start, - { timeout: 120000, polling: 5000 } - ); - - // ===== VALIDATION: Check that dedup suppressed the duplicate ===== - const finalCount = await pm.commonActions.countAlertNotificationsInStream( - validationInfra.streamName, - alertName, - 15 - ); - testLogger.info('Final notification count after second trigger', { - initialCount: initialCount.count, - finalCount: finalCount.count, - alertName - }); - - // The key assertion: count should NOT have increased - // If dedup is working, we should have same count as before - expect( - finalCount.count, - `Dedup should suppress duplicate: expected ${initialCount.count} notifications, got ${finalCount.count}` - ).toBe(initialCount.count); - - testLogger.info('DEDUPLICATION VALIDATION PASSED', { - alertName, - dedupConfig, - firstTriggerCount: initialCount.count, - secondTriggerCount: finalCount.count, - duplicateSuppressed: true - }); - }); }); diff --git a/tests/ui-testing/playwright-tests/Alerts/alerts-vrl-encoding.spec.js b/tests/ui-testing/playwright-tests/Alerts/alerts-vrl-encoding.spec.js index 90749a58f7..b47e26011c 100644 --- a/tests/ui-testing/playwright-tests/Alerts/alerts-vrl-encoding.spec.js +++ b/tests/ui-testing/playwright-tests/Alerts/alerts-vrl-encoding.spec.js @@ -423,7 +423,7 @@ test.describe("VRL Encoding Tests @vrl @alerts", () => { await page.waitForLoadState('domcontentloaded', { timeout: 15000 }).catch(() => {}); // Navigate through wizard steps and submit - await pm.alertsPage.navigateThroughWizardSteps(5); + await pm.alertsPage.navigateThroughWizardSteps(4); const submitted = await pm.alertsPage.clickSubmitButton(); expect(submitted).toBe(true); diff --git a/tests/ui-testing/playwright-tests/Correlation/correlation-settings.spec.js b/tests/ui-testing/playwright-tests/Correlation/correlation-settings.spec.js index 18fb702e44..fa6367b753 100644 --- a/tests/ui-testing/playwright-tests/Correlation/correlation-settings.spec.js +++ b/tests/ui-testing/playwright-tests/Correlation/correlation-settings.spec.js @@ -133,7 +133,7 @@ test.describe( pm = new PageManager(page); }); - test("loads the Correlation Settings page with all four tabs visible", { + test("loads the Correlation Settings page with all three tabs visible", { tag: ["@smoke", "@P0"], }, async () => { const orgId = process.env["ORGNAME"]; @@ -142,7 +142,7 @@ test.describe( await pm.correlationSettingsPage.expectAllTabsVisible(); }); - test("switches between all four tabs successfully", { + test("switches between all three tabs successfully", { tag: ["@smoke", "@P0"], }, async () => { const orgId = process.env["ORGNAME"]; @@ -152,8 +152,8 @@ test.describe( await pm.correlationSettingsPage.expectServicesContentVisible(); await pm.correlationSettingsPage.clickServiceDiscoveryTab(); await pm.correlationSettingsPage.expectServiceDiscoveryContentVisible(); - await pm.correlationSettingsPage.clickAlertCorrelationTab(); - await pm.correlationSettingsPage.expectAlertCorrelationContentVisible(); + await pm.correlationSettingsPage.clickFieldAliasesTab(); + await pm.correlationSettingsPage.expectFieldAliasesContentVisible(); await pm.correlationSettingsPage.clickServicesTab(); await pm.correlationSettingsPage.expectServicesContentVisible(); }); diff --git a/tests/ui-testing/playwright-tests/Dashboards/dashboard-create-alert.spec.js b/tests/ui-testing/playwright-tests/Dashboards/dashboard-create-alert.spec.js index 7937d9c43e..d6c75501fb 100644 --- a/tests/ui-testing/playwright-tests/Dashboards/dashboard-create-alert.spec.js +++ b/tests/ui-testing/playwright-tests/Dashboards/dashboard-create-alert.spec.js @@ -629,10 +629,7 @@ test.describe("Dashboard Create Alert testcases", () => { await destOption.click(); await page.keyboard.press("Escape"); - // Step 5 (Dedup) → Skip, Continue - await continueBtn.click(); - - // Step 6 (Advanced) → Skip, Continue + // Step 5 (Advanced) → Skip, Continue await continueBtn.click(); // Submit the alert diff --git a/tests/ui-testing/playwright-tests/Dashboards/dashboard-html-chart.spec.js b/tests/ui-testing/playwright-tests/Dashboards/dashboard-html-chart.spec.js index 2c3cae9178..cabc11955d 100644 --- a/tests/ui-testing/playwright-tests/Dashboards/dashboard-html-chart.spec.js +++ b/tests/ui-testing/playwright-tests/Dashboards/dashboard-html-chart.spec.js @@ -7,12 +7,24 @@ const testLogger = require('../utils/test-logger.js'); import logData from "../../fixtures/log.json"; import { ingestion } from "./utils/dashIngestion.js"; import PageManager from "../../pages/page-manager"; -import { waitForDashboardPage, deleteDashboard } from "./utils/dashCreation.js"; +import { + waitForDashboardPage, + deleteDashboard, + setupTestDashboard, + cleanupTestDashboard, +} from "./utils/dashCreation.js"; +import { + BLANK_TARGET_HTML_SNIPPET, + DROPPED_TARGET_HTML_SNIPPET, + HTML_LINK_SELECTORS, + POPUP_HTML_SNIPPET, + SHADOWED_FORM_HTML_SNIPPET, +} from "./utils/htmlPanelFixtures.js"; // Each test runs in parallel (mode: "parallel" below), so the name must be // generated fresh per test — a single shared name caused cross-test races // where one test's create/delete collided with another's mid-flight. const generateDashboardName = () => - "Dashboard_" + Math.random().toString(36).substr(2, 9); + "Dashboard_" + Math.random().toString(36).slice(2, 11); // Basic HTML snippet for the first test const BASIC_HTML_SNIPPET = ` @@ -106,7 +118,6 @@ test.describe("HTML chart dashboard", () => { await deleteDashboard(page, randomDashboardName); }); - //skip because value is not set in the variable on other environments its is working test("Should correctly replace the dashboard variable value on the HTML chart.", async ({ page, }) => { @@ -176,8 +187,7 @@ test.describe("HTML chart dashboard", () => { const pm = new PageManager(page); const panelName = pm.dashboardPanelActions.generateUniquePanelName("panel-test"); - const dashboardName = - "Dashboard_" + Math.random().toString(36).substr(2, 9); + const dashboardName = generateDashboardName(); await pm.dashboardList.menuItem("dashboards-item"); await waitForDashboardPage(page); @@ -204,7 +214,6 @@ test.describe("HTML chart dashboard", () => { const panelName = pm.dashboardPanelActions.generateUniquePanelName("panel-test"); const dashboardName = generateDashboardName(); - const sentinel = () => page.evaluate(() => window.__o2HtmlPanelSentinel); await pm.dashboardList.menuItem("dashboards-item"); await waitForDashboardPage(page); @@ -217,7 +226,7 @@ test.describe("HTML chart dashboard", () => { await expect( pm.chartTypeSelector.getHtmlHeading("CSS Rewrite Test") ).toBeVisible(); - await expect(page.locator("img[onerror]")).toHaveCount(0); + await expect(pm.chartTypeSelector.getInjectedOnErrorImages()).toHaveCount(0); await pm.dashboardPanelActions.addPanelName(panelName); await pm.dashboardPanelActions.savePanel(); @@ -225,10 +234,10 @@ test.describe("HTML chart dashboard", () => { await expect( pm.chartTypeSelector.getHtmlHeading("CSS Rewrite Test") ).toBeVisible(); - await expect(page.locator("img[onerror]")).toHaveCount(0); + await expect(pm.chartTypeSelector.getInjectedOnErrorImages()).toHaveCount(0); // an injected image fails to load asynchronously, so give onerror time to fire await page.waitForTimeout(1000); - expect(await sentinel()).toBeUndefined(); + expect(await pm.chartTypeSelector.getHtmlPanelSentinel()).toBeUndefined(); await pm.dashboardCreate.backToDashboardList(); await deleteDashboard(page, dashboardName); @@ -240,8 +249,7 @@ test.describe("HTML chart dashboard", () => { const pm = new PageManager(page); const panelName = pm.dashboardPanelActions.generateUniquePanelName("panel-test"); - const dashboardName = - "Dashboard_" + Math.random().toString(36).substr(2, 9); + const dashboardName = generateDashboardName(); await pm.dashboardList.menuItem("dashboards-item"); await waitForDashboardPage(page); @@ -261,3 +269,173 @@ test.describe("HTML chart dashboard", () => { await deleteDashboard(page, dashboardName); }); }); + +test.describe("HTML chart link targets", () => { + test.beforeEach(async ({ page }, testInfo) => { + testLogger.testStart(testInfo.title, testInfo.file); + await navigateToBase(page); + await page.goto( + `${logData.logsUrl}?org_identifier=${process.env["ORGNAME"]}` + ); + }); + + test("Should keep target=_blank and force a safe rel on links, forms, area and SVG links", { + tag: ["@dashboards", "@htmlChart", "@P1"], + }, async ({ page }) => { + const pm = new PageManager(page); + const dashboardName = generateDashboardName(); + + await setupTestDashboard(page, pm, dashboardName); + await pm.dashboardCreate.addPanel(); + await pm.chartTypeSelector.addHtmlPanelContent(BLANK_TARGET_HTML_SNIPPET); + await expect( + pm.chartTypeSelector.getHtmlHeading("Blank Target Test") + ).toBeVisible(); + + for (const selector of [ + HTML_LINK_SELECTORS.blankLink, + HTML_LINK_SELECTORS.blankMixedCaseLink, + HTML_LINK_SELECTORS.blankForm, + HTML_LINK_SELECTORS.blankArea, + HTML_LINK_SELECTORS.blankSvgLink, + ]) { + await pm.chartTypeSelector.expectHtmlLinkTarget(selector, "_blank"); + } + + await pm.dashboardPanelActions.addPanelName( + pm.dashboardPanelActions.generateUniquePanelName("panel-test") + ); + await pm.dashboardPanelActions.savePanel(); + await cleanupTestDashboard(page, pm, dashboardName); + }); + + test("Should drop named, _top, _self and _parent targets without adding rel", { + tag: ["@dashboards", "@htmlChart", "@P1"], + }, async ({ page }) => { + const pm = new PageManager(page); + const dashboardName = generateDashboardName(); + + await setupTestDashboard(page, pm, dashboardName); + await pm.dashboardCreate.addPanel(); + await pm.chartTypeSelector.addHtmlPanelContent(DROPPED_TARGET_HTML_SNIPPET); + await expect( + pm.chartTypeSelector.getHtmlHeading("Dropped Target Test") + ).toBeVisible(); + + for (const selector of [ + HTML_LINK_SELECTORS.namedLink, + HTML_LINK_SELECTORS.topLink, + HTML_LINK_SELECTORS.selfLink, + HTML_LINK_SELECTORS.parentLink, + HTML_LINK_SELECTORS.namedForm, + ]) { + await pm.chartTypeSelector.expectHtmlLinkTarget(selector, null); + } + + // an element without a target keeps the author's rel + await expect( + pm.chartTypeSelector.getHtmlRendererElement(HTML_LINK_SELECTORS.plainLink) + ).toHaveAttribute("rel", "nofollow"); + + await pm.dashboardPanelActions.addPanelName( + pm.dashboardPanelActions.generateUniquePanelName("panel-test") + ); + await pm.dashboardPanelActions.savePanel(); + await cleanupTestDashboard(page, pm, dashboardName); + }); + + test("Should render a form with a control named getAttribute and drop a nodeName-clobbered form", { + tag: ["@dashboards", "@htmlChart", "@P1"], + }, async ({ page }) => { + const pm = new PageManager(page); + const dashboardName = generateDashboardName(); + + await setupTestDashboard(page, pm, dashboardName); + await pm.dashboardCreate.addPanel(); + await pm.chartTypeSelector.addHtmlPanelContent(SHADOWED_FORM_HTML_SNIPPET); + + await expect( + pm.chartTypeSelector.getHtmlHeading("Shadowed Form Test") + ).toBeVisible(); + await expect( + pm.chartTypeSelector.getHtmlRendererElement(HTML_LINK_SELECTORS.shadowedFormButton) + ).toBeVisible(); + await pm.chartTypeSelector.expectHtmlLinkTarget(HTML_LINK_SELECTORS.shadowedForm, "_blank"); + await expect( + pm.chartTypeSelector.getHtmlRendererElement(HTML_LINK_SELECTORS.clobberedForm) + ).toHaveCount(0); + await pm.chartTypeSelector.expectHtmlLinkTarget(HTML_LINK_SELECTORS.afterFormsLink, "_blank"); + + await pm.dashboardPanelActions.addPanelName( + pm.dashboardPanelActions.generateUniquePanelName("panel-test") + ); + await pm.dashboardPanelActions.savePanel(); + await cleanupTestDashboard(page, pm, dashboardName); + }); + + test("Should open a target=_blank link in a new tab without window.opener", { + tag: ["@dashboards", "@htmlChart", "@P1"], + }, async ({ page }) => { + const pm = new PageManager(page); + const dashboardName = generateDashboardName(); + const popupUrl = new URL("/healthz", process.env["ZO_BASE_URL"]).href; + + await setupTestDashboard(page, pm, dashboardName); + await pm.dashboardCreate.addPanel(); + await pm.chartTypeSelector.addHtmlPanelContent(POPUP_HTML_SNIPPET); + await pm.chartTypeSelector.expectHtmlLinkTarget(HTML_LINK_SELECTORS.popupLink, "_blank"); + + const popup = await pm.chartTypeSelector.clickHtmlRendererElementForPopup( + HTML_LINK_SELECTORS.popupLink + ); + expect(popup.url()).toBe(popupUrl); + expect(await pm.chartTypeSelector.isPopupOpenerNull(popup)).toBe(true); + await popup.close(); + + // the dashboard tab itself must not navigate away + await expect( + pm.chartTypeSelector.getHtmlHeading("Popup Link Test") + ).toBeVisible(); + + await pm.dashboardPanelActions.addPanelName( + pm.dashboardPanelActions.generateUniquePanelName("panel-test") + ); + await pm.dashboardPanelActions.savePanel(); + await cleanupTestDashboard(page, pm, dashboardName); + }); + + test("Should keep target=_blank and safe rel on the saved panel after reload", { + tag: ["@dashboards", "@htmlChart", "@P2"], + }, async ({ page }) => { + const pm = new PageManager(page); + const dashboardName = generateDashboardName(); + + await setupTestDashboard(page, pm, dashboardName); + await pm.dashboardCreate.addPanel(); + await pm.chartTypeSelector.addHtmlPanelContent(BLANK_TARGET_HTML_SNIPPET); + await expect( + pm.chartTypeSelector.getHtmlHeading("Blank Target Test") + ).toBeVisible(); + + await pm.dashboardPanelActions.addPanelName( + pm.dashboardPanelActions.generateUniquePanelName("panel-test") + ); + await pm.dashboardPanelActions.savePanel(); + + await expect( + pm.chartTypeSelector.getHtmlHeading("Blank Target Test") + ).toBeVisible(); + await pm.chartTypeSelector.expectHtmlLinkTarget(HTML_LINK_SELECTORS.blankLink, "_blank"); + await pm.chartTypeSelector.expectHtmlLinkTarget(HTML_LINK_SELECTORS.blankForm, "_blank"); + + await page.reload(); + await expect( + pm.chartTypeSelector.getHtmlHeading("Blank Target Test") + ).toBeVisible(); + await pm.chartTypeSelector.expectHtmlLinkTarget(HTML_LINK_SELECTORS.blankLink, "_blank"); + await pm.chartTypeSelector.expectHtmlLinkTarget(HTML_LINK_SELECTORS.blankMixedCaseLink, "_blank"); + await pm.chartTypeSelector.expectHtmlLinkTarget(HTML_LINK_SELECTORS.blankSvgLink, "_blank"); + + await cleanupTestDashboard(page, pm, dashboardName); + }); +}); diff --git a/tests/ui-testing/playwright-tests/Dashboards/utils/htmlPanelFixtures.js b/tests/ui-testing/playwright-tests/Dashboards/utils/htmlPanelFixtures.js new file mode 100644 index 0000000000..428c5e3e8b --- /dev/null +++ b/tests/ui-testing/playwright-tests/Dashboards/utils/htmlPanelFixtures.js @@ -0,0 +1,62 @@ +// Copyright 2026 OpenObserve Inc. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +export const HTML_LINK_SELECTORS = { + blankLink: "#o2-blank-link", + blankMixedCaseLink: "#o2-blank-mixed-case-link", + blankForm: "#o2-blank-form", + blankArea: "#o2-blank-area", + blankSvgLink: "#o2-blank-svg-link", + namedLink: "#o2-named-link", + topLink: "#o2-top-link", + selfLink: "#o2-self-link", + parentLink: "#o2-parent-link", + namedForm: "#o2-named-form", + plainLink: "#o2-plain-link", + shadowedForm: "#o2-shadowed-form", + shadowedFormButton: "#o2-shadowed-form button", + clobberedForm: "#o2-clobbered-form", + afterFormsLink: "#o2-after-forms-link", + popupLink: "#o2-popup-link", +}; + +export const BLANK_TARGET_HTML_SNIPPET = `

Blank Target Test

+Blank link +Mixed case link +
+area +`; + +export const DROPPED_TARGET_HTML_SNIPPET = `

Dropped Target Test

+Named link +Top link +Self link +Parent link +
+Plain link`; + +// a shadowed getAttribute must not blank the panel; a shadowed nodeName form is dropped as clobbered +export const SHADOWED_FORM_HTML_SNIPPET = `

Shadowed Form Test

+
+ + +
+
+ +
+After forms`; + +export const POPUP_HTML_SNIPPET = `

Popup Link Test

+Open in new tab`; diff --git a/tests/ui-testing/playwright-tests/IAM/iam-roles-editor.spec.js b/tests/ui-testing/playwright-tests/IAM/iam-roles-editor.spec.js index 3729f01c48..6f121c87ea 100644 --- a/tests/ui-testing/playwright-tests/IAM/iam-roles-editor.spec.js +++ b/tests/ui-testing/playwright-tests/IAM/iam-roles-editor.spec.js @@ -272,7 +272,7 @@ test.describe('IAM · Edit Role · navigation and filtering', { tag: '@enterpris // ---------------- empty role, presets, loading ---------------- - test('U-13 · an empty role offers the three presets', { + test('U-13 · the Role Overview Presets panel offers the built-in and job presets', { tag: ['@iam', '@iamRolesEditor', '@P1', '@all'] }, async () => { await pm.rolesPage.gotoRoles(); @@ -280,9 +280,21 @@ test.describe('IAM · Edit Role · navigation and filtering', { tag: '@enterpris await pm.rolesPage.waitForGrantsSettled(0); await expect(pm.rolesPage.summaryEmpty).toBeVisible({ timeout: 15000 }); - await expect(pm.rolesPage.presetCard('Read-Only')).toBeVisible(); - await expect(pm.rolesPage.presetCard('Database Monitoring Viewer')).toBeVisible(); - await expect(pm.rolesPage.presetCard('Kubernetes Viewer')).toBeVisible(); + await expect(pm.rolesPage.presetsPanel).toBeVisible({ timeout: 15000 }); + // Job presets hide only when a module they grant is missing; an enterprise org has all of them. + for (const id of [ + 'readonly', 'dbm', 'k8s', 'rum_viewer', 'rum_editor', + 'alert_manager', 'alert_viewer', 'dashboard_editor', 'dashboard_viewer', 'pipeline_editor', + ]) { + await expect(pm.rolesPage.presetRow(id), `preset ${id} missing from the panel`).toBeVisible(); + } + + // A row opens its detail in place; Back returns to the list. + await pm.rolesPage.openPreset('readonly'); + await expect(pm.rolesPage.presetRow('alert_manager')).toBeHidden(); + await pm.rolesPage.presetBack.click(); + await expect(pm.rolesPage.presetDetail).toBeHidden(); + await expect(pm.rolesPage.presetRow('alert_manager')).toBeVisible(); }); // Guards o2-enterprise#2697: a populated role once read "0 Permissions" before its grants loaded. diff --git a/tests/ui-testing/playwright-tests/IAM/iam-roles-grants.spec.js b/tests/ui-testing/playwright-tests/IAM/iam-roles-grants.spec.js index 891afd95de..e3413c1497 100644 --- a/tests/ui-testing/playwright-tests/IAM/iam-roles-grants.spec.js +++ b/tests/ui-testing/playwright-tests/IAM/iam-roles-grants.spec.js @@ -1,4 +1,4 @@ -// IAM → Edit Role · grant semantics (G-01 .. G-18) +// IAM → Edit Role · grant semantics (G-01 .. G-19) // // Plan: .claude/commands/nvpworkflow/iam-roles-redesign-tests.md // @@ -335,8 +335,8 @@ test.describe('IAM · Edit Role · grant semantics', { tag: '@enterprise' }, () tag: ['@iam', '@iamRolesGrants', '@P1', '@all'] }, async ({ page }) => { const name = await openFresh(page, 'preset'); - await expect(pm.rolesPage.summaryEmpty).toBeVisible({ timeout: 15000 }); - await pm.rolesPage.presetCard('Read-Only').click(); + await expect(pm.rolesPage.presetsPanel).toBeVisible({ timeout: 15000 }); + await pm.rolesPage.applyPreset('readonly'); await pm.rolesPage.save(); const stored = await getPerms(page, name); @@ -384,6 +384,42 @@ test.describe('IAM · Edit Role · grant semantics', { tag: '@enterprise' }, () .toBeGreaterThan(0); }); + test('G-19 · the Alert Manager preset adds module-wide grants plus Streams List and Get, leaving nothing missing', { + tag: ['@iam', '@iamRolesGrants', '@P1', '@all'] + }, async ({ page }) => { + const name = await openFresh(page, 'alertmgr'); + const want = [ + { object: obj('afolder'), permission: 'AllowAll' }, + { object: obj('destination'), permission: 'AllowAll' }, + { object: obj('template'), permission: 'AllowAll' }, + { object: obj('stream'), permission: 'AllowList' }, + { object: obj('stream'), permission: 'AllowGet' }, + ]; + + await expect(pm.rolesPage.presetStatus('alert_manager')).toContainText('+5', { timeout: 15000 }); + await pm.rolesPage.openPreset('alert_manager'); + await expect(pm.rolesPage.presetDetail).toContainText('Nothing will be missing after you add it.'); + await expect(pm.rolesPage.presetApply).toBeEnabled({ timeout: 15000 }); + await pm.rolesPage.presetApply.click(); + await pm.rolesPage.waitForPresetStaged(); + + await pm.rolesPage.openDrawer(); + // One row per object, so Streams List and Get share the stream row. + for (const resource of ['afolder', 'destination', 'template', 'stream']) { + await expect(pm.rolesPage.drawerChange(`${resource}-_all_${org()}-added`)).toBeVisible(); + } + await expect(pm.rolesPage.drawerCompanions).toHaveCount(0); + + const payload = await pm.rolesPage.saveAndCapture({ via: pm.rolesPage.drawerSave }); + expect(payload, 'save fired no PUT').toBeTruthy(); + expect(payload.remove).toEqual([]); + expect(tuples(payload.add)).toEqual(tuples(want)); + + await expect + .poll(async () => tuples(await getPerms(page, name)), { timeout: 15000 }) + .toEqual(tuples(want)); + }); + // ---------------- column select-all ---------------- test('G-14 · ticking the List header in All Modules grants List on every module at once', { diff --git a/tests/ui-testing/playwright-tests/IAM/iam-roles-scale.spec.js b/tests/ui-testing/playwright-tests/IAM/iam-roles-scale.spec.js index 0fdca7b6a0..c074908673 100644 --- a/tests/ui-testing/playwright-tests/IAM/iam-roles-scale.spec.js +++ b/tests/ui-testing/playwright-tests/IAM/iam-roles-scale.spec.js @@ -151,8 +151,9 @@ test.describe('IAM · Edit Role · scale and layout', { tag: '@enterprise' }, () // Looking at the JSON is not editing it — a round trip that stages changes // would let an untouched role be rewritten by a stray Save. - await expect(pm.rolesPage.reviewChangesButton, - 'the Table -> JSON -> Table round trip staged changes').toHaveCount(0); + // Review Changes always renders in table view; List-only grants raise no companion gap, so it stays disabled. + await expect(pm.rolesPage.unsavedCount, 'the Table -> JSON -> Table round trip staged changes').toHaveCount(0); + await expect(pm.rolesPage.reviewChangesButton).toBeDisabled(); const payload = await pm.rolesPage.saveAndCapture({ expectRequest: false }); if (payload) { diff --git a/tests/ui-testing/playwright-tests/IAM/iam-roles-staging.spec.js b/tests/ui-testing/playwright-tests/IAM/iam-roles-staging.spec.js index affdf24f36..7161f97692 100644 --- a/tests/ui-testing/playwright-tests/IAM/iam-roles-staging.spec.js +++ b/tests/ui-testing/playwright-tests/IAM/iam-roles-staging.spec.js @@ -129,7 +129,7 @@ test.describe('IAM · Edit Role · staged changes', { tag: '@enterprise' }, () = }); // The drawer closes once nothing is left to review (useRoleSummary), so there is no empty state to wait for. - test('S-05 · undoing every change closes the drawer and drops Review Changes', { + test('S-05 · undoing every change closes the drawer and disables Review Changes', { tag: ['@iam', '@iamRolesStaging', '@P1', '@all'] }, async ({ page }) => { const name = await openFresh(page, 'undoall'); @@ -143,7 +143,7 @@ test.describe('IAM · Edit Role · staged changes', { tag: '@enterprise' }, () = await page.waitForTimeout(300); } await expect(pm.rolesPage.drawer).toBeHidden({ timeout: 10000 }); - await expect(pm.rolesPage.reviewChangesButton).toBeHidden(); + await expect(pm.rolesPage.reviewChangesButton).toBeDisabled(); // And nothing may reach the API afterwards. const payload = await pm.rolesPage.saveAndCapture({ expectRequest: false }); @@ -257,11 +257,11 @@ test.describe('IAM · Edit Role · staged changes', { tag: '@enterprise' }, () = await page.unroute('**/api/*/roles/*'); }); - // A clean role has nothing to review, so the affordance is not offered at all. - test('S-12 · a clean role offers no Review Changes', { + // A clean role has nothing to review, so Review Changes shows but is disabled. + test('S-12 · a clean role shows Review Changes disabled', { tag: ['@iam', '@iamRolesStaging', '@P2', '@all'] }, async ({ page }) => { await openFresh(page, 'clean'); - await expect(pm.rolesPage.reviewChangesButton).toBeHidden(); + await expect(pm.rolesPage.reviewChangesButton).toBeDisabled(); }); }); diff --git a/tests/ui-testing/playwright-tests/Metrics/metrics-query-history.spec.js b/tests/ui-testing/playwright-tests/Metrics/metrics-query-history.spec.js index 7b67185e11..d8ae345351 100644 --- a/tests/ui-testing/playwright-tests/Metrics/metrics-query-history.spec.js +++ b/tests/ui-testing/playwright-tests/Metrics/metrics-query-history.spec.js @@ -43,28 +43,41 @@ test.describe('Metrics editor query history', () => { await custom.click(); await expect(custom).toHaveAttribute('data-state', 'on'); await runAndRecord(page, pm, first); + await runAndRecord(page, pm, second); await metrics.openQueryHistory(); await metrics.searchQueryHistory(String(marker)); await expect(metrics.historyRow(first)).toHaveCount(1, { timeout: 15000 }); + await expect(metrics.historyRow(second)).toHaveCount(1, { timeout: 15000 }); - // Star it: the toggle flips, and the entry survives "starred only". + // Star one: the toggle flips, and "starred only" keeps it and drops the unstarred one. const starred = page.waitForResponse(isHistoryCall('PATCH')); await metrics.historyStarButton(first).click(); expect((await starred).ok()).toBe(true); await expect(metrics.historyStarButton(first)).toHaveAttribute('aria-label', 'Unstar'); + const starredOnlyBox = metrics.historyStarredOnly.locator('[role="checkbox"]'); + // The new filter shows a skeleton until its list answers, and a skeleton has no rows to find. + const starredList = page.waitForResponse( + (res) => { + const params = new URL(res.url()).searchParams; + return isHistoryCall('GET')(res) && params.get('starred') === 'true' && !params.has('count_only'); + }, + { timeout: 30000 }, + ); await metrics.historyStarredOnly.click(); + await expect(starredOnlyBox).toHaveAttribute('aria-checked', 'true'); + const starredRows = await (await starredList).json(); + expect(starredRows.map((row) => row.query)).toContain(first); + expect(starredRows.map((row) => row.query)).not.toContain(second); await expect(metrics.historyRow(first)).toHaveCount(1, { timeout: 15000 }); + await expect(metrics.historyRow(second)).toHaveCount(0); await metrics.historyStarredOnly.click(); - await page.keyboard.press('Escape'); - await expect(metrics.historyTable).toBeHidden(); - - // Replace the editor's query, then load the starred entry back in place. - await runAndRecord(page, pm, second); - await metrics.openQueryHistory(); + await expect(starredOnlyBox).toHaveAttribute('aria-checked', 'false'); await expect(metrics.historyRow(second)).toHaveCount(1, { timeout: 15000 }); + + // The editor holds the second query; load the starred entry back in place. await metrics.loadHistoryEntry(first); - await expect(metrics.historyTable).toBeHidden(); + await expect(metrics.historyDrawer).toBeHidden(); await expect .poll(() => pm.metricsQueryEditorPage.getCurrentQueryText(), { timeout: 15000 }) .toBe(first); @@ -72,7 +85,8 @@ test.describe('Metrics editor query history', () => { // Starred entries are never reaped, so remove this run's from the user's history. await metrics.openQueryHistory(); for (const query of [first, second]) { - const deleted = page.waitForResponse(isHistoryCall('DELETE')); + // The DELETE waits for the row's 6s Undo toast to close. + const deleted = page.waitForResponse(isHistoryCall('DELETE'), { timeout: 30000 }); await metrics.deleteHistoryEntry(query); expect((await deleted).ok()).toBe(true); await expect(metrics.historyRow(query)).toHaveCount(0, { timeout: 15000 }); diff --git a/tests/ui-testing/playwright-tests/Profiles/profiles-chart-breakdown.spec.js b/tests/ui-testing/playwright-tests/Profiles/profiles-chart-breakdown.spec.js new file mode 100644 index 0000000000..f3618df837 --- /dev/null +++ b/tests/ui-testing/playwright-tests/Profiles/profiles-chart-breakdown.spec.js @@ -0,0 +1,187 @@ +// Continuous Profiling time-series card: units, single points, drag-to-range, Breakdown and its legend. + +const { test, expect, navigateToBase } = require('../utils/enhanced-baseFixtures.js'); +const testLogger = require('../utils/test-logger.js'); +const PageManager = require('../../pages/page-manager.js'); +const { + seedReferenceStream, + seedProfilesStream, + manyVersionsResources, + deleteProfilesStream, + MINUTE_MS, +} = require('../utils/profiles-ingestion.js'); + +test.describe('Profiles chart and breakdown', () => { + test.describe.configure({ mode: 'parallel' }); + let api; + let ref; + let many; + let pm; + + test.beforeAll(async ({ playwright }) => { + api = await playwright.request.newContext(); + ref = await seedReferenceStream(api, 'chart'); + many = await seedProfilesStream(api, 'chartmany', manyVersionsResources, { service: 'checkout', total: 7_800_000_000 }); + }); + + test.afterAll(async () => { + await deleteProfilesStream(api, ref.stream); + await deleteProfilesStream(api, many.stream); + await api.dispose(); + }); + + test.beforeEach(async ({ page }, testInfo) => { + testLogger.testStart(testInfo.title, testInfo.file); + pm = new PageManager(page); + await pm.profilesPage.mockServerConfig(); + await pm.profilesPage.hideFlameZoomHint(); + await navigateToBase(page); + }); + + test.afterEach(async ({}, testInfo) => { + await pm.profilesPage.teardown(); + testLogger.testEnd(testInfo.title, testInfo.status); + }); + + test('CPU chart is drawn in cores', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref); + const tips = await pm.profilesPage.collectSeriesTooltips(); + testLogger.info('Series tooltips', { tips }); + expect(tips.some((tip) => tip.endsWith('cpu 0.4 cores'))).toBe(true); + expect(tips.some((tip) => tip.endsWith('cpu 0.785 cores'))).toBe(true); + expect(tips.some((tip) => tip.endsWith('cpu 0.015 cores'))).toBe(true); + }); + + test('Memory chart uses binary byte units', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref, { service: 'payments', type: 'inuse_space', unit: 'bytes' }); + const tips = await pm.profilesPage.collectSeriesTooltips(); + testLogger.info('Series tooltips', { tips }); + expect(tips.some((tip) => tip.endsWith('inuse_space 2 MiB'))).toBe(true); + expect(tips.some((tip) => tip.endsWith('inuse_space 4 MiB'))).toBe(true); + expect(tips.filter((tip) => / MB$| KB$/.test(tip))).toEqual([]); + }); + + test('A single point is drawn where it sits in the run window', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref, { service: 'catalog' }); + const tips = await pm.profilesPage.collectSeriesTooltips(40); + expect(tips.some((tip) => tip.endsWith('cpu 0.5515 cores'))).toBe(true); + const drawn = await pm.profilesPage.seriesBlueCentre(); + expect(drawn.count).toBeGreaterThan(0); + expect(drawn.centre).toBeGreaterThan(0.7); + expect(drawn.centre).toBeLessThan(0.95); + }); + + test('Dragging across the chart picks a minute-rounded range and runs', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref); + const calls = pm.profilesPage.trackRequests(); + await pm.profilesPage.dragAcrossChart(0.3, 0.7); + await expect.poll(() => calls.merge.length, { timeout: 15000 }).toBeGreaterThan(0); + expect(calls.series.length).toBeGreaterThan(0); + await pm.profilesPage.waitForRunIdle(); + await expect.poll(() => pm.profilesPage.currentQuery().from).not.toBe(String(ref.window.from)); + const query = pm.profilesPage.currentQuery(); + const from = Number(query.from); + const to = Number(query.to); + expect(query.period).toBeUndefined(); + expect(from % (MINUTE_MS * 1000)).toBe(0); + expect(to % (MINUTE_MS * 1000)).toBe(0); + expect(from).toBeGreaterThan(ref.window.from); + expect(to).toBeLessThan(ref.window.to); + expect(to).toBeGreaterThan(from); + expect(calls.merge.at(-1).body.start_time).toBe(from); + await pm.profilesPage.expectRunPending(false); + }); + + test('Breakdown lists labels but not per-request ids', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref); + expect(await pm.profilesPage.breakdownOptionLabels()).toEqual([ + 'None', + 'deployment_environment', + 'host_name', + 'service_instance_id', + 'service_version', + ]); + }); + + test('Breakdown draws one line per value and re-fetches only the series', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref); + const calls = pm.profilesPage.trackRequests(); + await pm.profilesPage.pickBreakdown('service_version'); + await pm.profilesPage.expectBreakdownChips(['v1.4.0', 'v1.5.0']); + expect(calls.series).toHaveLength(1); + expect(calls.series[0].body.group_by).toBe('service_version'); + expect(calls.series[0].body.group_limit).toBe(10); + expect(calls.merge).toHaveLength(0); + await pm.profilesPage.expectUrlParam('breakdown', 'service_version'); + }); + + test('A legend chip filters the page to that value', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref, { extra: { breakdown: 'service_version' } }); + await pm.profilesPage.expectBreakdownChips(['v1.4.0', 'v1.5.0']); + await pm.profilesPage.clickBreakdownChip('1'); + await pm.profilesPage.expectFilterChip('service_version', 'v1.5.0'); + await pm.profilesPage.expectRunPending(true); + await pm.profilesPage.clickRun(); + await pm.profilesPage.expectTopRow(0, { name: 'db_query', self: '4 s', selfPercent: '50.00%' }); + await pm.profilesPage.expectUrlParam('filters', 'service_version=v1.5.0'); + }); + + test('More than ten values fold into a disabled Others chip', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(many, { extra: { breakdown: 'service_version' } }); + await pm.profilesPage.expectBreakdownChips([ + 'v1.3.0', + 'v1.4.0', + 'v1.5.0', + 'v1.6.0', + 'v1.7.0', + 'v1.8.0', + 'v1.9.0', + 'v1.10.0', + 'v1.11.0', + 'v1.12.0', + 'Others', + ]); + await pm.profilesPage.expectOthersChipDisabled(); + }); + + test('Breakdown is disabled until the first run answers', { tag: ['@all', '@profiles', '@P3'] }, async () => { + const release = await pm.profilesPage.holdApi('merge'); + await pm.profilesPage.openSeededNoWait({ stream: ref.stream, ...ref.window, service: 'checkout' }); + await pm.profilesPage.expectState('seriesSkeleton'); + await pm.profilesPage.expectBreakdownDisabled(); + release(); + await pm.profilesPage.waitForRunIdle(); + await pm.profilesPage.expectTopRow(0, { name: 'db_query', self: '6 s' }); + }); + + test('Breakdown re-charts the last run, not unrun selector changes', { tag: ['@all', '@profiles', '@P3'] }, async () => { + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.selectService('payments'); + await pm.profilesPage.expectRunPending(true); + const calls = pm.profilesPage.trackRequests(); + await pm.profilesPage.pickBreakdown('service_version'); + await pm.profilesPage.expectBreakdownChips(['v1.4.0', 'v1.5.0']); + expect(calls.series).toHaveLength(1); + expect(calls.series[0].body.service_name).toBe('checkout'); + expect(calls.series[0].body.group_by).toBe('service_version'); + }); + + test('Expand hides the chart and collapse brings it back', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.expectSeriesCardVisible(true); + await pm.profilesPage.toggleResultsExpand(); + await pm.profilesPage.expectSeriesCardVisible(false); + await pm.profilesPage.expectTopRow(0, { name: 'db_query' }); + await pm.profilesPage.toggleResultsExpand(); + await pm.profilesPage.expectSeriesCardVisible(true); + }); + + test('An empty series shows No data while the tables keep their rows', { tag: ['@all', '@profiles', '@P3'] }, async () => { + await pm.profilesPage.mockApi('series', { + json: { unit: 'nanoseconds', profile_type: 'cpu', step_secs: 10, series: [], took: 1 }, + }); + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.expectState('noSeries', { title: 'No data' }); + await pm.profilesPage.expectTopRow(0, { name: 'db_query', self: '6 s' }); + }); +}); diff --git a/tests/ui-testing/playwright-tests/Profiles/profiles-compare.spec.js b/tests/ui-testing/playwright-tests/Profiles/profiles-compare.spec.js new file mode 100644 index 0000000000..775df91b5f --- /dev/null +++ b/tests/ui-testing/playwright-tests/Profiles/profiles-compare.spec.js @@ -0,0 +1,174 @@ +// Continuous Profiling Compare view: diff table, diff flame, side by side, sides and messages. + +const { test, expect, navigateToBase } = require('../utils/enhanced-baseFixtures.js'); +const testLogger = require('../utils/test-logger.js'); +const PageManager = require('../../pages/page-manager.js'); +const { CHECKOUT_FRAMES: F, fullWidthFrame } = require('../../pages/profilesPages/profilesPage.js'); +const { + seedReferenceStream, + seedProfilesStream, + noLabelResources, + deleteProfilesStream, +} = require('../utils/profiles-ingestion.js'); + +test.describe('Profiles compare', () => { + test.describe.configure({ mode: 'parallel' }); + let api; + let ref; + let bare; + let pm; + + test.beforeAll(async ({ playwright }) => { + api = await playwright.request.newContext(); + ref = await seedReferenceStream(api, 'compare'); + bare = await seedProfilesStream(api, 'cmpbare', noLabelResources, { service: 'bare-svc', total: 1_000_000_000 }); + }); + + test.afterAll(async () => { + await deleteProfilesStream(api, ref.stream); + await deleteProfilesStream(api, bare.stream); + await api.dispose(); + }); + + test.beforeEach(async ({ page }, testInfo) => { + testLogger.testStart(testInfo.title, testInfo.file); + pm = new PageManager(page); + await pm.profilesPage.mockServerConfig(); + await pm.profilesPage.hideFlameZoomHint(); + await navigateToBase(page); + }); + + test.afterEach(async ({}, testInfo) => { + await pm.profilesPage.teardown(); + testLogger.testEnd(testInfo.title, testInfo.status); + }); + + test('Compare defaults to service_version with the latest two values', { tag: ['@all', '@profiles', '@P0'] }, async () => { + const calls = pm.profilesPage.trackRequests(); + await pm.profilesPage.gotoRef(ref, { view: 'compare' }); + await pm.profilesPage.waitForCompareIdle(); + await pm.profilesPage.expectCompareSides('service_version', 'v1.4.0', 'v1.5.0'); + expect(calls.diff).toHaveLength(1); + const body = calls.diff[0].body; + expect(body.max_nodes).toBe(4096); + expect(body.baseline.filters).toEqual([{ key: 'service_version', op: '=', value: 'v1.4.0' }]); + expect(body.comparison.filters).toEqual([{ key: 'service_version', op: '=', value: 'v1.5.0' }]); + expect(body.baseline.service_name).toBe('checkout'); + expect(body.comparison.profile_type).toBe('cpu'); + }); + + test('Diff Table shows each side and the share change in points', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'compare', extra: { compare_display: 'table' } }); + await pm.profilesPage.waitForCompareIdle(); + await pm.profilesPage.expectCompareTableRow('json_encode', { + before: '1 s', + after: '3 s', + beforePercent: '25.00%', + afterPercent: '37.50%', + change: '+12.50 pt', + }); + await pm.profilesPage.expectCompareTableRow('process_payment', { before: '0 ns', after: '250 ms', change: '+3.13 pt' }); + await pm.profilesPage.expectCompareTableRow('db_query', { before: '2 s', after: '4 s', change: '0.00 pt' }); + await pm.profilesPage.expectCompareTableRow('handle_order', { before: '4 s', after: '7.85 s', change: '−1.88 pt' }); + await pm.profilesPage.expectCompareTableRow('gc_sweep', { before: '0 ns', after: '150 ms', change: '+1.88 pt' }); + }); + + test('Diff Flame Graph tooltip shows both sides and the change', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'compare' }); + await pm.profilesPage.waitForCompareIdle(); + const scope = pm.profilesPage.compareFlame; + await pm.profilesPage.hoverFrame(fullWidthFrame(0), 'v1.5.0: 8 s', { scope }); + await pm.profilesPage.hoverFrame(F.db_query, 'db_query', { scope }); + await pm.profilesPage.expectFlameTooltipContains('db_query', ['v1.4.0', '2 s', 'v1.5.0', '4 s', '50.00%', 'Change', '0.00 pt']); + await pm.profilesPage.hoverFrame({ depth: 3, left: 50, width: 37.5 }, 'json_encode', { scope }); + await pm.profilesPage.expectFlameTooltipContains('json_encode', ['1 s', '25.00%', '3 s', '37.50%', '+12.50 pt']); + }); + + test('Side by side names each graph by its value and total', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'compare' }); + await pm.profilesPage.waitForCompareIdle(); + await pm.profilesPage.pickCompareDisplay('sides'); + await pm.profilesPage.hoverFrame(fullWidthFrame(0), 'v1.4.0: 4 s', { scope: pm.profilesPage.compareSideBaseline }); + await pm.profilesPage.hoverFrame(fullWidthFrame(0), 'v1.5.0: 8 s', { scope: pm.profilesPage.compareSideComparison }); + await pm.profilesPage.expectUrlParam('compare_display', 'sides'); + }); + + test('Swap exchanges the sides and re-runs the diff', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'compare' }); + await pm.profilesPage.waitForCompareIdle(); + const calls = pm.profilesPage.trackRequests(); + await pm.profilesPage.clickCompareSwap(); + await pm.profilesPage.expectCompareSides('service_version', 'v1.5.0', 'v1.4.0'); + expect(calls.diff).toHaveLength(1); + expect(calls.diff[0].body.baseline.filters).toEqual([{ key: 'service_version', op: '=', value: 'v1.5.0' }]); + expect(calls.diff[0].body.comparison.filters).toEqual([{ key: 'service_version', op: '=', value: 'v1.4.0' }]); + await pm.profilesPage.expectUrlParam('compare_a', 'v1.5.0'); + await pm.profilesPage.expectUrlParam('compare_b', 'v1.4.0'); + }); + + test('Compare state survives a reload', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'compare' }); + await pm.profilesPage.waitForCompareIdle(); + await pm.profilesPage.pickCompareDisplay('sides'); + await pm.profilesPage.clickCompareSwap(); + await pm.profilesPage.expectUrlParam('compare_a', 'v1.5.0'); + await pm.profilesPage.reload(); + await pm.profilesPage.waitForCompareIdle(); + await pm.profilesPage.expectViewActive('compare'); + await pm.profilesPage.expectCompareSides('service_version', 'v1.5.0', 'v1.4.0'); + await pm.profilesPage.hoverFrame(fullWidthFrame(0), 'v1.5.0: 8 s', { scope: pm.profilesPage.compareSideBaseline }); + }); + + test("Each side's list leaves out the other side's value", { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'compare' }); + await pm.profilesPage.waitForCompareIdle(); + await expect.poll(() => pm.profilesPage.compareBaselineOptions(), { timeout: 15000 }).toEqual(['v1.4.0']); + }); + + test('Changing the label picks that label\'s defaults', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'compare' }); + await pm.profilesPage.waitForCompareIdle(); + await pm.profilesPage.pickCompareLabel('deployment_environment'); + await pm.profilesPage.expectCompareSides('deployment_environment', 'prod', 'stage'); + }); + + test('A label with one value in range explains why there is no diff', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'compare', filters: 'service_version=v1.4.0' }); + await pm.profilesPage.waitForCompareIdle(); + await pm.profilesPage.pickCompareLabel('host_name'); + await pm.profilesPage.expectCompareMessage('This label has fewer than two values in the time range.'); + }); + + test('One side with no data says so', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.patchApiResponse('diff', () => ({ comparison_total: 0 })); + await pm.profilesPage.gotoRef(ref, { view: 'compare' }); + await pm.profilesPage.waitForCompareIdle(); + await pm.profilesPage.expectCompareMessage('One side has no data. Pick other values or a wider time range.'); + }); + + test('A filter on the compare label pre-selects that value as B', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'compare', filters: 'service_version=v1.4.0' }); + await pm.profilesPage.waitForCompareIdle(); + await pm.profilesPage.expectCompareSides('service_version', 'v1.5.0', 'v1.4.0'); + }); + + test('Search filters the Diff Table', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'compare', extra: { compare_display: 'table' } }); + await pm.profilesPage.waitForCompareIdle(); + await pm.profilesPage.search('json'); + await pm.profilesPage.expectCompareTableRowNames(['json_encode']); + }); + + test('A diff error is shown in place of the graph', { tag: ['@all', '@profiles', '@P3'] }, async () => { + await pm.profilesPage.mockApi('diff', { status: 400, json: { code: 400, message: 'boom' } }); + await pm.profilesPage.gotoRef(ref, { view: 'compare' }); + await pm.profilesPage.waitForCompareIdle(); + await pm.profilesPage.expectCompareMessage('boom'); + }); + + test('A stream with no labels has nothing to compare by', { tag: ['@all', '@profiles', '@P3'] }, async () => { + await pm.profilesPage.gotoRef(bare, { service: 'bare-svc', view: 'compare' }); + await pm.profilesPage.waitForCompareIdle(); + await pm.profilesPage.expectCompareMessage('No label to compare by.'); + }); +}); diff --git a/tests/ui-testing/playwright-tests/Profiles/profiles-flame-graph.spec.js b/tests/ui-testing/playwright-tests/Profiles/profiles-flame-graph.spec.js new file mode 100644 index 0000000000..30fe4cf7f8 --- /dev/null +++ b/tests/ui-testing/playwright-tests/Profiles/profiles-flame-graph.spec.js @@ -0,0 +1,245 @@ +// Continuous Profiling flame graph: zoom, frame menu, hint, layout options, grouped frames. + +const { test, expect, navigateToBase } = require('../utils/enhanced-baseFixtures.js'); +const testLogger = require('../utils/test-logger.js'); +const PageManager = require('../../pages/page-manager.js'); +const { CHECKOUT_FRAMES: F, CATALOG_FRAMES: C, fullWidthFrame: FULL_WIDTH } = require('../../pages/profilesPages/profilesPage.js'); +const { seedReferenceStream, deleteProfilesStream } = require('../utils/profiles-ingestion.js'); + +test.describe('Profiles flame graph', () => { + test.describe.configure({ mode: 'parallel' }); + let api; + let ref; + let pm; + + test.beforeAll(async ({ playwright }) => { + api = await playwright.request.newContext(); + ref = await seedReferenceStream(api, 'flame'); + }); + + test.afterAll(async () => { + await deleteProfilesStream(api, ref.stream); + await api.dispose(); + }); + + test.beforeEach(async ({ page }, testInfo) => { + testLogger.testStart(testInfo.title, testInfo.file); + pm = new PageManager(page); + await pm.profilesPage.mockServerConfig(); + await navigateToBase(page); + }); + + test.afterEach(async ({}, testInfo) => { + await pm.profilesPage.teardown(); + testLogger.testEnd(testInfo.title, testInfo.status); + }); + + test.describe('frames', () => { + test.beforeEach(async () => { + await pm.profilesPage.hideFlameZoomHint(); + }); + + test('Flame graph renders with the root total', { tag: ['@all', '@profiles', '@P0'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'flame' }); + await pm.profilesPage.expectFlameViewOptions(); + await pm.profilesPage.expectFlameRootSummary('Total: 12 s (CPU duration)'); + await pm.profilesPage.hoverFrame(F.db_query, 'db_query'); + await pm.profilesPage.expectFlameTooltipContains('db_query', ['6 s', '50.00%', 'Share of caller', '50.63%']); + }); + + test('Left-click zooms into a frame and writes focus to the URL without history', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'flame' }); + const before = await pm.profilesPage.historyLength(); + await pm.profilesPage.clickFrame(F.db_query, 'db_query'); + await pm.profilesPage.expectUrlParam('focus', F.db_query.path); + expect(await pm.profilesPage.historyLength()).toBe(before); + await pm.profilesPage.hoverFrame(FULL_WIDTH(3), 'db_query'); + }); + + test('Clicking a dimmed caller zooms out to it and the root resets', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'flame' }); + await pm.profilesPage.clickFrame(F.db_query, 'db_query'); + await pm.profilesPage.expectUrlParam('focus', F.db_query.path); + await pm.profilesPage.clickFrame(FULL_WIDTH(2), 'handle_order'); + await pm.profilesPage.expectUrlParam('focus', F.handle_order.path); + await pm.profilesPage.clickFrame(FULL_WIDTH(0), 'Click to zoom out'); + await pm.profilesPage.expectUrlParamAbsent('focus'); + await pm.profilesPage.hoverFrame(F.json_encode, 'json_encode'); + }); + + test('A re-run clears a stale flame focus', { tag: ['@all', '@profiles', '@P0'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'flame' }); + await pm.profilesPage.clickFrame(F.handle_order, 'handle_order'); + await pm.profilesPage.expectUrlParam('focus', F.handle_order.path); + await pm.profilesPage.addFilter('deployment_environment', 'stage'); + await pm.profilesPage.clickRun(); + await pm.profilesPage.expectUrlParam('filters', 'deployment_environment=stage'); + await pm.profilesPage.expectUrlParamAbsent('focus'); + await pm.profilesPage.expectFlameRootSummary('Total: 150 ms (CPU duration)'); + await pm.profilesPage.hoverFrame({ depth: 1, left: 0, width: 100 }, 'runtime_main'); + await pm.profilesPage.hoverFrame({ depth: 2, left: 0, width: 100 }, 'gc_sweep'); + await pm.profilesPage.hoverFrame(FULL_WIDTH(0), 'Total: 150 ms'); + await pm.profilesPage.expectFlameTooltipLacks('Total: 150 ms', 'Click to zoom out'); + }); + + test('A focus seeded in the URL survives the first run', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'flame', extra: { focus: F.handle_order.path } }); + await pm.profilesPage.expectUrlParam('focus', F.handle_order.path); + await pm.profilesPage.hoverFrame(FULL_WIDTH(2), 'handle_order'); + await pm.profilesPage.hoverFrame(FULL_WIDTH(0), 'Click to zoom out'); + await pm.profilesPage.hoverFrame({ depth: 3, left: 0, width: 50.633 }, 'db_query'); + }); + + test('Right-click opens the frame menu with the frame actions', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'flame' }); + await pm.profilesPage.rightClickFrame(F.json_encode, 'json_encode'); + await pm.profilesPage.expectFlameMenu( + ['copy-name', 'zoom-in', 'search-function', 'sandwich', 'details'], + ['zoom-reset', 'zoom-out', 'zoom-out-here'], + ); + }); + + test('Frame menu on the zoomed frame and on a caller offers zoom out', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'flame' }); + await pm.profilesPage.clickFrame(F.db_query, 'db_query'); + await pm.profilesPage.expectUrlParam('focus', F.db_query.path); + await pm.profilesPage.rightClickFrame(FULL_WIDTH(3), 'db_query'); + await pm.profilesPage.expectFlameMenu(['zoom-out', 'zoom-reset'], ['zoom-in', 'zoom-out-here']); + await pm.profilesPage.pressEscape(); + await pm.profilesPage.expectFlameMenuHidden(); + await pm.profilesPage.rightClickFrame(FULL_WIDTH(2), 'handle_order'); + await pm.profilesPage.expectFlameMenu(['zoom-out-here', 'zoom-reset'], ['zoom-in', 'zoom-out']); + await pm.profilesPage.clickFlameMenuItem('zoom-out-here'); + await pm.profilesPage.expectUrlParam('focus', F.handle_order.path); + await pm.profilesPage.rightClickFrame(FULL_WIDTH(2), 'handle_order'); + await pm.profilesPage.clickFlameMenuItem('zoom-reset'); + await pm.profilesPage.expectUrlParamAbsent('focus'); + }); + + test('Frame menu actions copy, highlight, open details and open sandwich', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'flame' }); + await pm.profilesPage.rightClickFrame(F.json_encode, 'json_encode'); + await pm.profilesPage.clickFlameMenuItem('copy-name'); + await pm.profilesPage.expectCopiedToast(); + expect(await pm.profilesPage.readClipboard()).toBe('json_encode'); + await pm.profilesPage.rightClickFrame(F.json_encode, 'json_encode'); + await pm.profilesPage.clickFlameMenuItem('search-function'); + await pm.profilesPage.expectSearchValue('json_encode'); + await pm.profilesPage.expectUrlParam('search', 'json_encode'); + await pm.profilesPage.rightClickFrame(F.json_encode, 'json_encode'); + await pm.profilesPage.clickFlameMenuItem('details'); + await pm.profilesPage.expectDetails({ name: 'json_encode', self: '4 s · 33.33%', total: '4 s · 33.33%' }); + await pm.profilesPage.rightClickFrame(F.db_query, 'db_query'); + await pm.profilesPage.clickFlameMenuItem('sandwich'); + await pm.profilesPage.expectViewActive('sandwich'); + await pm.profilesPage.expectSandwichFunction('db_query'); + await pm.profilesPage.expectUrlParam('fn', 'db_query'); + }); + + test('The ⋮ button on a hovered frame opens its menu and Escape closes it', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'flame' }); + await pm.profilesPage.hoverFrame(F.db_query, 'db_query'); + await pm.profilesPage.expectFrameMoreVisible(); + await pm.profilesPage.clickFrameMore(); + await pm.profilesPage.expectFlameMenu(['copy-name', 'zoom-in']); + await pm.profilesPage.pressEscape(); + await pm.profilesPage.expectFlameMenuHidden(); + }); + + test('Ctrl + wheel zooms the view and a drag pans it', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'flame' }); + await pm.profilesPage.expectFlameGrabCursor(false); + const initial = await pm.profilesPage.flameCanvasPng(); + const point = await pm.profilesPage.ctrlWheelOnFrame(F.db_query, 'db_query', -600); + await pm.profilesPage.expectFlameGrabCursor(true); + const zoomed = await pm.profilesPage.flameCanvasPng(); + expect(zoomed.equals(initial)).toBe(false); + await pm.profilesPage.dragOnFlame(point, 200); + const panned = await pm.profilesPage.flameCanvasPng(); + expect(panned.equals(zoomed)).toBe(false); + await pm.profilesPage.expectNotZoomedVia(FULL_WIDTH(0), 'Total: 12 s'); + await pm.profilesPage.expectUrlParamAbsent('focus'); + }); + + test('Root at bottom flips the drawing and label alignment switches', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'flame' }); + await pm.profilesPage.expectRootAtBottom(false); + await pm.profilesPage.expectAlignActive('left'); + await pm.profilesPage.toggleRootAtBottom(); + await pm.profilesPage.expectRootAtBottom(true); + await pm.profilesPage.hoverFrame(FULL_WIDTH(0), 'Total: 12 s', { rootAtBottom: true, maxDepth: 3 }); + await pm.profilesPage.hoverFrame(F.db_query, 'db_query', { rootAtBottom: true, maxDepth: 3 }); + await pm.profilesPage.clickAlign('right'); + await pm.profilesPage.expectAlignActive('right'); + }); + + test('Grouped frames ignore clicks and have no menu', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref, { service: 'catalog', view: 'flame' }); + await pm.profilesPage.expectFlameRootSummary('Total: 5.52 s (CPU duration)'); + await pm.profilesPage.hoverFrame(C.truncated, '[truncated]'); + await pm.profilesPage.expectFlameTooltipContains('[truncated]', ['Callers beyond 1000 frames, cut from this stack.']); + await pm.profilesPage.clickFrame(C.truncated, '[truncated]'); + await pm.profilesPage.expectNotZoomedVia(C.catalog_main, 'catalog_main'); + await pm.profilesPage.rightClickFrame(C.truncated, '[truncated]'); + await pm.profilesPage.expectNoFrameMenuVia(C.catalog_main, 'catalog_main'); + await pm.profilesPage.hoverFrame(C.smallFrames, '55 small frames'); + await pm.profilesPage.expectFlameTooltipContains('55 small frames', ['Too thin to draw one by one.']); + await pm.profilesPage.clickFrame(C.smallFrames, '55 small frames'); + await pm.profilesPage.expectNotZoomedVia(C.catalog_main, 'catalog_main'); + await pm.profilesPage.expectUrlParamAbsent('focus'); + await pm.profilesPage.hoverFrame(FULL_WIDTH(0), 'Total: 5.52 s'); + await pm.profilesPage.expectFlameTooltipLacks('Total: 5.52 s', 'Click to zoom out'); + }); + + test('Flame graph resizes with the Function details panel', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'flame' }); + const wide = await pm.profilesPage.flamePlotWidth(); + await expect.poll(() => pm.profilesPage.flameCanvasWidth()).toBeGreaterThan(wide - 4); + await pm.profilesPage.rightClickFrame(F.db_query, 'db_query'); + await pm.profilesPage.clickFlameMenuItem('details'); + await pm.profilesPage.expectDetails({ name: 'db_query' }); + await expect.poll(() => pm.profilesPage.flamePlotWidth()).toBeLessThan(wide - 200); + const narrow = await pm.profilesPage.flamePlotWidth(); + await expect.poll(() => pm.profilesPage.flameCanvasWidth()).toBeLessThan(narrow + 4); + await pm.profilesPage.clickDetails('close'); + await expect.poll(() => pm.profilesPage.flamePlotWidth()).toBeGreaterThan(wide - 4); + await expect.poll(() => pm.profilesPage.flameCanvasWidth()).toBeGreaterThan(wide - 4); + await pm.profilesPage.hoverFrame(F.json_encode, 'json_encode'); + }); + + test('Search highlights only the matching frame', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'flame' }); + await pm.profilesPage.hoverFrame(F.runtime_main, 'runtime_main'); + const jsonPlain = await pm.profilesPage.frameLabelPng(F.json_encode); + const dbPlain = await pm.profilesPage.frameLabelPng(F.db_query); + await pm.profilesPage.search('json'); + await pm.profilesPage.expectSearchSummary('1 function · 33.33% of samples'); + await expect.poll(async () => (await pm.profilesPage.frameLabelPng(F.json_encode)).equals(jsonPlain)).toBe(false); + expect((await pm.profilesPage.frameLabelPng(F.db_query)).equals(dbPlain)).toBe(true); + }); + }); + + test.describe('zoom hint', () => { + test("Don't show again hides the hint for good", { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'flame' }); + await pm.profilesPage.expectZoomHint(true); + await pm.profilesPage.expectZoomHintText(/Use (⌘|Ctrl) \+ scroll to zoom/); + await pm.profilesPage.clickZoomHint('never'); + await pm.profilesPage.expectZoomHint(false); + expect(await pm.profilesPage.zoomHintStorageValue()).toBe('1'); + await pm.profilesPage.reload(); + await pm.profilesPage.hoverFrame(F.db_query, 'db_query'); + await pm.profilesPage.expectZoomHint(false); + }); + + test('Closing the hint hides it for this graph only', { tag: ['@all', '@profiles', '@P3'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'flame' }); + await pm.profilesPage.expectZoomHint(true); + await pm.profilesPage.clickZoomHint('close'); + await pm.profilesPage.expectZoomHint(false); + expect(await pm.profilesPage.zoomHintStorageValue()).toBeNull(); + await pm.profilesPage.reload(); + await pm.profilesPage.expectZoomHint(true); + }); + }); +}); diff --git a/tests/ui-testing/playwright-tests/Profiles/profiles-navigation-entry.spec.js b/tests/ui-testing/playwright-tests/Profiles/profiles-navigation-entry.spec.js new file mode 100644 index 0000000000..85438f30f7 --- /dev/null +++ b/tests/ui-testing/playwright-tests/Profiles/profiles-navigation-entry.spec.js @@ -0,0 +1,159 @@ +// Continuous Profiling entry points: left menu, feature flag, Data page, Data Sources, no-stream CTA, Trace span link. + +const { test, expect, navigateToBase } = require('../utils/enhanced-baseFixtures.js'); +const testLogger = require('../utils/test-logger.js'); +const PageManager = require('../../pages/page-manager.js'); +const { + seedReferenceStream, + deleteProfilesStream, + ingestTraceSpans, + deleteTracesStream, + randomHex, + MINUTE_MS, +} = require('../utils/profiles-ingestion.js'); + +test.describe('Profiles navigation and entry points', () => { + test.describe.configure({ mode: 'parallel' }); + let api; + let ref; + let trace; + let pm; + + test.beforeAll(async ({ playwright }) => { + api = await playwright.request.newContext(); + ref = await seedReferenceStream(api, 'nav'); + const startMs = ref.t0Ms + 4 * MINUTE_MS - 500; + trace = { + stream: `e2e_prof_trace_${randomHex(4)}`, + startMs, + traceId: ref.link.traceId, + linkedSpanId: ref.link.spanId, + unlinkedSpanId: randomHex(8), + }; + await ingestTraceSpans(api, trace.stream, { + traceId: trace.traceId, + spanIds: [trace.linkedSpanId, trace.unlinkedSpanId], + startMs, + }); + }); + + test.afterAll(async () => { + await deleteProfilesStream(api, ref.stream); + await deleteTracesStream(api, trace.stream); + await api.dispose(); + }); + + test.beforeEach(async ({ page }, testInfo) => { + testLogger.testStart(testInfo.title, testInfo.file); + pm = new PageManager(page); + await pm.profilesPage.hideFlameZoomHint(); + }); + + test.afterEach(async ({}, testInfo) => { + await pm.profilesPage.teardown(); + testLogger.testEnd(testInfo.title, testInfo.status); + }); + + test('The left menu opens Profiles with its Beta tag', { tag: ['@all', '@profiles', '@P0'] }, async ({ page }) => { + await navigateToBase(page); + await pm.profilesPage.clickMenuProfiles(); + expect(new URL(page.url()).pathname).toBe('/web/profiles'); + await pm.profilesPage.expectPageHeader(); + }); + + test('Profiles sits right after Traces in the menu', { tag: ['@all', '@profiles', '@P2'] }, async ({ page }) => { + await navigateToBase(page); + await pm.profilesPage.expectMenuProfilesVisible(true); + const order = await pm.profilesPage.visibleMenuOrder(); + testLogger.info('Menu order', { order }); + expect(order.indexOf('menu-link-/profiles-item')).toBe(order.indexOf('menu-link-/traces-item') + 1); + }); + + test('With profiling off the menu hides Profiles and the route goes home', { tag: ['@all', '@profiles', '@P1'] }, async ({ page }) => { + await pm.profilesPage.mockServerConfig({ profiling: false }); + await navigateToBase(page); + await pm.profilesPage.expectMenuProfilesVisible(false); + await pm.profilesPage.routerPush({ name: 'profiles', query: { org_identifier: pm.profilesPage.orgName() } }); + await expect.poll(() => new URL(page.url()).pathname).toBe('/web/'); + expect(pm.profilesPage.currentQuery().org_identifier).toBe(pm.profilesPage.orgName()); + }); + + test('The Data page Profiles filter lists the stream', { tag: ['@all', '@profiles', '@P1'] }, async ({ page }) => { + await navigateToBase(page); + await pm.profilesPage.gotoDataPageProfiles(ref.stream); + }); + + test('The Data page explore button opens Logs search on the profiles stream', { tag: ['@all', '@profiles', '@P1'] }, async ({ page }) => { + await navigateToBase(page); + await pm.profilesPage.gotoDataPageProfiles(ref.stream); + await pm.profilesPage.clickDataPageExplore(ref.stream); + await expect.poll(() => new URL(page.url()).pathname).toBe('/web/logs'); + const query = pm.profilesPage.currentQuery(); + expect(query.stream_type).toBe('profiles'); + expect(query.stream).toBe(ref.stream); + expect(query.type).toBe('stream_explorer'); + await pm.profilesPage.expectLogsExplorerRows(); + }); + + test('Data Sources shows the profiles collector snippets', { tag: ['@all', '@profiles', '@P2'] }, async ({ page }) => { + await navigateToBase(page); + await pm.profilesPage.gotoProfilesIngestion(); + await pm.profilesPage.expectIngestionProfilesTabsActive(); + const otel = await pm.profilesPage.ingestionSnippetText('otel'); + expect(otel).toContain(`/api/${pm.profilesPage.orgName()}/v1/profiles`); + expect(otel).toContain('stream-name'); + expect(await pm.profilesPage.ingestionSnippetText('start')).toMatch(/otelcol/); + expect(await pm.profilesPage.ingestionSnippetText('grpc')).toContain('stream-name'); + }); + + test('With no profile streams the empty state links to ingestion setup', { tag: ['@all', '@profiles', '@P2'] }, async ({ page }) => { + await pm.profilesPage.mockStreamList([]); + await navigateToBase(page); + await pm.profilesPage.gotoProfiles(); + await pm.profilesPage.expectState('noStream', { title: 'No profile streams available' }); + await pm.profilesPage.clickStateAction('noStream'); + await expect.poll(() => new URL(page.url()).pathname).toMatch(/^\/web\/ingestion\/custom\/profiles/); + }); + + test('A linked trace span opens its profile', { tag: ['@all', '@profiles', '@P1'] }, async ({ page }) => { + await pm.profilesPage.mockStreamList([ref.stream]); + await navigateToBase(page); + await pm.profilesPage.gotoTraceDetails({ + traceId: trace.traceId, + stream: trace.stream, + fromUs: (trace.startMs - 5 * MINUTE_MS) * 1000, + toUs: (trace.startMs + 5 * MINUTE_MS) * 1000, + }); + await pm.profilesPage.clickTraceSpanAndAwaitProbe(ref.stream, trace.linkedSpanId); + await pm.profilesPage.expectViewProfileButton(true); + await pm.profilesPage.clickViewProfile(); + const query = pm.profilesPage.currentQuery(); + expect(query.stream).toBe(ref.stream); + expect(query.view).toBe('flame'); + expect(query.service_name).toBe('checkout'); + expect(query.profile_type).toBe('cpu'); + expect(Math.abs(Number(query.from) - (trace.startMs * 1000 - 60_000_000))).toBeLessThan(1_000_000); + expect(Math.abs(Number(query.to) - ((trace.startMs + 1000) * 1000 + 60_000_000))).toBeLessThan(1_000_000); + await pm.profilesPage.expectFilterChip('trace_id', trace.traceId); + await pm.profilesPage.expectFilterChip('span_id', trace.linkedSpanId); + await pm.profilesPage.expectFlameRootSummary('Total: 250 ms (CPU duration)'); + }); + + test('A span without a linked profile has no View Profile', { tag: ['@all', '@profiles', '@P2'] }, async ({ page }) => { + await pm.profilesPage.mockStreamList([ref.stream]); + await navigateToBase(page); + await pm.profilesPage.gotoTraceDetails({ + traceId: trace.traceId, + stream: trace.stream, + fromUs: (trace.startMs - 5 * MINUTE_MS) * 1000, + toUs: (trace.startMs + 5 * MINUTE_MS) * 1000, + }); + await pm.profilesPage.clickTraceSpanAndAwaitProbe(ref.stream, trace.unlinkedSpanId); + await pm.profilesPage.clickTraceSpanAndAwaitProbe(ref.stream, trace.linkedSpanId); + await pm.profilesPage.expectViewProfileButton(true); + // The unlinked span's probe finished before the linked one started, so its cached miss now renders synchronously. + await pm.profilesPage.openTraceSpan(trace.unlinkedSpanId); + await pm.profilesPage.expectTraceSidebarSpan(trace.unlinkedSpanId); + await pm.profilesPage.expectViewProfileButton(false); + }); +}); diff --git a/tests/ui-testing/playwright-tests/Profiles/profiles-query-controls.spec.js b/tests/ui-testing/playwright-tests/Profiles/profiles-query-controls.spec.js new file mode 100644 index 0000000000..b36fd8e112 --- /dev/null +++ b/tests/ui-testing/playwright-tests/Profiles/profiles-query-controls.spec.js @@ -0,0 +1,343 @@ +// Continuous Profiling header and query controls: run rules, Auto Run, cancel, label filters, date, URL and Share. + +const { test, expect, navigateToBase } = require('../utils/enhanced-baseFixtures.js'); +const testLogger = require('../utils/test-logger.js'); +const PageManager = require('../../pages/page-manager.js'); +const { CHECKOUT_FRAMES: F } = require('../../pages/profilesPages/profilesPage.js'); +const { + seedReferenceStream, + seedProfilesStream, + noLabelResources, + deleteProfilesStream, + MINUTE_MS, +} = require('../utils/profiles-ingestion.js'); + +test.describe('Profiles query controls', () => { + test.describe.configure({ mode: 'parallel' }); + let api; + let ref; + let bare; + let pm; + + test.beforeAll(async ({ playwright }) => { + api = await playwright.request.newContext(); + ref = await seedReferenceStream(api, 'query'); + bare = await seedProfilesStream(api, 'querybare', noLabelResources, { service: 'bare-svc', total: 1_000_000_000 }); + }); + + test.afterAll(async () => { + await deleteProfilesStream(api, ref.stream); + await deleteProfilesStream(api, bare.stream); + await api.dispose(); + }); + + test.beforeEach(async ({ page }, testInfo) => { + testLogger.testStart(testInfo.title, testInfo.file); + pm = new PageManager(page); + await pm.profilesPage.mockServerConfig(); + await pm.profilesPage.hideFlameZoomHint(); + await navigateToBase(page); + }); + + test.afterEach(async ({}, testInfo) => { + await pm.profilesPage.teardown(); + testLogger.testEnd(testInfo.title, testInfo.status); + }); + + test('A seeded page load runs the query exactly once', { tag: ['@all', '@profiles', '@P0'] }, async () => { + const calls = pm.profilesPage.trackRequests(); + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.expectTopRow(0, { name: 'db_query', self: '6 s' }); + expect(calls.series).toHaveLength(1); + expect(calls.merge).toHaveLength(1); + expect(calls.merge[0].body).toMatchObject({ + start_time: ref.window.from, + end_time: ref.window.to, + service_name: 'checkout', + profile_type: 'cpu', + profile_unit: 'nanoseconds', + filters: [], + max_nodes: 4096, + }); + }); + + test('Selectors show the seeded stream, service and type', { tag: ['@all', '@profiles', '@P0'] }, async () => { + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.expectSelected(pm.profilesPage.streamSelect, ref.stream); + await pm.profilesPage.expectSelected(pm.profilesPage.serviceSelect, 'checkout'); + await pm.profilesPage.expectSelected(pm.profilesPage.typeSelect, 'cpu (cores)'); + }); + + test('An unseeded open picks the first stream, CPU and Past 15 minutes and writes the URL', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.mockStreamList([ref.stream, bare.stream].sort()); + await pm.profilesPage.gotoProfiles(); + const first = [ref.stream, bare.stream].sort()[0]; + await pm.profilesPage.expectUrlParam('stream', first); + await pm.profilesPage.expectUrlParam('period', '15m'); + await pm.profilesPage.expectUrlParam('profile_type', 'cpu'); + await pm.profilesPage.expectUrlParam('view', 'top'); + await pm.profilesPage.expectSelected(pm.profilesPage.streamSelect, first); + }); + + test('The type list is scoped to the chosen service', { tag: ['@all', '@profiles', '@P0'] }, async () => { + await pm.profilesPage.gotoRef(ref); + expect(await pm.profilesPage.selectOptionLabels(pm.profilesPage.typeSelect)).toEqual(['cpu (cores)']); + const calls = pm.profilesPage.trackRequests(); + await pm.profilesPage.selectService('payments'); + await expect.poll(() => calls.meta.filter((call) => call.url.includes('service_name=payments')).length).toBeGreaterThan(0); + await expect + .poll(() => pm.profilesPage.selectOptionLabels(pm.profilesPage.typeSelect), { timeout: 15000 }) + .toEqual(['alloc_space (bytes)', 'cpu (cores)', 'inuse_space (bytes)']); + }); + + test('With Auto Run off a change marks Run as pending without querying', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref); + const calls = pm.profilesPage.trackRequests(); + await pm.profilesPage.selectService('payments'); + await pm.profilesPage.expectRunPending(true); + expect(calls.series).toHaveLength(0); + expect(calls.merge).toHaveLength(0); + expect(pm.profilesPage.currentQuery().service_name).toBe('checkout'); + }); + + test('Run clears the pending state and writes the new inputs to the URL', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.selectService('payments'); + await pm.profilesPage.expectRunPending(true); + const calls = pm.profilesPage.trackRequests(); + await pm.profilesPage.clickRun(); + await pm.profilesPage.expectRunPending(false); + expect(calls.series).toHaveLength(1); + expect(calls.merge).toHaveLength(1); + await pm.profilesPage.expectUrlParam('service_name', 'payments'); + await pm.profilesPage.expectTopRow(0, { name: 'charge_card', self: '2.7 s' }); + }); + + test('Run is disabled when the window has no profile type', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.mockApi('meta', { + json: { data_sources: [], services: ['checkout'], profile_types: [], label_names: [], took: 1 }, + }); + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.expectRunDisabled(); + await pm.profilesPage.hoverRunAndExpectTooltip('Pick a stream and a profile type with data in this time range'); + }); + + test('With Auto Run on a change runs at once, and the switch turns it off', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.mockServerConfig({ autoQuery: true }); + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.openRunOptions(); + await pm.profilesPage.expectAutoRunToggleText('Turn off Auto Run'); + await pm.profilesPage.pressEscape(); + const calls = pm.profilesPage.trackRequests(); + await pm.profilesPage.selectService('payments'); + await expect.poll(() => calls.merge.filter((call) => call.body.service_name === 'payments').length).toBe(1); + await pm.profilesPage.waitForRunIdle(); + expect(calls.series.filter((call) => call.body.service_name === 'payments')).toHaveLength(1); + await pm.profilesPage.expectRunPending(false); + await pm.profilesPage.expectUrlParam('service_name', 'payments'); + await pm.profilesPage.openRunOptions(); + await pm.profilesPage.clickAutoRunToggle(); + expect(await pm.profilesPage.autoRunStorageValue()).toBe('false'); + await pm.profilesPage.openRunOptions(); + await pm.profilesPage.expectAutoRunToggleText('Turn on Auto Run'); + await pm.profilesPage.pressEscape(); + const offCalls = pm.profilesPage.trackRequests(); + await pm.profilesPage.selectService('checkout'); + await pm.profilesPage.expectRunPending(true); + expect(offCalls.merge).toHaveLength(0); + }); + + test('Run options are hidden when the server has Auto Run off', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.expectRunOptionsHidden(); + }); + + test('Cancelling the first query shows Search cancelled and its Run query recovers', { tag: ['@all', '@profiles', '@P1'] }, async () => { + const release = await pm.profilesPage.holdApi('merge'); + await pm.profilesPage.openSeededNoWait({ stream: ref.stream, ...ref.window, service: 'checkout' }); + await pm.profilesPage.clickCancel(); + await pm.profilesPage.expectState('cancelled', { title: 'Search cancelled' }); + await pm.profilesPage.expectRunPending(true); + release(); + await pm.profilesPage.clickStateAction('cancelled'); + await pm.profilesPage.waitForRunIdle(); + await pm.profilesPage.expectTopRow(0, { name: 'db_query', self: '6 s' }); + await pm.profilesPage.expectStateHidden('cancelled'); + }); + + test('Cancelling a re-run keeps the results on screen', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.selectService('payments'); + const release = await pm.profilesPage.holdApi('merge'); + await pm.profilesPage.clickRunNoWait(); + await pm.profilesPage.clickCancel(); + release(); + await pm.profilesPage.expectTopRow(0, { name: 'db_query', self: '6 s' }); + await pm.profilesPage.expectRunPending(true); + expect(pm.profilesPage.currentQuery().service_name).toBe('checkout'); + }); + + test('Picking a filter value applies it and Run narrows the result', { tag: ['@all', '@profiles', '@P0'] }, async () => { + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.addFilter('service_version', 'v1.5.0'); + await pm.profilesPage.expectFilterChip('service_version', 'v1.5.0'); + await pm.profilesPage.expectRunPending(true); + await pm.profilesPage.clickRun(); + await pm.profilesPage.expectTopRow(0, { name: 'db_query', self: '4 s', selfPercent: '50.00%' }); + await pm.profilesPage.expectUrlParam('filters', 'service_version=v1.5.0'); + }); + + test('A second value of the same label replaces the first', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref, { filters: 'service_version=v1.5.0' }); + await pm.profilesPage.addFilter('service_version', 'v1.4.0'); + await pm.profilesPage.expectFilterChipCount(1); + await pm.profilesPage.expectFilterChip('service_version', 'v1.4.0'); + const calls = pm.profilesPage.trackRequests(); + await pm.profilesPage.clickRun(); + expect(calls.merge[0].body.filters).toEqual([{ key: 'service_version', op: '=', value: 'v1.4.0' }]); + await pm.profilesPage.expectTopRow(0, { name: 'db_query', self: '2 s', selfPercent: '50.00%' }); + }); + + test('A second label ANDs and its value list is narrowed by the first', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref, { filters: 'service_version=v1.4.0' }); + expect(await pm.profilesPage.filterValueOptionLabels('deployment_environment')).toEqual(['prod']); + await pm.profilesPage.addFilter('deployment_environment', 'prod'); + await pm.profilesPage.expectFilterChipCount(2); + await pm.profilesPage.clickRun(); + await pm.profilesPage.expectTopRow(0, { name: 'db_query', self: '2 s', selfPercent: '50.00%' }); + }); + + test('Editing a chip changes its value', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref, { filters: 'service_version=v1.4.0,deployment_environment=prod' }); + await pm.profilesPage.editFilter('service_version', 'v1.5.0'); + await pm.profilesPage.expectFilterChip('service_version', 'v1.5.0'); + await pm.profilesPage.clickRun(); + await pm.profilesPage.expectTopRow(0, { name: 'db_query', self: '4 s', selfPercent: '50.96%' }); + }); + + test('Removing every chip restores the unfiltered result', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref, { filters: 'service_version=v1.4.0,deployment_environment=prod' }); + await pm.profilesPage.removeFilter('service_version'); + await pm.profilesPage.removeFilter('deployment_environment'); + await pm.profilesPage.expectFilterChipCount(0); + await pm.profilesPage.clickRun(); + await pm.profilesPage.expectTopRow(0, { name: 'db_query', self: '6 s', selfPercent: '50.00%' }); + await pm.profilesPage.expectUrlParamAbsent('filters'); + }); + + test('The value select stays disabled until a label is picked', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.expectValueSelectDisabled(); + }); + + test('Changing stream clears the filters and the service', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.mockStreamList([ref.stream, bare.stream].sort()); + await pm.profilesPage.gotoRef(ref, { filters: 'service_version=v1.4.0' }); + await pm.profilesPage.expectFilterChipCount(1); + await pm.profilesPage.selectStream(bare.stream); + await pm.profilesPage.expectFilterChipCount(0); + await pm.profilesPage.expectSelected(pm.profilesPage.serviceSelect, 'bare-svc'); + await pm.profilesPage.expectRunPending(true); + }); + + test('Shifting the date reloads meta and marks Run as pending', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref); + const calls = pm.profilesPage.trackRequests(); + await pm.profilesPage.clickDatePrev(); + await expect.poll(() => calls.meta.length).toBeGreaterThan(0); + await pm.profilesPage.expectRunPending(true); + expect(calls.merge).toHaveLength(0); + expect(pm.profilesPage.currentQuery().from).toBe(String(ref.window.from)); + }); + + test('An empty window keeps the chosen service', { tag: ['@all', '@profiles', '@P1'] }, async () => { + const span = ref.window.to - ref.window.from; + await pm.profilesPage.gotoSeeded({ stream: ref.stream, from: ref.window.from - span, to: ref.window.to - span, service: 'checkout' }); + await pm.profilesPage.expectState('noResults', { title: 'No profiles found' }); + await pm.profilesPage.expectSelected(pm.profilesPage.serviceSelect, 'checkout'); + await pm.profilesPage.clickDateNext(); + await pm.profilesPage.expectRunPending(true); + await pm.profilesPage.clickRun(); + await pm.profilesPage.expectSelected(pm.profilesPage.serviceSelect, 'checkout'); + await pm.profilesPage.expectTopRow(0, { name: 'db_query', self: '6 s' }); + await pm.profilesPage.expectUrlParam('from', String(ref.window.from)); + }); + + test('The URL restores the view, search and filters on reload with one run', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.addFilter('service_version', 'v1.5.0'); + await pm.profilesPage.clickRun(); + await pm.profilesPage.switchView('tree'); + await pm.profilesPage.search('json'); + await pm.profilesPage.expectUrlParam('search', 'json'); + await pm.profilesPage.expectUrlParam('view', 'tree'); + const calls = pm.profilesPage.trackRequests(); + await pm.profilesPage.reload(); + await pm.profilesPage.expectViewActive('tree'); + await pm.profilesPage.expectSearchValue('json'); + await pm.profilesPage.expectFilterChip('service_version', 'v1.5.0'); + await pm.profilesPage.expectSelected(pm.profilesPage.serviceSelect, 'checkout'); + await pm.profilesPage.expectSelected(pm.profilesPage.typeSelect, 'cpu (cores)'); + expect(calls.merge).toHaveLength(1); + expect(calls.series).toHaveLength(1); + }); + + test('View, search and focus changes replace the URL without adding history', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref); + const before = await pm.profilesPage.historyLength(); + await pm.profilesPage.switchView('flame'); + await pm.profilesPage.search('json'); + await pm.profilesPage.expectUrlParam('search', 'json'); + await pm.profilesPage.search(''); + await pm.profilesPage.clickFrame(F.db_query, 'db_query'); + await pm.profilesPage.expectUrlParam('focus', F.db_query.path); + expect(await pm.profilesPage.historyLength()).toBe(before); + }); + + test('Garbage URL seeds fall back without breaking the page', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'bogus' }); + await pm.profilesPage.expectViewActive('top'); + await pm.profilesPage.expectUrlParam('view', 'top'); + await pm.profilesPage.gotoSeeded({ stream: ref.stream, from: ref.window.to, to: ref.window.from, service: 'checkout' }); + await pm.profilesPage.expectUrlParam('period', '15m'); + await pm.profilesPage.expectDateButtonText('Past 15 Minutes'); + await pm.profilesPage.mockStreamList([ref.stream]); + await pm.profilesPage.gotoSeeded({ stream: 'e2e_prof_nope', ...ref.window, service: 'checkout' }); + await pm.profilesPage.expectSelected(pm.profilesPage.streamSelect, ref.stream); + await pm.profilesPage.expectTopRow(0, { name: 'db_query', self: '6 s' }); + }); + + test('Share is disabled when there is no data', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoSeeded({ + stream: ref.stream, + from: ref.window.from - 180 * MINUTE_MS * 1000, + to: ref.window.from - 120 * MINUTE_MS * 1000, + service: 'checkout', + }); + await pm.profilesPage.expectState('noResults'); + await pm.profilesPage.expectShareDisabled(true); + }); + + test('Share copies a short link that restores the run with an absolute window', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoSeeded({ + stream: ref.stream, + period: '15m', + service: 'checkout', + view: 'tree', + filters: 'service_version=v1.5.0', + }); + await pm.profilesPage.expectShareDisabled(false); + const link = await pm.profilesPage.clickShareAndReadClipboard(); + expect(link).toMatch(/\/web\/short\/[0-9a-f]+\?org_identifier=/); + await pm.profilesPage.gotoUrl(link); + await pm.profilesPage.expectUrlParam('stream', ref.stream); + await pm.profilesPage.expectUrlParam('service_name', 'checkout'); + await pm.profilesPage.expectUrlParam('filters', 'service_version=v1.5.0'); + await pm.profilesPage.expectUrlParam('view', 'tree'); + const query = pm.profilesPage.currentQuery(); + expect(query.period).toBeUndefined(); + expect(Number(query.to) - Number(query.from)).toBe(15 * MINUTE_MS * 1000); + await pm.profilesPage.expectFilterChip('service_version', 'v1.5.0'); + await pm.profilesPage.expectViewActive('tree'); + }); +}); diff --git a/tests/ui-testing/playwright-tests/Profiles/profiles-states-edge.spec.js b/tests/ui-testing/playwright-tests/Profiles/profiles-states-edge.spec.js new file mode 100644 index 0000000000..864d0c30a3 --- /dev/null +++ b/tests/ui-testing/playwright-tests/Profiles/profiles-states-edge.spec.js @@ -0,0 +1,177 @@ +// Continuous Profiling page states (no results, errors, partial, loading), deep stacks, theme, mobile and edge data. + +const { test, expect, navigateToBase } = require('../utils/enhanced-baseFixtures.js'); +const testLogger = require('../utils/test-logger.js'); +const PageManager = require('../../pages/page-manager.js'); +const { CHECKOUT_FRAMES: F, CATALOG_FRAMES: C } = require('../../pages/profilesPages/profilesPage.js'); +const { + seedReferenceStream, + seedProfilesStream, + zeroValueResources, + unknownServiceResources, + deleteProfilesStream, + MINUTE_MS, +} = require('../utils/profiles-ingestion.js'); + +test.describe('Profiles states and edges', () => { + test.describe.configure({ mode: 'parallel' }); + let api; + let ref; + let zero; + let unknown; + let pm; + + test.beforeAll(async ({ playwright }) => { + api = await playwright.request.newContext(); + ref = await seedReferenceStream(api, 'states'); + zero = await seedProfilesStream(api, 'stzero', zeroValueResources, { service: 'zero-svc' }); + unknown = await seedProfilesStream(api, 'stunknown', unknownServiceResources, { + service: 'unknown_service:mybin', + total: 1_000_000_000, + }); + }); + + test.afterAll(async () => { + await deleteProfilesStream(api, ref.stream); + await deleteProfilesStream(api, zero.stream); + await deleteProfilesStream(api, unknown.stream); + await api.dispose(); + }); + + test.beforeEach(async ({ page }, testInfo) => { + testLogger.testStart(testInfo.title, testInfo.file); + pm = new PageManager(page); + await pm.profilesPage.mockServerConfig(); + await pm.profilesPage.hideFlameZoomHint(); + await navigateToBase(page); + }); + + test.afterEach(async ({}, testInfo) => { + await pm.profilesPage.teardown(); + testLogger.testEnd(testInfo.title, testInfo.status); + }); + + test('An empty window shows No profiles found and disables Share', { tag: ['@all', '@profiles', '@P0'] }, async () => { + await pm.profilesPage.gotoSeeded({ + stream: ref.stream, + from: ref.window.from - 180 * MINUTE_MS * 1000, + to: ref.window.from - 120 * MINUTE_MS * 1000, + service: 'checkout', + }); + await pm.profilesPage.expectState('noResults', { + title: 'No profiles found', + description: "Your time window doesn't contain any profile data from this stream.", + }); + await pm.profilesPage.expectShareDisabled(true); + }); + + test('A load error shows Retry and recovers once the bad filter is removed', { tag: ['@all', '@profiles', '@P1'] }, async () => { + const calls = pm.profilesPage.trackRequests(); + await pm.profilesPage.gotoRef(ref, { filters: 'a%3Bb=x' }); + await pm.profilesPage.expectState('error', { description: 'invalid filter key: a;b' }); + const before = calls.merge.length; + await pm.profilesPage.clickStateAction('error'); + await expect.poll(() => calls.merge.length).toBeGreaterThan(before); + await pm.profilesPage.waitForRunIdle(); + await pm.profilesPage.expectState('error', { description: 'invalid filter key: a;b' }); + await pm.profilesPage.removeFilter('a;b'); + await pm.profilesPage.clickRun(); + await pm.profilesPage.expectStateHidden('error'); + await pm.profilesPage.expectTopRow(0, { name: 'db_query', self: '6 s' }); + }); + + test('A failed re-run keeps the data and shows an error banner with Retry', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.mockApi('merge', { status: 500, json: { code: 500, message: 'x' } }); + await pm.profilesPage.clickRun(); + await pm.profilesPage.expectErrorBannerText('x'); + await pm.profilesPage.expectTopTableVisible(); + await pm.profilesPage.unrouteApi('merge'); + await pm.profilesPage.clickErrorBannerRetry(); + await pm.profilesPage.waitForRunIdle(); + await pm.profilesPage.expectStateHidden('errorBanner'); + await pm.profilesPage.expectTopRow(0, { name: 'db_query', self: '6 s' }); + }); + + test('A truncated merge says what share of samples is shown', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.patchApiResponse('merge', () => ({ truncated: true, total: 200, merged_total: 150 })); + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.expectState('partial', { + title: 'Showing 75.00% of samples. Less common call stacks are left out of this view.', + }); + }); + + test('Loading shows skeletons in every view', { tag: ['@all', '@profiles', '@P2'] }, async () => { + const release = await pm.profilesPage.holdApi('merge'); + await pm.profilesPage.openSeededNoWait({ stream: ref.stream, ...ref.window, service: 'checkout', view: 'flame' }); + await pm.profilesPage.expectSkeleton('series'); + await pm.profilesPage.expectSkeleton('flame'); + await pm.profilesPage.switchViewWhileLoading('sandwich'); + await pm.profilesPage.expectSkeleton('sandwich'); + await pm.profilesPage.switchViewWhileLoading('top'); + await pm.profilesPage.expectTopTableLoading(); + release(); + await pm.profilesPage.waitForRunIdle(); + await pm.profilesPage.expectTopRow(0, { name: 'db_query', self: '6 s' }); + }); + + test('A 1100-frame stack renders with its truncated root', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref, { service: 'catalog', view: 'flame' }); + await pm.profilesPage.hoverFrame(C.truncated, '[truncated]'); + await pm.profilesPage.expectFlameTooltipContains('[truncated]', ['1 s', '18.13%']); + await pm.profilesPage.switchView('top'); + await pm.profilesPage.expectTopRow(0, { name: 'deep_0000', self: '1 s' }); + }); + + test('Rows whose values are all zero show no results', { tag: ['@all', '@profiles', '@P3'] }, async () => { + await pm.profilesPage.gotoRef(zero, { service: 'zero-svc' }); + await pm.profilesPage.expectState('noResults', { title: 'No profiles found' }); + }); + + test('Switching theme recolours a live flame graph', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'flame' }); + await pm.profilesPage.hoverFrame(F.db_query, 'db_query'); + const before = await pm.profilesPage.framePixel(F.db_query); + await pm.themePage.toggleThemeMode(); + await expect.poll(async () => JSON.stringify(await pm.profilesPage.framePixel(F.db_query))).not.toBe(JSON.stringify(before)); + await pm.profilesPage.expectViewActive('flame'); + }); + + test('In dark mode the chart has no background of its own', { tag: ['@all', '@profiles', '@P3'] }, async () => { + await pm.profilesPage.gotoRef(ref); + await pm.themePage.toggleThemeMode(); + await pm.themePage.waitForDarkModeApplied(); + await expect + .poll(async () => { + const { chart, card } = await pm.profilesPage.chartAndCardBackground(); + return Math.max(Math.abs(chart.r - card.r), Math.abs(chart.g - card.g), Math.abs(chart.b - card.b)); + }) + .toBeLessThan(4); + }); + + test('At phone width the views move into a menu and the page does not scroll sideways', { tag: ['@all', '@profiles', '@P3'] }, async () => { + await pm.profilesPage.setViewport(390, 844); + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.openMobileViewMenu(); + expect(await pm.profilesPage.mobileViewMenuItems()).toEqual([ + 'profiles-view-top-item', + 'profiles-view-flame-item', + 'profiles-view-tree-item', + 'profiles-view-sandwich-item', + 'profiles-view-compare-item', + ]); + await pm.profilesPage.pressEscape(); + expect(await pm.profilesPage.hasHorizontalPageScroll()).toBe(false); + await pm.profilesPage.rowMenuAction('top', 0, 'details'); + await pm.profilesPage.expectDetails({ name: 'db_query', self: '6 s · 50.00%' }); + expect(await pm.profilesPage.detailsSitsBelowTable()).toBe(true); + await pm.profilesPage.pickMobileView('tree'); + await pm.profilesPage.expectUrlParam('view', 'tree'); + }); + + test('A profile without service.name is listed as unknown_service', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(unknown, { service: 'unknown_service:mybin' }); + expect(await pm.profilesPage.selectOptionLabels(pm.profilesPage.serviceSelect)).toEqual(['unknown_service:mybin']); + await pm.profilesPage.expectTopRow(0, { name: 'bin_work', self: '1 s', selfPercent: '100.00%' }); + }); +}); diff --git a/tests/ui-testing/playwright-tests/Profiles/profiles-views.spec.js b/tests/ui-testing/playwright-tests/Profiles/profiles-views.spec.js new file mode 100644 index 0000000000..14102f77cd --- /dev/null +++ b/tests/ui-testing/playwright-tests/Profiles/profiles-views.spec.js @@ -0,0 +1,229 @@ +// Continuous Profiling result views: Top Table, Call Tree, Sandwich, Function details. + +const { test, expect, navigateToBase } = require('../utils/enhanced-baseFixtures.js'); +const testLogger = require('../utils/test-logger.js'); +const PageManager = require('../../pages/page-manager.js'); +const { CHECKOUT_FRAMES } = require('../../pages/profilesPages/profilesPage.js'); +const { seedReferenceStream, deleteProfilesStream, XSS_FUNCTION } = require('../utils/profiles-ingestion.js'); + +test.describe('Profiles views', () => { + test.describe.configure({ mode: 'parallel' }); + let api; + let ref; + let pm; + + test.beforeAll(async ({ playwright }) => { + api = await playwright.request.newContext(); + ref = await seedReferenceStream(api, 'views'); + }); + + test.afterAll(async () => { + await deleteProfilesStream(api, ref.stream); + await api.dispose(); + }); + + test.beforeEach(async ({ page }, testInfo) => { + testLogger.testStart(testInfo.title, testInfo.file); + pm = new PageManager(page); + await pm.profilesPage.mockServerConfig(); + await pm.profilesPage.hideFlameZoomHint(); + await navigateToBase(page); + }); + + test.afterEach(async ({}, testInfo) => { + await pm.profilesPage.teardown(); + testLogger.testEnd(testInfo.title, testInfo.status); + }); + + test('Top Table shows exact self and total per function', { tag: ['@all', '@profiles', '@P0'] }, async () => { + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.expectTopRowNames([ + 'db_query', + 'json_encode', + 'handle_order', + 'process_payment', + 'gc_sweep', + XSS_FUNCTION, + 'main', + 'runtime_main', + ]); + await pm.profilesPage.expectTopRow(0, { self: '6 s', selfPercent: '50.00%', total: '6 s', totalPercent: '50.00%' }); + await pm.profilesPage.expectTopRow(1, { self: '4 s', selfPercent: '33.33%', total: '4 s' }); + await pm.profilesPage.expectTopRow(2, { self: '1.5 s', selfPercent: '12.50%', total: '11.85 s', totalPercent: '98.75%' }); + await pm.profilesPage.expectTopRow(3, { self: '250 ms', selfPercent: '2.08%' }); + await pm.profilesPage.expectTopRow(4, { self: '150 ms', selfPercent: '1.25%' }); + await pm.profilesPage.expectTopRow(5, { self: '100 ms', selfPercent: '0.83%' }); + await pm.profilesPage.expectTopRow(6, { self: '0 ns', selfPercent: '0.00%', total: '11.85 s', totalPercent: '98.75%' }); + await pm.profilesPage.expectTopRow(7, { self: '0 ns', total: '150 ms', totalPercent: '1.25%' }); + }); + + test('Sorting by Total reorders the Top Table both ways', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.sortTopBy('total'); + await pm.profilesPage.expectTopSort('total', 'descending'); + await pm.profilesPage.expectTopRow(2, { name: 'db_query', total: '6 s' }); + await pm.profilesPage.expectTopRow(7, { name: XSS_FUNCTION, total: '100 ms' }); + await pm.profilesPage.expectTopRow(0, { total: '11.85 s' }); + await pm.profilesPage.expectTopRow(1, { total: '11.85 s' }); + await pm.profilesPage.sortTopBy('total'); + await pm.profilesPage.expectTopSort('total', 'ascending'); + await pm.profilesPage.expectTopRow(0, { name: XSS_FUNCTION, total: '100 ms' }); + await pm.profilesPage.expectTopRow(5, { name: 'db_query', total: '6 s' }); + }); + + test('Search filters the Top Table, highlights the match and summarises the share', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.search('json'); + await pm.profilesPage.expectTopRowNames(['json_encode']); + await pm.profilesPage.expectSearchSummary('1 function · 33.33% of samples'); + await pm.profilesPage.expectMarkedText('json'); + await pm.profilesPage.expectUrlParam('search', 'json'); + }); + + test('Search with no match says so and empties the table', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.search('zzz'); + await pm.profilesPage.expectSearchSummary('No matches'); + await pm.profilesPage.expectTopRowCount(0); + }); + + test('Row menu Copy puts the function name on the clipboard', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.rowMenuAction('top', 0, 'copy'); + await pm.profilesPage.expectCopiedToast(); + expect(await pm.profilesPage.readClipboard()).toBe('db_query'); + }); + + test('Row menu Search fills the search box with the function', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.rowMenuAction('top', 1, 'search'); + await pm.profilesPage.expectSearchValue('json_encode'); + await pm.profilesPage.expectTopRowNames(['json_encode']); + }); + + test('Row menu Show in Flame Graph switches view and highlights the function', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.rowMenuAction('top', 2, 'show'); + await pm.profilesPage.expectViewActive('flame'); + await pm.profilesPage.expectSearchValue('handle_order'); + await pm.profilesPage.expectUrlParam('view', 'flame'); + await pm.profilesPage.expectUrlParam('search', 'handle_order'); + }); + + test('Function details shows exact self, total, callers and callees', { tag: ['@all', '@profiles', '@P0'] }, async () => { + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.rowMenuAction('top', 2, 'details'); + await pm.profilesPage.expectDetails({ + name: 'handle_order', + self: '1.5 s · 12.50%', + total: '11.85 s · 98.75%', + calledBy: ['main'], + calls: ['db_query', 'json_encode', 'process_payment', XSS_FUNCTION], + hasPackage: false, + }); + expect(await pm.profilesPage.detailsListRowCount('calls')).toBe(4); + expect(await pm.profilesPage.detailsListRowCount('calledBy')).toBe(1); + await pm.profilesPage.expectUrlParam('details', 'handle_order'); + }); + + test('Function details actions copy, open sandwich, show in flame and close', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref, { extra: { details: 'handle_order' } }); + await pm.profilesPage.expectDetails({ name: 'handle_order', self: '1.5 s · 12.50%' }); + await pm.profilesPage.clickDetails('copy'); + await pm.profilesPage.expectCopiedToast(); + expect(await pm.profilesPage.readClipboard()).toBe('handle_order'); + await pm.profilesPage.clickDetails('sandwich'); + await pm.profilesPage.expectViewActive('sandwich'); + await pm.profilesPage.expectSandwichFunction('handle_order'); + await pm.profilesPage.clickDetails('show'); + await pm.profilesPage.expectViewActive('flame'); + await pm.profilesPage.expectSearchValue('handle_order'); + await pm.profilesPage.clickDetails('close'); + await pm.profilesPage.expectDetailsHidden(); + await pm.profilesPage.expectUrlParamAbsent('details'); + }); + + test('Function details is hidden in Compare', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref, { extra: { details: 'db_query' } }); + await pm.profilesPage.expectDetails({ name: 'db_query', self: '6 s · 50.00%' }); + await pm.profilesPage.switchView('compare'); + await pm.profilesPage.waitForCompareIdle(); + await pm.profilesPage.expectDetailsHidden(); + }); + + test('Call Tree shows child and node counts and expands and collapses', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'tree' }); + await pm.profilesPage.expectTreeRow(0, { name: 'total', counts: '2 children · 8 nodes', total: '12 s' }); + await pm.profilesPage.expectTreeRow(1, { name: 'main', counts: '1 child · 5 nodes', total: '11.85 s' }); + await pm.profilesPage.expectTreeRow(2, { name: 'runtime_main', counts: '1 child · 1 node', total: '150 ms' }); + await pm.profilesPage.expectTreeRowCount(3); + await pm.profilesPage.toggleTreeRow(1); + await pm.profilesPage.expectTreeRow(2, { name: 'handle_order', counts: '4 children · 4 nodes', self: '1.5 s' }); + await pm.profilesPage.toggleTreeRow(2); + await pm.profilesPage.expectTreeRow(3, { name: 'db_query', counts: null, self: '6 s' }); + await pm.profilesPage.expectTreeRowCount(8); + await pm.profilesPage.toggleTreeRow(1); + await pm.profilesPage.expectTreeRowExpanded(1, false); + await pm.profilesPage.expectTreeRowCount(3); + }); + + test('Call Tree Focus in Flame Graph focuses that exact frame', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'tree' }); + await pm.profilesPage.toggleTreeRow(1); + await pm.profilesPage.toggleTreeRow(2); + await pm.profilesPage.expectTreeRow(3, { name: 'db_query' }); + await pm.profilesPage.rowMenuAction('tree', 3, 'show'); + await pm.profilesPage.expectViewActive('flame'); + await pm.profilesPage.expectUrlParam('focus', CHECKOUT_FRAMES.db_query.path); + }); + + test('Sandwich defaults to the top self function and follows the picked one', { tag: ['@all', '@profiles', '@P1'] }, async () => { + await pm.profilesPage.gotoRef(ref, { view: 'sandwich' }); + await pm.profilesPage.expectSandwichFunction('db_query'); + await pm.profilesPage.expectRootAtBottom(true); + await pm.profilesPage.pickSandwichFunction('handle_order'); + await pm.profilesPage.expectUrlParam('fn', 'handle_order'); + await pm.profilesPage.expectSandwichCoverage(false); + }); + + test('Sandwich notes partial coverage when the call tree is cut', { tag: ['@all', '@profiles', '@P3'] }, async () => { + await pm.profilesPage.forceMergeMaxNodes(50); + await pm.profilesPage.gotoRef(ref, { service: 'catalog', view: 'sandwich' }); + await pm.profilesPage.expectSandwichFunction('deep_0000'); + await pm.profilesPage.expectSandwichCoverageText("The call tree holds 0.00% of this function's samples"); + }); + + test('A script-like function name renders as text everywhere', { tag: ['@all', '@profiles', '@P0'] }, async () => { + await pm.profilesPage.gotoRef(ref); + await pm.profilesPage.expectTopRow(5, { name: XSS_FUNCTION }); + await pm.profilesPage.search('onerror'); + await pm.profilesPage.expectTopRowNames([XSS_FUNCTION]); + await pm.profilesPage.expectMarkedText('onerror'); + await pm.profilesPage.rowMenuAction('top', 0, 'details'); + await pm.profilesPage.expectDetails({ name: XSS_FUNCTION, calledBy: ['handle_order'] }); + await pm.profilesPage.search(''); + await pm.profilesPage.switchView('flame'); + await pm.profilesPage.hoverFrame({ depth: 3, left: 85.417, width: 0.833 }, XSS_FUNCTION); + await pm.profilesPage.switchView('tree'); + await pm.profilesPage.toggleTreeRow(1); + await pm.profilesPage.toggleTreeRow(2); + await pm.profilesPage.expectTreeRow(6, { name: XSS_FUNCTION }); + expect(await pm.profilesPage.isXssExecuted()).toBe(false); + }); + + test('Stale details and sandwich function seeds fall back safely', { tag: ['@all', '@profiles', '@P3'] }, async () => { + await pm.profilesPage.gotoRef(ref, { extra: { details: 'no_such_fn' } }); + await pm.profilesPage.expectTopRow(0, { name: 'db_query' }); + await pm.profilesPage.expectDetailsHidden(); + await pm.profilesPage.gotoRef(ref, { view: 'sandwich', extra: { fn: 'no_such_fn' } }); + await pm.profilesPage.expectSandwichFunction('db_query'); + }); + + test('Memory profiles use binary byte units', { tag: ['@all', '@profiles', '@P2'] }, async () => { + await pm.profilesPage.gotoRef(ref, { service: 'payments', type: 'alloc_space', unit: 'bytes' }); + await pm.profilesPage.expectTopRow(0, { name: 'alloc_buffer', self: '1 MiB', selfPercent: '66.67%' }); + await pm.profilesPage.expectTopRow(1, { name: 'alloc_json', self: '512 KiB', selfPercent: '33.33%' }); + await pm.profilesPage.switchView('flame'); + await pm.profilesPage.expectFlameRootSummary('Total: 1.5 MiB (bytes)'); + }); +}); diff --git a/tests/ui-testing/playwright-tests/cleanup.spec.js b/tests/ui-testing/playwright-tests/cleanup.spec.js index 09e7259912..d3bd7ffce7 100644 --- a/tests/ui-testing/playwright-tests/cleanup.spec.js +++ b/tests/ui-testing/playwright-tests/cleanup.spec.js @@ -409,11 +409,15 @@ test.describe("Pre-Test Cleanup", () => { [ /^sdr_traces_/, // sdr_traces_* (traces SDR test streams: redact/drop/hash × ingestion/query) /^trace_genai_parts_/, // trace_genai_parts_* (GenAI v5 parts rendering test streams) + /^e2e_prof_trace_/, // Profiles/profiles-navigation-entry.spec.js linked-span traces stream ], ['default'], // protect the default traces stream { streamType: 'traces' } ); + // Profiles specs delete their e2e_prof_* streams in afterAll; this sweeps what a crashed run left behind. + await pm.apiCleanup.cleanupStreams([/^e2e_prof_/], ['default'], { streamType: 'profiles' }); + // Note: Pipeline regression test streams (e2e_test_cpu_usage for metrics, e2e_test_traces for traces) // are created by pipeline-regression.spec.js for Issue #9901 regression tests. // Custom traces stream uses "stream-name" header (ZO_GRPC_STREAM_HEADER_KEY config). @@ -458,11 +462,6 @@ test.describe("Pre-Test Cleanup", () => { // This avoids conflicts both within a test run (parallelIndex) and across test runs (testRunId) // The cleanup patterns above will clean up old test streams via regex matching - // Clean up test semantic groups created by correlation settings tests - // Groups: k8s-cluster, k8s-namespace, k8s-deployment, service - // These are created by ensureSemanticGroupsExist() in correlationSettingsPage.js - await pm.apiCleanup.cleanupCorrelationSettings(); - // Clean up service graph test screenshots const path = require('path'); const screenshotDir = path.join(__dirname, '..', 'test-results', 'screenshots'); diff --git a/tests/ui-testing/playwright-tests/utils/profiles-ingestion.js b/tests/ui-testing/playwright-tests/utils/profiles-ingestion.js new file mode 100644 index 0000000000..4a0385a785 --- /dev/null +++ b/tests/ui-testing/playwright-tests/utils/profiles-ingestion.js @@ -0,0 +1,446 @@ +// Profiles OTLP/JSON ingestion, readiness gates and a reference dataset whose totals are computed by hand. + +const crypto = require('crypto'); +const { expect } = require('@playwright/test'); +const testLogger = require('./test-logger.js'); +const { getAuthHeaders, getOrgIdentifier } = require('./cloud-auth.js'); +const { ingestBase } = require('./ingest-url.js'); + +const SECOND_NS = 1_000_000_000; +const MINUTE_MS = 60_000; +const READY_TIMEOUT_MS = 60_000; +const READY_INTERVALS = [500, 1_000, 2_000]; + +const XSS_FUNCTION = ''; + +function baseUrl() { + return (process.env.ZO_BASE_URL || '').replace(/\/+$/, ''); +} + +function randomHex(bytes) { + return crypto.randomBytes(bytes).toString('hex'); +} + +/** Unique per-run stream name, e.g. `e2e_prof_views_1a2b3c4d`. */ +function uniqueProfilesStream(spec) { + return `e2e_prof_${spec}_${randomHex(4)}`; +} + +/** Start of the current minute − 10 min + 5 s, in ms: every profile sits alone in a one-minute bucket. */ +function referenceT0Ms(now = Date.now()) { + return Math.floor(now / MINUTE_MS) * MINUTE_MS - 10 * MINUTE_MS + 5_000; +} + +/** Absolute window around the reference data in microseconds (T0 − 1 min … T0 + 9 min). */ +function referenceWindowUs(t0Ms) { + return { from: (t0Ms - MINUTE_MS) * 1000, to: (t0Ms + 9 * MINUTE_MS) * 1000 }; +} + +function resourceAttrs({ service, version, env, instance, host, exe, extra = {} }) { + const attrs = []; + const add = (key, value) => { + if (value !== undefined && value !== null) attrs.push({ key, value: { stringValue: String(value) } }); + }; + add('service.name', service); + add('service.version', version); + add('deployment.environment', env); + add('service.instance.id', instance); + add('host.name', host); + add('process.executable.name', exe); + for (const [key, value] of Object.entries(extra)) add(key, value); + return attrs; +} + +/** One OTLP/JSON export with a shared dictionary; a sample is { stack: [leaf … root], value, link? }. */ +function buildProfilesRequest(resources) { + const strings = ['']; + const stringIndex = new Map([['', 0]]); + const functions = [{}]; + const functionIndex = new Map(); + const locations = [{}]; + const stacks = [{}]; + const stackIndex = new Map(); + const links = [{}]; + const linkIndex = new Map(); + + const str = (value) => { + if (!stringIndex.has(value)) { + stringIndex.set(value, strings.length); + strings.push(value); + } + return stringIndex.get(value); + }; + const fn = (name) => { + if (!functionIndex.has(name)) { + functionIndex.set(name, functions.length); + functions.push({ nameStrindex: str(name) }); + locations.push({ lines: [{ functionIndex: functions.length - 1 }] }); + } + return functionIndex.get(name); + }; + const stack = (frames) => { + const key = frames.join('\u0000'); + if (!stackIndex.has(key)) { + stackIndex.set(key, stacks.length); + stacks.push({ locationIndices: frames.map((frame) => fn(frame)) }); + } + return stackIndex.get(key); + }; + const link = ({ traceId, spanId }) => { + const key = `${traceId}/${spanId}`; + if (!linkIndex.has(key)) { + linkIndex.set(key, links.length); + links.push({ traceId, spanId }); + } + return linkIndex.get(key); + }; + + const resourceProfiles = resources.map((resource) => ({ + resource: { attributes: resourceAttrs(resource.attrs) }, + scopeProfiles: [ + { + scope: { name: 'e2e-profiles', version: '1.0.0' }, + profiles: resource.profiles.map((profile) => ({ + sampleType: { typeStrindex: str(profile.type), unitStrindex: str(profile.unit) }, + timeUnixNano: String(BigInt(resource.timeMs) * 1_000_000n), + durationNano: String(10 * SECOND_NS), + samples: profile.samples.map((sample) => ({ + stackIndex: stack(sample.stack), + values: [String(sample.value)], + ...(sample.link ? { linkIndex: link(sample.link) } : {}), + })), + })), + }, + ], + })); + + return { + dictionary: { + stringTable: strings, + functionTable: functions, + locationTable: locations, + stackTable: stacks, + linkTable: links, + }, + resourceProfiles, + }; +} + +const cpu = (samples) => ({ type: 'cpu', unit: 'nanoseconds', samples }); +const s = (frames, value, link) => ({ stack: frames.split('<-'), value, ...(link ? { link } : {}) }); + +/** Reference dataset: checkout totals 12 s (v1.4.0 4 s, v1.5.0 8 s), payments 3.5 s cpu, catalog 5.515 s; `link` tags checkout's process_payment. */ +function referenceResources(t0Ms, link) { + const at = (minutes) => t0Ms + minutes * MINUTE_MS; + const wide = Array.from({ length: 300 }, (_, i) => + s(`wide_fn_${String(i).padStart(3, '0')}<-wide_root<-catalog_main`, (i + 1) * 100_000), + ); + const deep = { + stack: Array.from({ length: 1100 }, (_, i) => `deep_${String(i).padStart(4, '0')}`), + value: SECOND_NS, + }; + return [ + { + attrs: { service: 'checkout', version: 'v1.4.0', env: 'prod', instance: 'checkout-1', host: 'host-a' }, + timeMs: at(0), + profiles: [ + cpu([ + s('db_query<-handle_order<-main', 2 * SECOND_NS), + s('json_encode<-handle_order<-main', SECOND_NS), + s('handle_order<-main', SECOND_NS), + ]), + ], + }, + { + attrs: { service: 'payments', version: 'v1.4.0', env: 'prod', instance: 'payments-1', host: 'host-d' }, + timeMs: at(2), + profiles: [ + cpu([s('charge_card<-process<-main', 1_200_000_000), s('validate<-process<-main', 800_000_000)]), + { + type: 'alloc_space', + unit: 'bytes', + samples: [s('alloc_buffer<-process<-main', 1_048_576), s('alloc_json<-process<-main', 524_288)], + }, + { type: 'inuse_space', unit: 'bytes', samples: [s('cache_put<-process<-main', 2_097_152)] }, + ], + }, + { + attrs: { service: 'checkout', version: 'v1.5.0', env: 'prod', instance: 'checkout-2', host: 'host-b' }, + timeMs: at(4), + profiles: [ + cpu([ + s('db_query<-handle_order<-main', 4 * SECOND_NS), + s('json_encode<-handle_order<-main', 3 * SECOND_NS), + s('handle_order<-main', 500_000_000), + s(`${XSS_FUNCTION}<-handle_order<-main`, 100_000_000), + s('process_payment<-handle_order<-main', 250_000_000, link), + ]), + ], + }, + { + attrs: { service: 'payments', version: 'v1.5.0', env: 'prod', instance: 'payments-2', host: 'host-e' }, + timeMs: at(5), + profiles: [ + cpu([s('charge_card<-process<-main', 1_500_000_000)]), + { type: 'inuse_space', unit: 'bytes', samples: [s('cache_put<-process<-main', 4_194_304)] }, + ], + }, + { + attrs: { service: 'checkout', version: 'v1.5.0', env: 'stage', instance: 'checkout-3', host: 'host-c' }, + timeMs: at(6), + profiles: [cpu([s('gc_sweep<-runtime_main', 150_000_000)])], + }, + { + attrs: { service: 'catalog', version: 'v1.5.0', env: 'stage', instance: 'catalog-1', host: 'host-f' }, + timeMs: at(7), + profiles: [cpu([...wide, deep])], + }, + ]; +} + +/** Twelve checkout versions (v1.1.0 … v1.12.0), one cpu profile each, for Breakdown "Others". */ +function manyVersionsResources(t0Ms) { + return Array.from({ length: 12 }, (_, i) => ({ + attrs: { service: 'checkout', version: `v1.${i + 1}.0`, env: 'prod', instance: `mv-${i}`, host: 'host-mv' }, + timeMs: t0Ms + (i % 9) * MINUTE_MS, + profiles: [cpu([s('work<-main', (i + 1) * 100_000_000)])], + })); +} + +/** One profile with no service.name (falls back to unknown_service:mybin). */ +function unknownServiceResources(t0Ms) { + return [{ attrs: { exe: 'mybin' }, timeMs: t0Ms, profiles: [cpu([s('bin_work<-bin_main', SECOND_NS)])] }]; +} + +/** Rows whose every value is 0: the stream has a service but no samples to draw. */ +function zeroValueResources(t0Ms) { + return [{ attrs: { service: 'zero-svc' }, timeMs: t0Ms, profiles: [cpu([s('idle<-main', 0)])] }]; +} + +/** Rows with only service.name, so the stream has no label to compare by. */ +function noLabelResources(t0Ms) { + return [{ attrs: { service: 'bare-svc' }, timeMs: t0Ms, profiles: [cpu([s('bare_work<-main', SECOND_NS)])] }]; +} + +async function postProfiles(request, stream, body) { + const org = getOrgIdentifier(); + const url = `${ingestBase(baseUrl())}/api/${org}/v1/profiles`; + const response = await request.post(url, { + headers: { ...getAuthHeaders(), 'Content-Type': 'application/json', 'stream-name': stream }, + data: body, + }); + const text = await response.text(); + if (response.status() !== 200) { + throw new Error(`Profiles ingest to ${stream} failed: ${response.status()} ${text}`); + } + if (text.includes('partialSuccess')) { + throw new Error(`Profiles ingest to ${stream} rejected some profiles: ${text}`); + } + testLogger.info('Ingested profiles', { stream, resources: body.resourceProfiles.length }); +} + +async function ingestProfiles(request, stream, resources) { + await postProfiles(request, stream, buildProfilesRequest(resources)); +} + +/** Ingests the reference dataset into a fresh stream, gates on its exact totals, returns { stream, t0Ms, window, link }. */ +async function seedReferenceStream(request, spec) { + const stream = uniqueProfilesStream(spec); + const t0Ms = referenceT0Ms(); + const link = { traceId: randomHex(16), spanId: randomHex(8) }; + await ingestProfiles(request, stream, referenceResources(t0Ms, link)); + const window = referenceWindowUs(t0Ms); + await waitForProfilesTotal(request, stream, { + window, + service: 'checkout', + type: 'cpu', + unit: 'nanoseconds', + total: 12_000_000_000, + }); + await waitForProfilesTotal(request, stream, { + window, + service: 'catalog', + type: 'cpu', + unit: 'nanoseconds', + total: 5_515_000_000, + }); + await waitForProfilesTotal(request, stream, { + window, + service: 'payments', + type: 'inuse_space', + unit: 'bytes', + total: 6_291_456, + }); + return { stream, t0Ms, window, link }; +} + +/** Ingests `build(t0Ms)` into a fresh stream and gates on `service` (and its exact cpu `total` when given). */ +async function seedProfilesStream(request, spec, build, { service, total }) { + const stream = uniqueProfilesStream(spec); + const t0Ms = referenceT0Ms(); + await ingestProfiles(request, stream, build(t0Ms)); + const window = referenceWindowUs(t0Ms); + await waitForProfilesService(request, stream, { window, service }); + if (total !== undefined) { + await waitForProfilesTotal(request, stream, { window, service, type: 'cpu', unit: 'nanoseconds', total }); + } + return { stream, t0Ms, window }; +} + +async function profilesApi(request, method, stream, path, body) { + const org = getOrgIdentifier(); + const url = `${baseUrl()}/api/${org}/${stream}/profiles/${path}`; + const options = { headers: { ...getAuthHeaders(), 'Content-Type': 'application/json' } }; + const response = + method === 'post' ? await request.post(url, { ...options, data: body }) : await request.get(url, options); + const text = await response.text(); + return { status: response.status(), body: text ? JSON.parse(text) : null, text }; +} + +/** Polls /profiles/merge until `total` matches exactly; fails loudly with the last response. */ +async function waitForProfilesTotal(request, stream, { window, service, type, unit, filters = [], total }) { + const body = { + start_time: window.from, + end_time: window.to, + ...(service ? { service_name: service } : {}), + profile_type: type, + profile_unit: unit, + filters, + }; + let last = null; + await expect + .poll( + async () => { + last = await profilesApi(request, 'post', stream, 'merge', body); + return last.status === 200 ? last.body?.total : `HTTP ${last.status} ${last.text.slice(0, 300)}`; + }, + { timeout: READY_TIMEOUT_MS, intervals: READY_INTERVALS, message: `profiles merge total on ${stream}` }, + ) + .toBe(total); + return last.body; +} + +/** Polls /profiles/meta until `service` is listed in the window. */ +async function waitForProfilesService(request, stream, { window, service }) { + let last = null; + await expect + .poll( + async () => { + last = await profilesApi(request, 'get', stream, `meta?start_time=${window.from}&end_time=${window.to}`); + return last.status === 200 ? last.body?.services ?? [] : [`HTTP ${last.status}`]; + }, + { timeout: READY_TIMEOUT_MS, intervals: READY_INTERVALS, message: `profiles meta services on ${stream}` }, + ) + .toContain(service); + return last.body; +} + +/** Deletes a profiles stream; logs instead of throwing so teardown never masks a test failure. */ +async function deleteProfilesStream(request, stream) { + const org = getOrgIdentifier(); + try { + const response = await request.delete( + `${baseUrl()}/api/${org}/streams/${stream}?type=profiles&delete_all=true`, + { headers: getAuthHeaders() }, + ); + testLogger.info('Deleted profiles stream', { stream, status: response.status() }); + } catch (error) { + testLogger.warn('Profiles stream delete failed', { stream, error: error.message }); + } +} + +/** Ingests one trace with a sibling span per id in `spanIds` (all same start/duration), into its own traces stream. */ +async function ingestTraceSpans(request, stream, { traceId, spanIds, startMs, durationMs = 1_000, service = 'checkout' }) { + const org = getOrgIdentifier(); + const start = BigInt(startMs) * 1_000_000n; + const end = start + BigInt(durationMs) * 1_000_000n; + const body = { + resourceSpans: [ + { + resource: { attributes: [{ key: 'service.name', value: { stringValue: service } }] }, + scopeSpans: [ + { + scope: { name: 'e2e-profiles', version: '1.0.0' }, + spans: spanIds.map((spanId) => ({ + traceId, + spanId, + name: `op_${spanId}`, + kind: 2, + startTimeUnixNano: String(start), + endTimeUnixNano: String(end), + attributes: [], + status: { code: 1 }, + })), + }, + ], + }, + ], + }; + const response = await request.post(`${ingestBase(baseUrl())}/api/${org}/v1/traces`, { + headers: { ...getAuthHeaders(), 'Content-Type': 'application/json', 'stream-name': stream }, + data: body, + }); + if (response.status() !== 200) { + throw new Error(`Trace ingest to ${stream} failed: ${response.status()} ${await response.text()}`); + } + await waitForTraceSearchable(request, stream, { traceId, startMs, endMs: startMs + durationMs }); +} + +async function waitForTraceSearchable(request, stream, { traceId, startMs, endMs }) { + const org = getOrgIdentifier(); + await expect + .poll( + async () => { + const response = await request.post(`${baseUrl()}/api/${org}/_search?type=traces`, { + headers: { ...getAuthHeaders(), 'Content-Type': 'application/json' }, + data: { + query: { + sql: `SELECT trace_id FROM "${stream}" WHERE trace_id = '${traceId}'`, + start_time: (startMs - MINUTE_MS) * 1000, + end_time: (endMs + MINUTE_MS) * 1000, + from: 0, + size: 10, + }, + }, + }); + if (response.status() !== 200) return -response.status(); + return ((await response.json()).hits ?? []).length; + }, + { timeout: READY_TIMEOUT_MS, intervals: READY_INTERVALS, message: `trace ${traceId} searchable in ${stream}` }, + ) + .toBeGreaterThan(0); +} + +async function deleteTracesStream(request, stream) { + const org = getOrgIdentifier(); + try { + await request.delete(`${baseUrl()}/api/${org}/streams/${stream}?type=traces&delete_all=true`, { + headers: getAuthHeaders(), + }); + } catch (error) { + testLogger.warn('Traces stream delete failed', { stream, error: error.message }); + } +} + +module.exports = { + XSS_FUNCTION, + MINUTE_MS, + randomHex, + uniqueProfilesStream, + referenceT0Ms, + referenceWindowUs, + buildProfilesRequest, + referenceResources, + manyVersionsResources, + unknownServiceResources, + zeroValueResources, + noLabelResources, + ingestProfiles, + seedReferenceStream, + seedProfilesStream, + waitForProfilesTotal, + waitForProfilesService, + deleteProfilesStream, + ingestTraceSpans, + deleteTracesStream, +}; diff --git a/web/src/assets/dashboards/host_metrics.dashboard.json b/web/src/assets/dashboards/host_metrics.dashboard.json index 9f487af3be..696f5e3827 100644 --- a/web/src/assets/dashboards/host_metrics.dashboard.json +++ b/web/src/assets/dashboards/host_metrics.dashboard.json @@ -98,7 +98,7 @@ "queryType": "promql", "queries": [ { - "query": "100 * (1 - avg(irate(system_cpu_time{state=\"idle\",host_name=~\"$host_name\"}[5m])))", + "query": "100 * (1 - sum(irate(system_cpu_time{state=\"idle\",host_name=~\"$host_name\"}[5m])) / sum(irate(system_cpu_time{host_name=~\"$host_name\"}[5m])))", "vrlFunctionQuery": "", "customQuery": true, "fields": { @@ -379,7 +379,7 @@ "queryType": "promql", "queries": [ { - "query": "topk(10, 100 * (1 - avg by (host_name)(irate(system_cpu_time{state=\"idle\",host_name=~\"$host_name\"}[5m]))))", + "query": "topk(10, 100 * (1 - sum by (host_name)(irate(system_cpu_time{state=\"idle\",host_name=~\"$host_name\"}[5m])) / sum by (host_name)(irate(system_cpu_time{host_name=~\"$host_name\"}[5m]))))", "vrlFunctionQuery": "", "customQuery": true, "fields": { @@ -629,7 +629,7 @@ "queryType": "promql", "queries": [ { - "query": "100 * (1 - avg by (host_name)(irate(system_cpu_time{state=\"idle\",host_name=~\"$host_name\"}[5m])))", + "query": "100 * (1 - sum by (host_name)(irate(system_cpu_time{state=\"idle\",host_name=~\"$host_name\"}[5m])) / sum by (host_name)(irate(system_cpu_time{host_name=~\"$host_name\"}[5m])))", "vrlFunctionQuery": "", "customQuery": true, "fields": { @@ -690,7 +690,7 @@ "queryType": "promql", "queries": [ { - "query": "avg by (state)(irate(system_cpu_time{host_name=~\"$host_name\",state!=\"idle\"}[5m]))", + "query": "sum by (state)(irate(system_cpu_time{host_name=~\"$host_name\",state!=\"idle\"}[5m])) / ignoring(state) group_left sum(irate(system_cpu_time{host_name=~\"$host_name\"}[5m]))", "vrlFunctionQuery": "", "customQuery": true, "fields": { diff --git a/web/src/assets/images/dashboard/charts/boxplot.svg b/web/src/assets/images/dashboard/charts/boxplot.svg new file mode 100644 index 0000000000..4d6d7ffa13 --- /dev/null +++ b/web/src/assets/images/dashboard/charts/boxplot.svg @@ -0,0 +1,9 @@ + + + + + + + + + diff --git a/web/src/assets/images/dashboard/charts/logs.svg b/web/src/assets/images/dashboard/charts/logs.svg new file mode 100644 index 0000000000..1065a7b49f --- /dev/null +++ b/web/src/assets/images/dashboard/charts/logs.svg @@ -0,0 +1,10 @@ + + + + + + + + + + diff --git a/web/src/components/CodeQueryEditor.spec.ts b/web/src/components/CodeQueryEditor.spec.ts index 7d8ba53d96..e982a62ec4 100644 --- a/web/src/components/CodeQueryEditor.spec.ts +++ b/web/src/components/CodeQueryEditor.spec.ts @@ -777,6 +777,44 @@ describe("CodeQueryEditor", () => { expect(wrapper.emitted("update:query")).toBeFalsy(); }); + it("passes whether the text is blank and was blank, counting a programmatic restore", async () => { + const lines = (...text: string[]) => { + mockModel.getLineCount.mockReturnValue(text.length); + mockModel.getLineContent.mockImplementation((line: number) => text[line - 1]); + }; + const wrapper = await mountAndSetup(); + mockEditorObj.setValue.mockImplementation(() => changeHandler()({ isFlush: false })); + lines("level='error'"); + const editorApi = wrapper.vm as unknown as { setValue: (value: string) => void }; + editorApi.setValue("level='error'"); + expect(wrapper.emitted("user-edit")).toBeFalsy(); + + lines(" ", ""); + changeHandler()({ isFlush: false }); + lines("", "u"); + changeHandler()({ isFlush: false }); + expect(wrapper.emitted("user-edit")).toEqual([ + [true, false], + [false, true], + ]); + lines(""); + }); + + it("reads only up to the first non-blank line on an edit, never the whole document", async () => { + await mountAndSetup(); + mockModel.getLineCount.mockReturnValue(10_000); + mockModel.getLineContent.mockImplementation((line: number) => (line === 1 ? "up" : "x")); + mockModel.getLineContent.mockClear(); + mockModel.getValue.mockClear(); + mockEditorObj.getValue.mockClear(); + changeHandler()({ isFlush: false }); + expect(mockModel.getLineContent).toHaveBeenCalledTimes(1); + expect(mockModel.getValue).not.toHaveBeenCalled(); + expect(mockEditorObj.getValue).not.toHaveBeenCalled(); + mockModel.getLineCount.mockReturnValue(1); + mockModel.getLineContent.mockImplementation(() => ""); + }); + it("ignores a flush (a whole-model replace)", async () => { const wrapper = await mountAndSetup(); changeHandler()({ isFlush: true }); diff --git a/web/src/components/CodeQueryEditor.vue b/web/src/components/CodeQueryEditor.vue index eea90e96c3..64c59d18d0 100644 --- a/web/src/components/CodeQueryEditor.vue +++ b/web/src/components/CodeQueryEditor.vue @@ -102,6 +102,7 @@ const loadMonaco = (): Promise => { }; import { vrlLanguageDefinition } from "@/utils/query/vrlLanguageDefinition"; +import { recentCompletionItems } from "@/utils/queryHistory/recentCompletions"; /** * Per-editor configuration, keyed by model URI. @@ -130,6 +131,8 @@ interface EditorConfig { showHint: (text: string | null) => void; /** Set when the value lookup searches a caller's own time range instead of the default lookback. */ valuesTimeRange: () => { startTime: number; endTime: number } | null; + /** The user's recent queries, offered as clock rows that replace the editor text. */ + recents: () => string[]; } interface ChainState { @@ -267,6 +270,15 @@ const editorConfigs = new Map(); const registeredLanguages = new Set(); const modelKey = (model: any): string => model?.uri?.toString?.() ?? ""; +// Stops at the first non-blank line, so an edit never copies the whole document. +const modelIsBlank = (model: any): boolean => { + const lines: number = model?.getLineCount?.() ?? 0; + for (let line = 1; line <= lines; line++) { + if (model.getLineContent(line).trim()) return false; + } + return true; +}; + // Global commands report here; the model says which editor accepted. const onSuggestAccepted = (editor: any, lane: SuggestLane | "", chained: boolean) => { const config = editorConfigs.get(modelKey(editor?.getModel?.())); @@ -392,6 +404,10 @@ export default defineComponent({ type: Array, default: null, }, + recentQueries: { + type: Array as PropType, + default: () => [], + }, debounceTime: { type: Number, default: 500, @@ -805,6 +821,7 @@ export default defineComponent({ announce: announceMenu, showHint, valuesTimeRange: () => props.valuesTimeRange ?? null, + recents: () => props.recentQueries, }); }; @@ -1010,7 +1027,11 @@ export default defineComponent({ hintDecorations?.clear(); syncValueQuoteKey(); }); + // Programmatic writes update it too, so a restore followed by a clear reads as non-empty → empty. + let isBlank = modelIsBlank(editorObj.getModel()); editorObj.onDidChangeModelContent((e: any) => { + const wasBlank = isBlank; + isBlank = modelIsBlank(editorObj.getModel()); const [change] = e?.changes ?? []; const chainSpace = chainState.typingSpace && e?.changes?.length === 1 && change.text === " "; @@ -1020,7 +1041,7 @@ export default defineComponent({ : null; if (!e?.isFlush && !programmatic) { pendingUserEdit = true; - emit("user-edit"); + emit("user-edit", isBlank, wasBlank); } else { // The debounce emits the latest value, so a write landing on a typed one owns the origin. pendingUserEdit = false; @@ -1909,7 +1930,19 @@ export default defineComponent({ if (language === "promql" && context?.triggerCharacter === "=") return { suggestions: [] }; - return reply(catalogEntries(config, model, position, textUntilPosition, word.word)); + const catalog = reply( + catalogEntries(config, model, position, textUntilPosition, word.word), + ); + const recents = recentCompletionItems( + config.recents(), + model, + position, + kinds().Reference, + t("recentSearches.recentRow"), + ); + return recents.length + ? { ...catalog, suggestions: [...recents, ...catalog.suggestions] } + : catalog; }, }); diff --git a/web/src/components/DateTime.spec.ts b/web/src/components/DateTime.spec.ts index 6922dc9ebe..5c4ba8eb57 100644 --- a/web/src/components/DateTime.spec.ts +++ b/web/src/components/DateTime.spec.ts @@ -807,4 +807,20 @@ describe("DateTime Component", () => { expect(prev(wrapper).props("disabled")).toBe(true); }); }); + + describe("hideSeconds", () => { + const units = (list: Array<{ value: string }>) => list.map((period) => period.value); + + it("offers Seconds by default, as every existing picker has it", () => { + wrapper = createWrapper(); + expect(units(wrapper.vm.relativePeriods)).toEqual(["s", "m", "h", "d", "w", "M"]); + expect(units(wrapper.vm.relativePeriodsSelect)).toEqual(["s", "m", "h", "d", "w", "M"]); + }); + + it("drops Seconds from the rows and the custom unit when the parent asks", () => { + wrapper = createWrapper({ hideSeconds: true }); + expect(units(wrapper.vm.relativePeriods)).toEqual(["m", "h", "d", "w", "M"]); + expect(units(wrapper.vm.relativePeriodsSelect)).toEqual(["m", "h", "d", "w", "M"]); + }); + }); }); diff --git a/web/src/components/DateTime.vue b/web/src/components/DateTime.vue index 1da1a913bb..1ce243a0a9 100644 --- a/web/src/components/DateTime.vue +++ b/web/src/components/DateTime.vue @@ -580,6 +580,11 @@ export default defineComponent({ type: Boolean, default: false, }, + /** Drops the relative Seconds unit, for a caller that accepts only whole minutes. */ + hideSeconds: { + type: Boolean, + default: false, + }, }, emits: ["on:date-change", "on:timezone-change", "hide", "show"], @@ -678,16 +683,9 @@ export default defineComponent({ { label: t("common.days"), value: "d" }, { label: t("common.weeks"), value: "w" }, { label: t("common.months"), value: "M" }, - ]; + ].filter((period) => !props.hideSeconds || period.value !== "s"); - let relativePeriodsSelect = ref([ - { label: t("common.seconds"), value: "s" }, - { label: t("common.minutes"), value: "m" }, - { label: t("common.hours"), value: "h" }, - { label: t("common.days"), value: "d" }, - { label: t("common.weeks"), value: "w" }, - { label: t("common.months"), value: "M" }, - ]); + let relativePeriodsSelect = ref([...relativePeriods]); const relativeDates: Record = { s: [1, 5, 10, 15, 30, 45], diff --git a/web/src/components/QueryEditor.vue b/web/src/components/QueryEditor.vue index 83c5274522..4794b88a55 100644 --- a/web/src/components/QueryEditor.vue +++ b/web/src/components/QueryEditor.vue @@ -99,8 +99,9 @@ :free-text-decorations="freeTextDecorations ?? null" :filter-mode="filterMode" :values-time-range="valuesTimeRange ?? null" + :recent-queries="recentQueries ?? []" @update:query="handleQueryUpdate" - @user-edit="emit('user-edit')" + @user-edit="(isBlank: boolean, wasBlank: boolean) => emit('user-edit', isBlank, wasBlank)" @run-query="emit('run-query')" @focus="handleEditorFocus" @blur="handleEditorBlur" @@ -189,6 +190,7 @@ interface Props { freeTextDecorations?: { ranges: { start: number; end: number }[]; hover: I18nText } | null; filterMode?: boolean; // The query is a bare WHERE fragment (Logs without SQL mode) valuesTimeRange?: { startTime: number; endTime: number } | null; // Range field values are looked up in (microseconds) + recentQueries?: string[]; // The user's recent queries, offered as clock rows while typing // NL Mode (optional external control) nlpMode?: boolean; // External NLP mode control (undefined = internal control) @@ -226,7 +228,7 @@ const props = withDefaults(defineProps(), { const emit = defineEmits<{ "update:query": [query: string, origin?: QueryChangeOrigin]; - "user-edit": []; + "user-edit": [isBlank: boolean, wasBlank: boolean]; "language-change": [language: Language]; "ask-ai": [naturalLanguage: string, language: Language]; "run-query": []; diff --git a/web/src/components/alerts/AddAlert.anomalyEdit.spec.ts b/web/src/components/alerts/AddAlert.anomalyEdit.spec.ts index 5e3fcc8a95..f13380c0df 100644 --- a/web/src/components/alerts/AddAlert.anomalyEdit.spec.ts +++ b/web/src/components/alerts/AddAlert.anomalyEdit.spec.ts @@ -96,7 +96,6 @@ const stubs = { QueryConfig: true, AlertSettings: true, CompareWithPast: true, - Deduplication: true, Advanced: true, PreviewAlert: true, AlertSummary: true, diff --git a/web/src/components/alerts/AddAlert.drafts.spec.ts b/web/src/components/alerts/AddAlert.drafts.spec.ts index 5911d35523..457b949064 100644 --- a/web/src/components/alerts/AddAlert.drafts.spec.ts +++ b/web/src/components/alerts/AddAlert.drafts.spec.ts @@ -151,7 +151,6 @@ const stubs = { QueryConfig: true, AlertSettings: true, CompareWithPast: true, - Deduplication: true, Advanced: true, PreviewAlert: true, AlertSummary: true, diff --git a/web/src/components/alerts/AddAlert.panelPrefill.spec.ts b/web/src/components/alerts/AddAlert.panelPrefill.spec.ts index 298f020608..2cceeed60d 100644 --- a/web/src/components/alerts/AddAlert.panelPrefill.spec.ts +++ b/web/src/components/alerts/AddAlert.panelPrefill.spec.ts @@ -113,7 +113,6 @@ vi.mock("@/services/reodotdev_analytics", () => ({ useReo: () => ({ track: vi.fn const stubs = { AlertSettings: true, CompareWithPast: true, - Deduplication: true, Advanced: true, PanelSchemaRenderer: true, AlertSummary: true, diff --git a/web/src/components/alerts/AddAlert.prefill.spec.ts b/web/src/components/alerts/AddAlert.prefill.spec.ts index 3aef922cae..aea1f51a8e 100644 --- a/web/src/components/alerts/AddAlert.prefill.spec.ts +++ b/web/src/components/alerts/AddAlert.prefill.spec.ts @@ -132,7 +132,6 @@ vi.mock("@/services/reodotdev_analytics", () => ({ useReo: () => ({ track: vi.fn const stubs = { AlertSettings: true, CompareWithPast: true, - Deduplication: true, Advanced: true, PanelSchemaRenderer: true, AlertSummary: true, diff --git a/web/src/components/alerts/AddAlert.schema.ts b/web/src/components/alerts/AddAlert.schema.ts index 3b1750fa4a..df90217f86 100644 --- a/web/src/components/alerts/AddAlert.schema.ts +++ b/web/src/components/alerts/AddAlert.schema.ts @@ -16,7 +16,7 @@ // AddAlert orchestrator schema. // // AddAlert owns the ONE for the whole wizard. The step children -// (QueryConfig, AlertSettings, Advanced, Deduplication) render as descendants +// (QueryConfig, AlertSettings, Advanced) render as descendants // and bind their fields by nested `name=` into THIS form, so the schema below // must cover every nested path they bind. This file composes their exported // rules (single source of truth) and adds the topbar scalars: @@ -55,7 +55,7 @@ export const ALERT_NAME_UNSUPPORTED_CHARS = /[:#?\s'"%&]+/; export const isBlank = (v: unknown): boolean => v === undefined || v === null || (typeof v === "string" && v.trim() === ""); -// Step-B field schemas (Advanced / Deduplication / CompareWithPast wizard step +// Step-B field schemas (Advanced / CompareWithPast wizard step // children). All optional/permissive. The step components bind these exact paths // by nested `name=` into THIS orchestrator form. @@ -89,18 +89,6 @@ export const advancedShape = { export const createAdvancedSchema = () => z.looseObject({ ...advancedShape }); export type AdvancedForm = z.infer>; -/** Deduplication (scheduled-only). All fields optional. `enabled` is derived - * from `fingerprint_fields.length > 0` by the component; `time_window_minutes` - * is sanitized to number|undefined by the component — the raw form state may - * transiently hold a string, so the union keeps validation permissive (never - * rejects while typing). */ -export const deduplicationSchema = z.object({ - enabled: z.boolean().optional(), - fingerprint_fields: z.array(z.string()).optional(), - time_window_minutes: z.union([z.number(), z.string(), z.undefined()]).optional(), -}); -export type DeduplicationForm = z.infer; - /** Compare-with-past reference-window row. `offSet` is a relative-time string * (e.g. "15m") edited by the bare CustomDateTimePicker. `uuid` is the row * identity used as the v-for `:key` — correct here because the row is bound by @@ -136,10 +124,9 @@ export const makeAddAlertSchema = (t: Translator) => trigger_condition: z.looseObject({}).optional(), query_condition: z.looseObject({}).optional(), logGroupBy: z.array(z.string()).optional(), - // Step-B fields (Advanced / Deduplication tabs). Permissive pass-throughs, + // Step-B fields (Advanced tab). Permissive pass-throughs, // not validated — the superRefine below adds no issues for them. context_attributes: contextAttributesSchema.optional(), - deduplication: deduplicationSchema.optional(), template: z.string().optional(), description: z.string().optional(), row_template: z.string().optional(), diff --git a/web/src/components/alerts/AddAlert.spec.ts b/web/src/components/alerts/AddAlert.spec.ts index 03159d1af9..1d993b11be 100644 --- a/web/src/components/alerts/AddAlert.spec.ts +++ b/web/src/components/alerts/AddAlert.spec.ts @@ -135,7 +135,6 @@ const stubs = { QueryConfig: true, AlertSettings: true, CompareWithPast: true, - Deduplication: true, Advanced: true, PreviewAlert: true, AlertSummary: true, @@ -680,6 +679,7 @@ describe("AddAlert (OForm owner)", () => { "notify_on_recovery", "recovery_destinations", "keep_firing_for", + "suppress_repeats", ].sort(), ); diff --git a/web/src/components/alerts/AddAlert.vue b/web/src/components/alerts/AddAlert.vue index ee2655961c..b7a3cfb41d 100644 --- a/web/src/components/alerts/AddAlert.vue +++ b/web/src/components/alerts/AddAlert.vue @@ -417,15 +417,6 @@ along with this program. If not, see . @update:multiTimeRange="updateMultiTimeRange" />
- - -
- -
@@ -650,7 +641,6 @@ import JsonEditor from "../common/JsonEditor.vue"; import QueryConfig from "./steps/QueryConfig.vue"; import AlertSettings from "./steps/AlertSettings.vue"; import CompareWithPast from "./steps/CompareWithPast.vue"; -import Deduplication from "./steps/Deduplication.vue"; import Advanced from "./steps/Advanced.vue"; import InlineSelectFolderDropdown from "../common/sidebar/InlineSelectFolderDropdown.vue"; import PreviewAlert from "./PreviewAlert.vue"; @@ -712,7 +702,6 @@ export default defineComponent({ QueryConfig, AlertSettings, CompareWithPast, - Deduplication, Advanced, PreviewAlert, AlertSummary, diff --git a/web/src/components/alerts/AlertHistory.ack.spec.ts b/web/src/components/alerts/AlertHistory.ack.spec.ts index a9083dbc55..2f69c2d0f5 100644 --- a/web/src/components/alerts/AlertHistory.ack.spec.ts +++ b/web/src/components/alerts/AlertHistory.ack.spec.ts @@ -46,7 +46,6 @@ const run = { is_realtime: false, is_silenced: false, retries: 0, - dedup_enabled: false, }; const hits = [ @@ -60,6 +59,7 @@ const hits = [ ack_source: "slack", }, { ...run, timestamp: 1_786_498_000_000_000, status: "firing" }, + { ...run, timestamp: 1_786_497_000_000_000, status: "firing", repeat_suppressed: true }, ]; let wrapper: VueWrapper; @@ -95,9 +95,9 @@ describe("AlertHistory ack rows", () => { it("shows an ack-suppressed run as Acknowledged, with whose ack suppressed it", async () => { await mountHistory(); - expect(rowsOf()).toHaveLength(3); + expect(rowsOf()).toHaveLength(4); expect(cell(0, "status").text()).toContain("Acknowledged"); - expect(cell(0, "dedup").text()).toContain("suppressed by ack (bob)"); + expect(cell(0, "suppression").text()).toContain("suppressed by ack (bob)"); }); it("gives an ack event its own row with who, how and no run fields", async () => { @@ -105,12 +105,18 @@ describe("AlertHistory ack rows", () => { expect(cell(1, "status").text()).toContain("Ack event"); expect(cell(1, "condition").text()).toContain("Acknowledged by bob@acme.io from Slack"); expect(cell(1, "duration").text()).toBe("—"); - expect(cell(1, "dedup").text()).toBe("—"); + expect(cell(1, "suppression").text()).toBe("—"); }); it("leaves an ordinary run as it was", async () => { await mountHistory(); expect(cell(2, "status").text()).toContain("Firing"); - expect(cell(2, "dedup").text()).not.toContain("suppressed by ack"); + expect(cell(2, "suppression").text()).not.toContain("suppressed by ack"); + expect(cell(2, "suppression").text()).toBe("—"); + }); + + it("marks a run whose notification was withheld as a repeat", async () => { + await mountHistory(); + expect(cell(3, "suppression").text()).toContain("Suppressed as a repeat"); }); }); diff --git a/web/src/components/alerts/AlertHistory.vue b/web/src/components/alerts/AlertHistory.vue index 66adcf29d3..8d56088880 100644 --- a/web/src/components/alerts/AlertHistory.vue +++ b/web/src/components/alerts/AlertHistory.vue @@ -247,7 +247,7 @@ along with this program. If not, see . - @@ -361,7 +399,14 @@ along with this program. If not, see . import { orgUsersQuery, assignableRolesQuery, allUserRolesQuery } from "@/services/users.queries"; import { rolesQuery } from "@/services/iam.queries"; import { userKeys } from "@/services/users.querykeys"; +import { + orgDomainsQuery, + orgSettingsQuery, + updateOrgSettingsMutation, +} from "@/services/organizations.queries"; +import { blockEmailsInConfig } from "@/components/settings/orgDomainRestrictions"; import { queryClient } from "@/composables/query/queryClient"; +import { useMutation } from "@tanstack/vue-query"; import { defineComponent, ref, onActivated, onBeforeMount, watch, computed } from "vue"; import OButton from "@/lib/core/Button/OButton.vue"; import ORefreshButton from "@/lib/core/RefreshButton/ORefreshButton.vue"; @@ -428,6 +473,12 @@ export default defineComponent({ const showAddUserDialog: any = ref(false); const confirmDelete = ref(false); const confirmRevoke = ref(false); + const confirmBlock = ref(false); + const blockUserEmail = ref(""); + const verifiedDomains = ref>(new Set()); + const updateSettings = useMutation(() => + updateOrgSettingsMutation(store.state.selectedOrganization.identifier), + ); const selectedUser: any = ref({}); const orgData: any = ref(store.state.selectedOrganization); const isUpdated: any = ref(false); @@ -461,7 +512,7 @@ export default defineComponent({ onBeforeMount(async () => { isEnterprise.value = config.isEnterprise == "true"; - await getOrgMembers(); + await Promise.all([getOrgMembers(), getVerifiedDomains()]); updateUserActions(); await getRoles(); await getCustomRoles(); @@ -1039,6 +1090,7 @@ export default defineComponent({ member.enableEdit = shouldAllowEdit(member); member.enableChangeRole = shouldAllowChangeRole(member); member.enableDelete = shouldAllowDelete(member); + member.isDomainMapped = isDomainMapped(member); }); }; @@ -1355,6 +1407,70 @@ export default defineComponent({ }); }; + // Only verified domains auto-add users on SSO login, so only those users would come back after a plain delete. + const getVerifiedDomains = async () => { + if (config.isCloud !== "true") return; + try { + const domains = await queryClient.fetchQuery( + orgDomainsQuery(store.state.selectedOrganization.identifier), + ); + verifiedDomains.value = new Set( + domains.filter((d) => d.verification_state === 1).map((d) => d.domain.toLowerCase()), + ); + } catch { + verifiedDomains.value = new Set(); + } + }; + + const isDomainMapped = (user: any) => { + if (config.isCloud !== "true") return false; + const domain = String(user.email ?? "") + .toLowerCase() + .split("@")[1]; + return !!domain && verifiedDomains.value.has(domain); + }; + + const confirmBlockAction = (row: any) => { + confirmBlock.value = true; + blockUserEmail.value = row.email; + }; + + const writeBlocks = async (org: string, emails: string[]) => { + try { + // Fetch fresh so the write is not based on a stale cached config. + const settings = await queryClient.fetchQuery({ ...orgSettingsQuery(org), staleTime: 0 }); + await updateSettings.mutateAsync({ + domain_management_config: blockEmailsInConfig( + settings?.data?.domain_management_config, + emails, + ), + }); + return true; + } catch (err: any) { + toast({ + variant: "error", + message: + raw(err?.response?.data?.message || err?.message) || t("iam.user.errorBlockingUser"), + }); + return false; + } + }; + + // The user is removed from this org only; the next SSO login evaluates the block and removes them from mapped child orgs. + const blockUser = async () => { + confirmBlock.value = false; + const org = store.state.selectedOrganization.identifier; + if (!(await writeBlocks(org, [blockUserEmail.value]))) return; + try { + await usersService.delete(org, blockUserEmail.value); + toast({ variant: "success", message: t("iam.user.userBlockedSuccess") }); + } catch { + toast({ variant: "error", message: t("iam.user.blockedButNotRemoved") }); + } + await getOrgMembers(true); + updateUserActions(); + }; + const confirmRevokeAction = (row: any) => { confirmRevoke.value = true; revokeInviteToken = row.token; @@ -1435,53 +1551,83 @@ export default defineComponent({ confirmBulkDelete.value = true; }; - const bulkDeleteUsers = async () => { - bulkDeleteLoading.value = true; - const userEmails = selectedUsers.value.map((user: any) => user.email); - + const bulkRemove = async (org: string, emails: string[], blocked: boolean) => { try { - const res = await usersService.bulkDelete(store.state.selectedOrganization.identifier, { - ids: userEmails, - }); + const res = await usersService.bulkDelete(org, { ids: emails }); const { successful, unsuccessful } = res.data; if (successful.length > 0 && unsuccessful.length === 0) { toast({ - message: t("iam.user.deletedUsersSuccess", { count: successful.length }), + message: blocked + ? t("iam.user.blockedUsersSuccess", { count: successful.length }) + : t("iam.user.deletedUsersSuccess", { count: successful.length }), variant: "success", }); } else if (successful.length > 0 && unsuccessful.length > 0) { + const counts = { count: successful.length, failed: unsuccessful.length }; toast({ - message: t("iam.user.deletedUsersPartial", { - count: successful.length, - failed: unsuccessful.length, - }), + message: blocked + ? t("iam.user.blockedUsersPartial", counts) + : t("iam.user.deletedUsersPartial", counts), variant: "warning", }); } else if (unsuccessful.length > 0) { toast({ - message: t("iam.user.failedToDeleteUsers", { count: unsuccessful.length }), + message: blocked + ? t("iam.user.blockedUsersNotRemoved", { count: unsuccessful.length }) + : t("iam.user.failedToDeleteUsers", { count: unsuccessful.length }), variant: "error", }); } + } catch (err: any) { + if (err.response?.status != 403 || err?.status != 403) { + toast({ + message: blocked + ? t("iam.user.blockedUsersNotRemoved", { count: emails.length }) + : err.response?.data?.message || err?.message || t("iam.user.errorDeletingUsers"), + variant: "error", + }); + } + } + }; + const bulkBlock = async (org: string, emails: string[]) => { + if (await writeBlocks(org, emails)) await bulkRemove(org, emails, true); + }; + + const bulkDeleteUsers = async () => { + bulkDeleteLoading.value = true; + const org = store.state.selectedOrganization.identifier; + const blockable = selectedUsers.value.filter((u: any) => u.isDomainMapped); + const deletable = selectedUsers.value.filter((u: any) => !u.isDomainMapped); + + try { + await Promise.all([ + deletable.length > 0 && + bulkRemove( + org, + deletable.map((u: any) => u.email), + false, + ), + blockable.length > 0 && + bulkBlock( + org, + blockable.map((u: any) => u.email), + ), + ]); selectedUsers.value = []; confirmBulkDelete.value = false; await getOrgMembers(true); updateUserActions(); - } catch (err: any) { - if (err.response?.status != 403 || err?.status != 403) { - toast({ - message: - err.response?.data?.message || err?.message || t("iam.user.errorDeletingUsers"), - variant: "error", - }); - } } finally { bulkDeleteLoading.value = false; } }; + const bulkBlockCount = computed( + () => selectedUsers.value.filter((u: any) => u.isDomainMapped).length, + ); + const updateUserRole = (row: any) => { const dismiss = toast({ variant: "loading", @@ -1585,6 +1731,10 @@ export default defineComponent({ deleteUserEmail, deleteUser, confirmDeleteAction, + confirmBlock, + blockUser, + blockUserEmail, + confirmBlockAction, confirmRevoke, revokeInvite, revokeInviteEmail, @@ -1640,6 +1790,7 @@ export default defineComponent({ bulkDeleteLoading, openBulkDeleteDialog, bulkDeleteUsers, + bulkBlockCount, rows, displayedRows, roleFilter, diff --git a/web/src/components/ingestion/recommended/KubernetesConfig.spec.ts b/web/src/components/ingestion/recommended/KubernetesConfig.spec.ts index 3a828dfe47..2fac5084ae 100644 --- a/web/src/components/ingestion/recommended/KubernetesConfig.spec.ts +++ b/web/src/components/ingestion/recommended/KubernetesConfig.spec.ts @@ -84,7 +84,7 @@ describe("kubernetesCard builder", () => { const card = await buildCard(); expect(card.provider.name).toBe("Kubernetes"); expect(card.provider.metaBadges).toEqual(["Logs", "Metrics", "Events", "Traces"]); - expect(card.steps.map((s) => s.id)).toEqual(["install", "instrument", "verify"]); + expect(card.steps.map((s) => s.id)).toEqual(["install", "instrument", "network", "verify"]); expect(card.detect).toMatchObject({ streamType: "logs", streamName: "default", @@ -220,6 +220,21 @@ describe("kubernetesCard builder", () => { expect(ebpf.inputs?.every((i) => i.default === "")).toBe(true); }); + it("offers eBPF network flows as an optional upgrade that keeps trace selectors", async () => { + const step = (await buildCard()).steps.find((s) => s.id === "network")!; + expect(step.required).toBeUndefined(); + expect(step.code?.raw).toBe( + [ + "helm repo update openobserve", + "helm --namespace openobserve-collector upgrade o2c openobserve/openobserve-collector \\", + " --version '>=0.8.0' --reset-then-reuse-values \\", + " --set ebpf.enabled=true \\", + " --set ebpf.network.enabled=true", + ].join("\n"), + ); + expect(step.note).toContain("`127.0.0.1:8888`"); + }); + it("drops the deprecated Go operator sidecar", async () => { const step = (await buildCard()).steps.find((s) => s.id === "instrument")!; const json = JSON.stringify(step); @@ -247,6 +262,9 @@ describe("kubernetesCard builder", () => { expect(urls).toContain( "https://github.com/openobserve/openobserve-helm-chart/blob/main/charts/openobserve-collector/docs/traces.md#ebpf-tracing", ); + expect(urls).toContain( + "https://github.com/openobserve/openobserve-helm-chart/blob/main/charts/openobserve-collector/docs/metrics.md#13-ebpf-network-flows", + ); }); it("carries troubleshooting guidance without markdown link syntax", async () => { diff --git a/web/src/components/ingestion/recommended/LinuxConfig.spec.ts b/web/src/components/ingestion/recommended/LinuxConfig.spec.ts index e79d9d51a1..8c8f067197 100644 --- a/web/src/components/ingestion/recommended/LinuxConfig.spec.ts +++ b/web/src/components/ingestion/recommended/LinuxConfig.spec.ts @@ -67,7 +67,7 @@ describe("linuxCard builder", () => { const card = linuxCard(SUBS, gt); expect(card.provider.name).toBe("Linux"); expect(card.provider.metaBadges).toEqual(["Logs", "Metrics", "Traces"]); - expect(card.steps.map((s) => s.id)).toEqual(["install", "verify", "dashboard"]); + expect(card.steps.map((s) => s.id)).toEqual(["install", "network", "verify", "dashboard"]); // Host metrics fan out per metric, so detection is existence-based. expect(card.detect).toMatchObject({ streamType: "metrics", @@ -123,12 +123,32 @@ describe("linuxCard builder", () => { const card = linuxCard(SUBS, gt); expect(card.docLinks?.map((l) => l.url)).toEqual([ "https://github.com/openobserve/agents/blob/main/obi.md", + "https://github.com/openobserve/agents/blob/main/obi.md#network-flows", ]); - expect(card.steps[1].note).toContain("traced app handles a request"); + expect(card.steps.find((s) => s.id === "verify")!.note).toContain( + "traced app handles a request", + ); // Detection stays on host metrics, tracing or not. expect(card.detect?.streamName).toBe("system_"); }); + it("offers network flows as an optional re-run that needs at least one CIDR", () => { + const step = linuxCard(SUBS, gt).steps.find((s) => s.id === "network")!; + expect(step.required).toBeUndefined(); + expect(step.variantToggle).toBe(false); + expect(step.variants).toHaveLength(1); + const [flows] = step.variants!; + if ("variants" in flows) throw new Error("network flows must not be nested"); + expect(flows.code.raw).toBe(`${BASE_INSTALL}{flags}`); + expect(flows.code.masked).not.toContain(SUBS.token); + expect(flows.inputs?.map((i) => [i.id, i.flag, i.default])).toEqual([ + ["networkFlows", "--network-flows", ""], + ]); + expect(flows.requireOneOf?.ids).toEqual(["networkFlows"]); + // Re-running the installer replaces the config, so the note says how to keep tracing. + expect(flows.note).toContain("`--trace-*`"); + }); + it("offers the uninstall command", () => { const uninstall = linuxCard(SUBS, gt).extras?.uninstall; expect(uninstall?.code.raw).toContain("/linux/uninstall.sh"); diff --git a/web/src/components/ingestion/setupCard/SetupCardRenderer.spec.ts b/web/src/components/ingestion/setupCard/SetupCardRenderer.spec.ts index 3bbd417dfe..98d82b5464 100644 --- a/web/src/components/ingestion/setupCard/SetupCardRenderer.spec.ts +++ b/web/src/components/ingestion/setupCard/SetupCardRenderer.spec.ts @@ -747,6 +747,25 @@ describe("SetupCardRenderer — Linux install step tracing toggle", () => { expect(step().props("done")).toBe(true); }); + it("shows the network flows command only once a CIDR is given, with no toggle", async () => { + wrapper = mountLinux(); + const step = () => wrapper.find('[data-test="ai-step-network"]'); + expect(step().find('[data-test="ai-variant-tabs"]').exists()).toBe(false); + expect(wrapper.find('[data-test="ai-variant-require-network"]').exists()).toBe(true); + expect(step().findComponent(OCodeBlock).exists()).toBe(false); + + await type("networkFlows", "10.0.1.5/32=web1,0.0.0.0/0=other"); + expect(wrapper.find('[data-test="ai-variant-require-network"]').exists()).toBe(false); + expect( + step() + .findComponent(OCodeBlock) + .props("code") + .endsWith(`${BASE_TAIL} --network-flows '10.0.1.5/32=web1,0.0.0.0/0=other'`), + ).toBe(true); + // The install step is untouched by the network field. + expect(installBlock().props("code").endsWith(BASE_TAIL)).toBe(true); + }); + it("leaves step-level inputs where they were (Kubernetes cluster name)", async () => { wrapper = mount(SetupCardRenderer, { props: { content: kubernetesCard(LINUX_SUBS, gt), subs: LINUX_SUBS }, @@ -775,7 +794,12 @@ describe("SetupCardRenderer — Kubernetes instrument step (nested toggle)", () helm --namespace openobserve-collector upgrade o2c openobserve/openobserve-collector \\ --version '>=0.6.0' --reset-then-reuse-values \\ --set ebpf.enabled=true`; - const codes = () => wrapper.findAllComponents(OCodeBlock).map((b) => b.props("code") as string); + // Scoped to the instrument step: the network flows step below also runs `upgrade o2c`. + const codes = () => + wrapper + .find('[data-test="ai-step-instrument"]') + .findAllComponents(OCodeBlock) + .map((b) => b.props("code") as string); const instrumentCode = () => codes().find((c) => c.includes("kubectl annotate") || c.includes("upgrade o2c")); const field = (id: string) => wrapper.find(`[data-test="ai-input-${id}"]`); diff --git a/web/src/components/ingestion/setupCard/content/kubernetes.ts b/web/src/components/ingestion/setupCard/content/kubernetes.ts index 996a31c1a7..e2293554f7 100644 --- a/web/src/components/ingestion/setupCard/content/kubernetes.ts +++ b/web/src/components/ingestion/setupCard/content/kubernetes.ts @@ -13,7 +13,7 @@ // You should have received a copy of the GNU Affero General Public License // along with this program. If not, see . -// Kubernetes setup card: install → optionally instrument for traces (language libraries or eBPF) → verify. +// Kubernetes setup card: install → optional traces (libraries or eBPF) → optional eBPF network flows → verify. // // Step 1 deliberately offers ONLY the two endpoint choices — external (reach // OpenObserve over the network) and internal (OpenObserve runs in this same @@ -131,6 +131,16 @@ helm --namespace openobserve-collector upgrade o2c openobserve/openobserve-colle --version '>=0.6.0' --reset-then-reuse-values \\ --set ebpf.enabled=true{flags}`; +const EBPF_NETWORK_DOC_URL = + "https://github.com/openobserve/openobserve-helm-chart/blob/main/charts/openobserve-collector/docs/metrics.md#13-ebpf-network-flows"; + +// Network flows arrive in chart 0.8.0. Reusing values keeps any trace selectors set above. +const EBPF_NETWORK_UPGRADE = `helm repo update openobserve +helm --namespace openobserve-collector upgrade o2c openobserve/openobserve-collector \\ + --version '>=0.8.0' --reset-then-reuse-values \\ + --set ebpf.enabled=true \\ + --set ebpf.network.enabled=true`; + const instrumentVariants = (t: TranslateFn): RichCardStepVariant[] => LANGUAGES.map((l) => ({ id: l.id, @@ -268,6 +278,15 @@ export default function kubernetesCard(subs: CardSubstitutions, t: TranslateFn): completeOn: "copy", variants: instrumentChoices(t), }, + { + id: "network", + titleKey: "ingestion.setupCard.k8sNetworkFlowsTitle", + descriptionKey: "ingestion.setupCard.k8sNetworkFlowsDesc", + chip: { kind: "editor", labelKey: "ingestion.setupCard.optionalLabel" }, + completeOn: "copy", + code: { lang: "bash", raw: EBPF_NETWORK_UPGRADE }, + note: t("ingestion.setupCard.k8sNetworkFlowsNote"), + }, { id: "verify", titleKey: "ingestion.setupCard.verifyDataTitle", @@ -348,6 +367,7 @@ kubectl get pods -n openobserve-collector`, url: "https://github.com/open-telemetry/opentelemetry-operator", }, { label: t("ingestion.setupCard.ebpfTracingGuideLink"), url: EBPF_DOC_URL }, + { label: t("ingestion.setupCard.ebpfNetworkFlowsGuideLink"), url: EBPF_NETWORK_DOC_URL }, ], }; } diff --git a/web/src/components/ingestion/setupCard/content/linux.ts b/web/src/components/ingestion/setupCard/content/linux.ts index 468260439d..8910af9fc4 100644 --- a/web/src/components/ingestion/setupCard/content/linux.ts +++ b/web/src/components/ingestion/setupCard/content/linux.ts @@ -34,6 +34,8 @@ const install = `curl -O ${AGENTS_REPO}/linux/install.sh \\ && sudo ./install.sh {url}/api/{org}/ {token}`; const TRACING_DOC_URL = "https://github.com/openobserve/agents/blob/main/obi.md"; +const NETWORK_FLOWS_DOC_URL = + "https://github.com/openobserve/agents/blob/main/obi.md#network-flows"; export default function linuxCard(subs: CardSubstitutions, t: TranslateFn): RichCardContent { return { @@ -108,6 +110,37 @@ export default function linuxCard(subs: CardSubstitutions, t: TranslateFn): Rich ], note: t("ingestion.setupCard.anyLinuxHostNote"), }, + { + id: "network", + titleKey: "ingestion.setupCard.linuxNetworkFlowsTitle", + descriptionKey: "ingestion.setupCard.linuxNetworkFlowsDesc", + chip: { kind: "editor", labelKey: "ingestion.setupCard.optionalLabel" }, + completeOn: "copy", + // One untoggled variant: only variants get {flags} and requireOneOf. + variantToggle: false, + variants: [ + { + id: "network-flows", + code: agentCode(`${install}{flags}`, subs, "bash"), + inputs: [ + { + id: "networkFlows", + flag: "--network-flows", + labelKey: "ingestion.setupCard.networkFlowsLabel", + default: "", + placeholder: raw("10.0.1.5/32=web1,10.0.2.0/24=db,0.0.0.0/0=other"), + helpKey: "ingestion.setupCard.networkFlowsHelp", + }, + ], + // The installer requires a value: the CIDRs are what give flows an identity. + requireOneOf: { + ids: ["networkFlows"], + messageKey: "ingestion.setupCard.networkFlowsRequired", + }, + note: t("ingestion.setupCard.linuxNetworkFlowsNote"), + }, + ], + }, { id: "verify", titleKey: "ingestion.setupCard.verifyDataTitle", @@ -139,7 +172,10 @@ sudo journalctl -u openobserve-agent -n 50 --no-pager`, uninstall: agentUninstall("linux"), }, docUrl: "https://github.com/openobserve/agents", - // Step notes can't link externally, so the tracing guide is a footer link. - docLinks: [{ label: t("ingestion.setupCard.ebpfTracingGuideLink"), url: TRACING_DOC_URL }], + // Step notes can't link externally, so the eBPF guides are footer links. + docLinks: [ + { label: t("ingestion.setupCard.ebpfTracingGuideLink"), url: TRACING_DOC_URL }, + { label: t("ingestion.setupCard.ebpfNetworkFlowsGuideLink"), url: NETWORK_FLOWS_DOC_URL }, + ], }; } diff --git a/web/src/components/ingestion/setupCard/content/osAgent.spec.ts b/web/src/components/ingestion/setupCard/content/osAgent.spec.ts index bd6bfd93f1..62e8e398ec 100644 --- a/web/src/components/ingestion/setupCard/content/osAgent.spec.ts +++ b/web/src/components/ingestion/setupCard/content/osAgent.spec.ts @@ -60,13 +60,14 @@ describe("dashboardReadyStep", () => { describe("host agent cards include the dashboard step", () => { it.each([ - ["linux", linuxCard], - ["windows", windowsCard], - ["macos", macosCard], - ])("%s card ends install → verify → dashboard", (_name, builder: any) => { + // Linux has an optional network flows step between install and verify. + ["linux", linuxCard, ["install", "network", "verify", "dashboard"]], + ["windows", windowsCard, ["install", "verify", "dashboard"]], + ["macos", macosCard, ["install", "verify", "dashboard"]], + ])("%s card ends verify → dashboard", (_name, builder: any, ids: string[]) => { const card = builder(SUBS, gt); - expect(card.steps.map((s: any) => s.id)).toEqual(["install", "verify", "dashboard"]); - expect(card.steps[2].completeOn).toBe("detect"); + expect(card.steps.map((s: any) => s.id)).toEqual(ids); + expect(card.steps.at(-1).completeOn).toBe("detect"); }); }); diff --git a/web/src/components/queryHistory/RecentSearchesButton.spec.ts b/web/src/components/queryHistory/RecentSearchesButton.spec.ts new file mode 100644 index 0000000000..1d3d21a732 --- /dev/null +++ b/web/src/components/queryHistory/RecentSearchesButton.spec.ts @@ -0,0 +1,37 @@ +// Copyright 2026 OpenObserve Inc. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +import { describe, expect, it, vi } from "vitest"; +import { mount } from "@vue/test-utils"; +import store from "@/test/unit/helpers/store"; + +vi.mock("@/services/query_history", () => ({ + default: { list: vi.fn().mockResolvedValue({ data: [] }) }, +})); + +import RecentSearchesButton from "./RecentSearchesButton.vue"; + +describe("RecentSearchesButton", () => { + it("puts a caller's class on the clock button, since the popover renders no element", () => { + const wrapper = mount(RecentSearchesButton, { + props: { source: "logs", dataTest: "clock" }, + attrs: { class: "me-1" }, + global: { plugins: [store], stubs: { RecentSearchesPopover: true, OTooltip: true } }, + }); + const button = wrapper.find('[data-test="clock"]'); + expect(button.exists()).toBe(true); + expect(button.classes()).toContain("me-1"); + }); +}); diff --git a/web/src/components/queryHistory/RecentSearchesButton.vue b/web/src/components/queryHistory/RecentSearchesButton.vue new file mode 100644 index 0000000000..22e8f778ff --- /dev/null +++ b/web/src/components/queryHistory/RecentSearchesButton.vue @@ -0,0 +1,112 @@ + + + + + diff --git a/web/src/components/queryHistory/RecentSearchesDayPicker.spec.ts b/web/src/components/queryHistory/RecentSearchesDayPicker.spec.ts new file mode 100644 index 0000000000..721ae41c4f --- /dev/null +++ b/web/src/components/queryHistory/RecentSearchesDayPicker.spec.ts @@ -0,0 +1,129 @@ +// Copyright 2026 OpenObserve Inc. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +import { afterEach, describe, expect, it, vi } from "vitest"; +import { enableAutoUnmount, flushPromises, mount } from "@vue/test-utils"; +import { DateTime } from "luxon"; +import i18n from "@/locales"; +import RecentSearchesDayPicker from "./RecentSearchesDayPicker.vue"; + +enableAutoUnmount(afterEach); + +const mountPicker = (modelValue?: string, zone?: string) => + mount(RecentSearchesDayPicker, { + props: { + modelValue, + zone, + placeholder: "dd/mm/yyyy", + fieldLabel: "Filter by date run", + dataTest: "ran-on", + }, + attachTo: document.body, + global: { plugins: [i18n] }, + }); + +const cell = (iso: string) => + document.querySelector(`[data-test="daterangecalendar-cell-${iso}"]`); + +const openCalendar = async (wrapper: ReturnType) => { + await wrapper.find('[data-test="ran-on-trigger"]').trigger("click"); + await flushPromises(); +}; + +describe("RecentSearchesDayPicker", () => { + it("reads as a date field: the dd/mm/yyyy placeholder, then the chosen day in that format", () => { + const empty = mountPicker(); + const emptyValue = empty.find('[data-test="ran-on-value"]'); + expect(emptyValue.text()).toBe("dd/mm/yyyy"); + expect(emptyValue.classes()).toContain("text-input-placeholder"); + expect(empty.find('[data-test="ran-on-trigger"]').attributes("aria-label")).toBe( + "Filter by date run", + ); + const picked = mountPicker("2026-10-29").find('[data-test="ran-on-value"]'); + expect(picked.text()).toBe("29/10/2026"); + expect(picked.classes()).toContain("text-input-text"); + }); + + it("opens the shared range calendar and picks one day with a single click", async () => { + const wrapper = mountPicker(); + await openCalendar(wrapper); + expect(document.querySelector('[data-test="daterangecalendar-root"]')).not.toBeNull(); + const today = DateTime.local().toISODate()!; + cell(today)!.click(); + await flushPromises(); + expect(wrapper.emitted("update:modelValue")?.at(-1)).toEqual([today]); + expect(document.querySelector('[data-test="ran-on-calendar"]')).toBeNull(); + }); + + it("marks the chosen day selected and keeps days after today unavailable", async () => { + const today = DateTime.local(); + const wrapper = mountPicker(today.toISODate()!); + await openCalendar(wrapper); + expect(cell(today.toISODate()!)?.hasAttribute("data-selected")).toBe(true); + const later = [ + ...document.querySelectorAll('[data-test^="daterangecalendar-cell-"]'), + ].filter((el) => el.dataset.test!.replace("daterangecalendar-cell-", "") > today.toISODate()!); + later.forEach((el) => expect(el.hasAttribute("data-disabled")).toBe(true)); + }); + + it("offers the new day after midnight without a remount", async () => { + vi.useFakeTimers({ toFake: ["Date"] }); + try { + vi.setSystemTime(new Date(2026, 9, 28, 23, 59)); + const wrapper = mountPicker(); + await openCalendar(wrapper); + expect(cell("2026-10-29")?.hasAttribute("data-disabled")).toBe(true); + await wrapper.find('[data-test="ran-on-trigger"]').trigger("click"); + await flushPromises(); + vi.setSystemTime(new Date(2026, 9, 29, 0, 1)); + await openCalendar(wrapper); + expect(cell("2026-10-29")?.hasAttribute("data-disabled")).toBe(false); + } finally { + vi.useRealTimers(); + } + }); + + it("ends the pickable days at today in the given timezone", async () => { + vi.useFakeTimers({ toFake: ["Date"] }); + try { + vi.setSystemTime(new Date(Date.UTC(2026, 9, 28, 20, 0))); + await openCalendar(mountPicker(undefined, "Asia/Kolkata")); + expect(cell("2026-10-29")?.hasAttribute("data-disabled")).toBe(false); + expect(cell("2026-10-30")?.hasAttribute("data-disabled")).toBe(true); + } finally { + vi.useRealTimers(); + } + }); + + it("joins the field and its clear button in a button group, without overriding the buttons' box", () => { + const wrapper = mountPicker("2026-10-27"); + const group = wrapper.find('[data-test="ran-on"]'); + expect(group.attributes("role")).toBe("group"); + const buttons = group.findAll(":scope > button"); + expect(buttons.map((b) => b.attributes("data-test"))).toEqual([ + "ran-on-trigger", + "ran-on-clear", + ]); + const forced = /^(h|rounded|border)(-\S*)?!$/; + buttons.forEach((b) => expect(b.classes().filter((c) => forced.test(c))).toEqual([])); + }); + + it("clears the day with its own button, shown only while a day is set", async () => { + expect(mountPicker().find('[data-test="ran-on-clear"]').exists()).toBe(false); + const wrapper = mountPicker("2026-10-27"); + await wrapper.find('[data-test="ran-on-clear"]').trigger("click"); + expect(wrapper.emitted("update:modelValue")?.at(-1)).toEqual([undefined]); + }); +}); diff --git a/web/src/components/queryHistory/RecentSearchesDayPicker.vue b/web/src/components/queryHistory/RecentSearchesDayPicker.vue new file mode 100644 index 0000000000..aa65d115d0 --- /dev/null +++ b/web/src/components/queryHistory/RecentSearchesDayPicker.vue @@ -0,0 +1,136 @@ + + + + + diff --git a/web/src/components/queryHistory/RecentSearchesDetails.vue b/web/src/components/queryHistory/RecentSearchesDetails.vue new file mode 100644 index 0000000000..cff8aafe12 --- /dev/null +++ b/web/src/components/queryHistory/RecentSearchesDetails.vue @@ -0,0 +1,321 @@ + + + + + diff --git a/web/src/components/queryHistory/RecentSearchesDrawer.spec.ts b/web/src/components/queryHistory/RecentSearchesDrawer.spec.ts new file mode 100644 index 0000000000..5e126292b8 --- /dev/null +++ b/web/src/components/queryHistory/RecentSearchesDrawer.spec.ts @@ -0,0 +1,720 @@ +// Copyright 2026 OpenObserve Inc. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { enableAutoUnmount, flushPromises, mount } from "@vue/test-utils"; +import { nextTick } from "vue"; +import store from "@/test/unit/helpers/store"; +import type { QueryHistoryEntry } from "@/services/query_history"; +import { b64EncodeUnicode } from "@/utils/formatters"; + +const api = vi.hoisted(() => ({ + list: vi.fn(), + record: vi.fn(), + star: vi.fn(), + remove: vi.fn(), + clear: vi.fn(), + count: vi.fn(), +})); +vi.mock("@/services/query_history", () => ({ default: api })); + +const clipboard = vi.hoisted(() => ({ copyToClipboard: vi.fn() })); +vi.mock("@/utils/clipboard", () => clipboard); + +const push = vi.hoisted(() => vi.fn()); +vi.mock("vue-router", () => ({ + useRouter: () => ({ + push, + resolve: (to: { path: string; query: Record }) => ({ + href: `/web${to.path}?${new URLSearchParams(to.query).toString()}`, + }), + }), +})); + +import i18n from "@/locales"; +import RecentSearchesDrawer from "./RecentSearchesDrawer.vue"; +import { dismissAll, pauseTimer, toastRecords } from "@/lib/feedback/Toast/useToast"; + +enableAutoUnmount(afterEach); + +const NOW_US = Date.now() * 1000; +const DAY_US = 86_400_000_000; + +const row = (id: string, overrides: Partial = {}): QueryHistoryEntry => ({ + id, + source: "logs", + query: `query ${id}`, + query_key: "k", + streams: ["k8s"], + context: { time: { type: "relative", period: "15m", start_time: 1, end_time: 2 } }, + status: "ok", + starred: false, + run_count: 1, + created_at: NOW_US, + updated_at: NOW_US, + ...overrides, +}); + +const input = (name: string) => ({ + props: ["modelValue"], + emits: ["update:modelValue"], + template: ``, +}); + +const mountDrawer = (open = true) => + mount(RecentSearchesDrawer, { + props: { source: "logs", open }, + global: { + plugins: [store], + stubs: { + ODrawer: { + props: ["open"], + template: `
`, + }, + ODialog: { + props: ["open"], + emits: ["click:primary", "click:secondary"], + template: `
+
`, + }, + // The real menu is portalled, so its clicks never reach the row. + ODropdown: { template: `
` }, + ODropdownItem: { + emits: ["select"], + template: ``, + }, + OSearchInput: input("search"), + OSelect: input("select"), + RecentSearchesDayPicker: input("date"), + OSwitch: { + props: ["modelValue"], + emits: ["update:modelValue"], + template: ``, + }, + OCheckbox: { + props: ["modelValue"], + emits: ["update:modelValue"], + template: ``, + }, + OToggleGroup: { + props: ["modelValue"], + emits: ["update:modelValue"], + template: `
`, + }, + OToggleGroupItem: { template: "" }, + OTimeCell: true, + OTooltip: true, + }, + }, + }); + +const lastParams = () => api.list.mock.calls.at(-1)?.[1]; + +describe("RecentSearchesDrawer", () => { + beforeEach(() => { + vi.clearAllMocks(); + api.list.mockResolvedValue({ + data: [row("today"), row("old", { updated_at: NOW_US - 3 * DAY_US, run_count: 4 })], + }); + api.star.mockResolvedValue({ data: {} }); + api.remove.mockResolvedValue({ data: {} }); + api.clear.mockResolvedValue({ data: { deleted: 2 } }); + }); + afterEach(() => { + store.state.zoConfig.query_history_retention_days = undefined; + }); + + it("lists this page's rows grouped by day", async () => { + const wrapper = mountDrawer(); + await flushPromises(); + expect(lastParams()).toEqual({ source: "logs", limit: 50, offset: 0 }); + const days = wrapper.findAll('[data-test="recent-searches-day"]'); + expect(days).toHaveLength(2); + expect(days[0].text()).toContain("Today"); + expect(days[1].text()).toContain("ran 4x"); + }); + + it("sends every filter to the server", async () => { + const wrapper = mountDrawer(); + await flushPromises(); + await wrapper.find('[data-stub="search"]').setValue("checkout"); + const selects = wrapper.findAll('[data-stub="select"]'); + await selects[0].setValue("k8s"); + await selects[1].setValue("error"); + await wrapper.find('[data-stub="date"]').setValue("2026-10-27"); + await wrapper.find('[data-test="recent-searches-starred-only"]').setValue(true); + await flushPromises(); + const bounds = { + ran_from: new Date(2026, 9, 27).getTime() * 1000, + ran_to: new Date(2026, 9, 28).getTime() * 1000 - 1, + }; + expect(lastParams()).toEqual({ + source: "logs", + q: "checkout", + stream: "k8s", + status: "error", + starred: true, + ...bounds, + limit: 50, + offset: 0, + }); + + await wrapper.find('[data-test="scope-all"]').trigger("click"); + await flushPromises(); + expect(lastParams().source).toBe("logs,traces,metrics"); + }); + + describe("Reset filters", () => { + const RESET = '[data-test="recent-searches-reset-filters"]'; + // Opened already filtered, so the unfiltered list is never cached and Reset must fetch it. + const mountFiltered = async () => { + const wrapper = mountDrawer(false); + wrapper.vm.show("error"); + await wrapper.setProps({ open: true }); + await flushPromises(); + await wrapper.find('[data-stub="search"]').setValue("checkout"); + await wrapper.findAll('[data-stub="select"]')[0].setValue("k8s"); + await wrapper.find('[data-stub="date"]').setValue("2026-10-27"); + await wrapper.find('[data-test="recent-searches-starred-only"]').setValue(true); + await flushPromises(); + expect(api.list.mock.calls.every(([, params]) => params.status === "error")).toBe(true); + return wrapper; + }; + const expectCleared = (wrapper: ReturnType) => { + expect((wrapper.find('[data-stub="search"]').element as HTMLInputElement).value).toBe(""); + const selects = wrapper.findAll('[data-stub="select"]'); + expect((selects[0].element as HTMLInputElement).value).toBe(""); + expect((selects[1].element as HTMLInputElement).value).toBe("all"); + expect((wrapper.find('[data-stub="date"]').element as HTMLInputElement).value).toBe(""); + expect( + (wrapper.find('[data-test="recent-searches-starred-only"]').element as HTMLInputElement) + .checked, + ).toBe(false); + expect(lastParams()).toEqual({ source: "logs", limit: 50, offset: 0 }); + expect(wrapper.find(RESET).exists()).toBe(false); + }; + + it("is hidden with no filter and shows once any one filter is set", async () => { + const wrapper = mountDrawer(); + await flushPromises(); + expect(wrapper.find(RESET).exists()).toBe(false); + await wrapper.find('[data-stub="date"]').setValue("2026-10-27"); + expect(wrapper.find(RESET).exists()).toBe(true); + await wrapper.find('[data-stub="date"]').setValue(""); + await wrapper.findAll('[data-stub="select"]')[1].setValue("ok"); + expect(wrapper.find(RESET).exists()).toBe(true); + }); + + it("clears all five filters and refetches unfiltered", async () => { + const wrapper = await mountFiltered(); + await wrapper.find(RESET).trigger("click"); + await flushPromises(); + expectCleared(wrapper); + }); + + it("does the same from the filtered empty state", async () => { + api.list.mockResolvedValue({ data: [] }); + const wrapper = await mountFiltered(); + const empty = wrapper.find('[data-test="recent-searches-drawer-empty"]'); + expect(empty.text()).toContain("No searches match these filters"); + const button = empty.findAll("button").filter((b) => b.text().includes("Reset filters")); + expect(button).toHaveLength(1); + await button[0].trigger("click"); + await flushPromises(); + expectCleared(wrapper); + }); + }); + + it("re-runs now on a row click from today, and at original time on demand", async () => { + const wrapper = mountDrawer(); + await flushPromises(); + await wrapper.find('[data-test="recent-searches-drawer-row-today"]').trigger("click"); + expect(wrapper.emitted("restore")?.[0]).toEqual([ + expect.objectContaining({ id: "today" }), + "now", + true, + ]); + expect(wrapper.emitted("update:open")?.at(-1)).toEqual([false]); + await wrapper.setProps({ open: false }); + await wrapper.setProps({ open: true }); + await wrapper.find('[data-test="recent-searches-rerun-original-old"]').trigger("click"); + expect(wrapper.emitted("restore")?.[1]?.slice(1)).toEqual(["original", true]); + await wrapper.setProps({ open: false }); + await wrapper.setProps({ open: true }); + await wrapper.find('[data-test="recent-searches-load-old"]').trigger("click"); + expect(wrapper.emitted("restore")?.[2]?.slice(1)).toEqual(["original", false]); + }); + + it("stars, deletes and copies without restoring", async () => { + const wrapper = mountDrawer(); + await flushPromises(); + await wrapper.find('[data-test="recent-searches-star-today"]').trigger("click"); + await flushPromises(); + expect(api.star).toHaveBeenCalledWith("default", "today", true); + await wrapper.find('[data-test="recent-searches-copy-query-old"]').trigger("click"); + expect(clipboard.copyToClipboard).toHaveBeenLastCalledWith( + "query old", + expect.any(Function), + expect.any(Object), + ); + await wrapper.find('[data-test="recent-searches-copy-link-today"]').trigger("click"); + const link = clipboard.copyToClipboard.mock.calls.at(-1)?.[0] as string; + expect(link).toContain("/web/logs?"); + expect(link).toContain("from=1"); + expect(link).toContain("to=2"); + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] }); + try { + await wrapper.find('[data-test="recent-searches-delete-old"]').trigger("click"); + await flushPromises(); + expect(wrapper.find('[data-test="recent-searches-drawer-row-old"]').exists()).toBe(false); + expect(api.remove).not.toHaveBeenCalled(); + vi.advanceTimersByTime(6000); + await flushPromises(); + expect(api.remove).toHaveBeenCalledWith("default", "old"); + } finally { + vi.useRealTimers(); + } + expect(wrapper.emitted("restore")).toBeUndefined(); + }); + + it("keeps a row's VRL function: its own copy, a line under the query, and in the cut query's tooltip", async () => { + const vrl = '.severity = if .status >= 500 {\n "critical"\n} else { "info" }'; + api.list.mockResolvedValue({ + data: [ + row("fn", { + query: "SELECT 1", + context: { page: { stream: "k8s", functionContent: b64EncodeUnicode(vrl) ?? "" } }, + }), + row("plain"), + ], + }); + const wrapper = mount(RecentSearchesDrawer, { + props: { source: "logs", open: true }, + global: { + plugins: [store], + stubs: { + ODrawer: { props: ["open"], template: `
` }, + ODialog: true, + OTimeCell: true, + }, + }, + attachTo: document.body, + }); + await flushPromises(); + expect(wrapper.find('[data-test="recent-searches-copy-function-plain"]').exists()).toBe(false); + expect(wrapper.find('[data-test="recent-searches-drawer-function"]').text()).toContain( + "Function (VRL): .severity", + ); + + const cell = wrapper.find('[data-test="recent-searches-drawer-query"]').element; + Object.defineProperty(cell, "scrollWidth", { configurable: true, value: 1200 }); + Object.defineProperty(cell, "clientWidth", { configurable: true, value: 300 }); + vi.useFakeTimers(); + try { + cell.dispatchEvent(new MouseEvent("mouseenter")); + vi.advanceTimersByTime(700); + await nextTick(); + await nextTick(); + const bubble = document.body.querySelector('[data-test="o-tooltip-content"]'); + expect(bubble?.textContent).toContain("SELECT 1"); + expect(bubble?.textContent).toContain(vrl); + } finally { + vi.useRealTimers(); + wrapper.unmount(); + } + }); + + it("copies the function from its own menu item and the query alone from Copy query", async () => { + const vrl = ".a = 1\n.b = 2"; + api.list.mockResolvedValue({ + data: [ + row("fn", { + query: "SELECT 1", + context: { page: { stream: "k8s", functionContent: b64EncodeUnicode(vrl) ?? "" } }, + }), + ], + }); + const wrapper = mountDrawer(); + await flushPromises(); + await wrapper.find('[data-test="recent-searches-copy-function-fn"]').trigger("click"); + expect(clipboard.copyToClipboard).toHaveBeenLastCalledWith(vrl, expect.any(Function), { + successMessage: "Function copied", + }); + await wrapper.find('[data-test="recent-searches-copy-query-fn"]').trigger("click"); + expect(clipboard.copyToClipboard).toHaveBeenLastCalledWith("SELECT 1", expect.any(Function), { + successMessage: "Query copied", + }); + }); + + describe("delete with Undo", () => { + const deleteToast = () => toastRecords.find((r) => r.message.startsWith("Deleted")); + const rowShown = (wrapper: ReturnType) => + wrapper.find('[data-test="recent-searches-drawer-row-old"]').exists(); + const deleteOld = async (wrapper: ReturnType) => { + await wrapper.find('[data-test="recent-searches-delete-old"]').trigger("click"); + await flushPromises(); + }; + + beforeEach(() => { + dismissAll(); + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] }); + }); + afterEach(() => { + vi.useRealTimers(); + dismissAll(); + }); + + it("shows the row again when the server refuses the delete", async () => { + api.remove.mockRejectedValueOnce(new Error("500")); + const wrapper = mountDrawer(); + await flushPromises(); + await deleteOld(wrapper); + expect(rowShown(wrapper)).toBe(false); + vi.advanceTimersByTime(20000); + await flushPromises(); + expect(api.remove).toHaveBeenCalledWith("default", "old"); + expect(rowShown(wrapper)).toBe(true); + }); + + it("brings the row back on Undo and never deletes it", async () => { + const wrapper = mountDrawer(); + await flushPromises(); + await deleteOld(wrapper); + expect(rowShown(wrapper)).toBe(false); + expect(deleteToast()?.message).toBe("Deleted query old"); + + deleteToast()?.action?.handler(); + await flushPromises(); + expect(rowShown(wrapper)).toBe(true); + expect(deleteToast()?.open ?? false).toBe(false); + vi.advanceTimersByTime(20000); + await flushPromises(); + expect(api.remove).not.toHaveBeenCalled(); + }); + + it("deletes only when the toast closes, so a paused toast keeps Undo working", async () => { + const wrapper = mountDrawer(); + await flushPromises(); + await deleteOld(wrapper); + const record = deleteToast(); + pauseTimer(record!.id); + vi.advanceTimersByTime(20000); + await flushPromises(); + expect(api.remove).not.toHaveBeenCalled(); + + record?.action?.handler(); + await flushPromises(); + expect(rowShown(wrapper)).toBe(true); + expect(api.remove).not.toHaveBeenCalled(); + }); + + it("deletes once the toast times out, and Undo then does nothing", async () => { + const wrapper = mountDrawer(); + await flushPromises(); + await deleteOld(wrapper); + const action = deleteToast()?.action; + vi.advanceTimersByTime(6000); + await flushPromises(); + expect(api.remove).toHaveBeenCalledWith("default", "old"); + + action?.handler(); + await flushPromises(); + expect(rowShown(wrapper)).toBe(false); + expect(api.remove).toHaveBeenCalledTimes(1); + }); + + it("deletes in the org the delete started in, after a switch to another org", async () => { + const org = store.state.selectedOrganization; + const wrapper = mountDrawer(); + await flushPromises(); + await deleteOld(wrapper); + store.state.selectedOrganization = { ...org, identifier: "other" }; + try { + vi.advanceTimersByTime(6000); + await flushPromises(); + expect(api.remove).toHaveBeenCalledWith("default", "old"); + expect(api.remove).not.toHaveBeenCalledWith("other", "old"); + } finally { + store.state.selectedOrganization = org; + } + }); + + it("keeps Undo working when the message starts with the lowercase query", async () => { + const setMessage = (deletedQuery: string) => + i18n.global.mergeLocaleMessage("en-us", { recentSearches: { deletedQuery } }); + setMessage("{query} deleted"); + try { + const wrapper = mountDrawer(); + await flushPromises(); + await deleteOld(wrapper); + const record = toastRecords.find((r) => r.open); + expect(record?.message).toBe("Query old deleted"); + record?.action?.handler(); + await flushPromises(); + expect(rowShown(wrapper)).toBe(true); + vi.advanceTimersByTime(20000); + await flushPromises(); + expect(api.remove).not.toHaveBeenCalled(); + } finally { + setMessage("Deleted {query}"); + } + }); + + it("closes the toast before committing when the drawer goes away", async () => { + const wrapper = mountDrawer(); + await flushPromises(); + await deleteOld(wrapper); + const record = deleteToast(); + wrapper.unmount(); + await flushPromises(); + expect(api.remove).toHaveBeenCalledWith("default", "old"); + expect(record?.open).toBe(false); + }); + }); + + it("stars or unstars only the row clicked, by its own flag", async () => { + api.list.mockResolvedValue({ + data: [ + row("today", { query: "same", streams: ["k8s"] }), + row("other", { query: "same", streams: ["other"], starred: true }), + ], + }); + const wrapper = mountDrawer(); + await flushPromises(); + expect(wrapper.find('[data-test="recent-searches-star-today"]').attributes("aria-label")).toBe( + "Star", + ); + await wrapper.find('[data-test="recent-searches-star-today"]').trigger("click"); + await wrapper.find('[data-test="recent-searches-star-other"]').trigger("click"); + await flushPromises(); + expect(api.star.mock.calls).toEqual([ + ["default", "today", true], + ["default", "other", false], + ]); + }); + + it("says search or searches to match each count in the clear dialog", async () => { + api.count.mockImplementation((_org: string, params: { starred: boolean }) => + Promise.resolve({ data: { count: params.starred ? 1 : 1 } }), + ); + const wrapper = mountDrawer(); + await flushPromises(); + await wrapper.find('[data-test="recent-searches-clear-btn"]').trigger("click"); + await flushPromises(); + expect(wrapper.find('[data-test="recent-searches-clear-body"]').text()).toContain( + "your 1 recent Logs search in", + ); + expect( + wrapper.find('[data-test="recent-searches-clear-include-starred"]').attributes("label"), + ).toBe("Also remove my 1 starred search"); + }); + + it("clears this page and keeps starred by default, all pages and starred on request", async () => { + api.count.mockImplementation((_org: string, params: { starred: boolean }) => + Promise.resolve({ data: { count: params.starred ? 3 : 1234 } }), + ); + const wrapper = mountDrawer(); + await flushPromises(); + await wrapper.find('[data-test="recent-searches-clear-btn"]').trigger("click"); + await flushPromises(); + expect(wrapper.find('[data-test="recent-searches-clear-body"]').text()).toContain( + "your 1234 recent Logs searches", + ); + expect(api.count).toHaveBeenCalledWith("default", { source: "logs", starred: false }); + expect(api.count).toHaveBeenCalledWith("default", { source: "logs", starred: true }); + expect(api.list).not.toHaveBeenCalledWith("default", expect.objectContaining({ limit: 200 })); + await wrapper.find('[data-test="dialog-confirm"]').trigger("click"); + await flushPromises(); + expect(api.clear).toHaveBeenLastCalledWith("default", { + source: "logs", + include_starred: false, + }); + + await wrapper.find('[data-test="scope-all"]').trigger("click"); + await wrapper.find('[data-test="recent-searches-clear-btn"]').trigger("click"); + await flushPromises(); + expect(wrapper.find('[data-test="recent-searches-clear-body"]').text()).toContain( + "across Logs, Traces and Metrics", + ); + await wrapper.find('[data-test="recent-searches-clear-include-starred"]').setValue(true); + await wrapper.find('[data-test="dialog-confirm"]').trigger("click"); + await flushPromises(); + expect(api.clear).toHaveBeenLastCalledWith("default", { + source: "logs,traces,metrics", + include_starred: true, + }); + expect(api.count).toHaveBeenCalledWith("default", { + source: "logs,traces,metrics", + starred: false, + }); + }); + + it("puts scope and retention in the info tooltip, from /config or kept until deleted", async () => { + store.state.zoConfig.query_history_retention_days = 30; + const wrapper = mountDrawer(); + await flushPromises(); + expect(wrapper.text()).not.toContain("visible only to you"); + expect(wrapper.find('[data-test="recent-searches-retention"]').exists()).toBe(false); + expect(wrapper.find('[data-test="recent-searches-about"]').attributes("aria-label")).toContain( + "Kept 30 days", + ); + store.state.zoConfig.query_history_retention_days = 0; + await flushPromises(); + expect(wrapper.find('[data-test="recent-searches-about"]').attributes("aria-label")).toContain( + "Kept until deleted", + ); + }); +}); + +describe("Starred only (real checkbox)", () => { + const ROWS = [row("today"), row("starred", { starred: true }), row("other")]; + + beforeEach(() => { + vi.clearAllMocks(); + api.list.mockImplementation((_org: string, params: { starred?: boolean }) => + Promise.resolve({ data: params.starred ? ROWS.filter((r) => r.starred) : ROWS }), + ); + }); + + const mountWithRealCheckbox = () => + mount(RecentSearchesDrawer, { + props: { source: "logs", open: true }, + global: { + plugins: [store], + stubs: { + ODrawer: { + props: ["open"], + template: `
`, + }, + ODialog: true, + OTimeCell: true, + OTooltip: true, + }, + }, + }); + const shownRows = (wrapper: ReturnType) => + wrapper.findAll('[data-test^="recent-searches-drawer-row-"]').length; + const box = (wrapper: ReturnType) => + wrapper.find('[data-test="recent-searches-starred-only"] [role="checkbox"]'); + + it("shows a checked box and only starred rows, also after the drawer is closed and reopened", async () => { + const wrapper = mountWithRealCheckbox(); + await flushPromises(); + expect(shownRows(wrapper)).toBe(3); + expect(box(wrapper).attributes("aria-checked")).toBe("false"); + + await box(wrapper).trigger("click"); + await flushPromises(); + expect(box(wrapper).attributes("aria-checked")).toBe("true"); + expect(api.list).toHaveBeenLastCalledWith( + "default", + expect.objectContaining({ starred: true }), + ); + expect(shownRows(wrapper)).toBe(1); + + await wrapper.setProps({ open: false }); + await wrapper.setProps({ open: true }); + await flushPromises(); + expect(box(wrapper).attributes("aria-checked")).toBe("true"); + expect(shownRows(wrapper)).toBe(1); + + await box(wrapper).trigger("click"); + await flushPromises(); + expect(box(wrapper).attributes("aria-checked")).toBe("false"); + expect(shownRows(wrapper)).toBe(3); + }); + + it("shows a cut query's full multi-line text on hover, line breaks and indentation kept", async () => { + const query = + "SELECT level,\n count(*)\n FROM k8s_logs\n -- errors only\n WHERE level = 'error'"; + api.list.mockResolvedValue({ data: [row("long", { query })] }); + const wrapper = mount(RecentSearchesDrawer, { + props: { source: "logs", open: true }, + global: { + plugins: [store], + stubs: { + ODrawer: { props: ["open"], template: `
` }, + ODialog: true, + OTimeCell: true, + }, + }, + attachTo: document.body, + }); + await flushPromises(); + const cell = wrapper.find('[data-test="recent-searches-drawer-query"]').element; + Object.defineProperty(cell, "scrollWidth", { configurable: true, value: 1200 }); + Object.defineProperty(cell, "clientWidth", { configurable: true, value: 300 }); + vi.useFakeTimers(); + try { + cell.dispatchEvent(new MouseEvent("mouseenter")); + vi.advanceTimersByTime(700); + await nextTick(); + await nextTick(); + const bubble = document.body.querySelector('[data-test="o-tooltip-content"]'); + expect(bubble?.textContent).toContain(query); + expect(bubble?.className).toContain("whitespace-pre-wrap"); + expect(bubble?.className).toContain("font-mono"); + } finally { + vi.useRealTimers(); + wrapper.unmount(); + } + }); + + it("shows a failed row's whole error on hover only when the line is cut", async () => { + const message = `Search field not found: ${"x".repeat(300)}`; + api.list.mockResolvedValue({ + data: [row("bad", { status: "error", context: { error: message } })], + }); + const wrapper = mount(RecentSearchesDrawer, { + props: { source: "logs", open: true }, + global: { + plugins: [store], + stubs: { + ODrawer: { props: ["open"], template: `
` }, + ODialog: true, + OTimeCell: true, + }, + }, + attachTo: document.body, + }); + await flushPromises(); + const line = wrapper.find('[data-test="recent-searches-drawer-error"]').element; + const hover = async () => { + line.dispatchEvent(new MouseEvent("mouseleave")); + await nextTick(); + line.dispatchEvent(new MouseEvent("mouseenter")); + vi.advanceTimersByTime(700); + await nextTick(); + await nextTick(); + }; + const bubble = () => document.body.querySelector('[data-test="o-tooltip-content"]'); + const setWidths = (scrollWidth: number, clientWidth: number) => { + Object.defineProperty(line, "scrollWidth", { configurable: true, value: scrollWidth }); + Object.defineProperty(line, "clientWidth", { configurable: true, value: clientWidth }); + }; + vi.useFakeTimers(); + try { + setWidths(300, 300); + await hover(); + expect(bubble()).toBeNull(); + setWidths(1200, 300); + await hover(); + expect(bubble()?.textContent).toContain(message); + } finally { + vi.useRealTimers(); + wrapper.unmount(); + } + }); +}); diff --git a/web/src/components/queryHistory/RecentSearchesDrawer.timezone.spec.ts b/web/src/components/queryHistory/RecentSearchesDrawer.timezone.spec.ts new file mode 100644 index 0000000000..fe3d7563da --- /dev/null +++ b/web/src/components/queryHistory/RecentSearchesDrawer.timezone.spec.ts @@ -0,0 +1,109 @@ +// Copyright 2026 OpenObserve Inc. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { enableAutoUnmount, flushPromises, mount } from "@vue/test-utils"; +import store from "@/test/unit/helpers/store"; +import type { QueryHistoryEntry } from "@/services/query_history"; + +const api = vi.hoisted(() => ({ + list: vi.fn(), + record: vi.fn(), + star: vi.fn(), + remove: vi.fn(), + clear: vi.fn(), + count: vi.fn(), +})); +vi.mock("@/services/query_history", () => ({ default: api })); +vi.mock("vue-router", () => ({ useRouter: () => ({ push: vi.fn(), resolve: vi.fn() }) })); + +import i18n from "@/locales"; +import RecentSearchesDrawer from "./RecentSearchesDrawer.vue"; + +enableAutoUnmount(afterEach); + +const at = (iso: string) => Date.parse(iso) * 1000; + +const row = (id: string, ranAt: string): QueryHistoryEntry => ({ + id, + source: "logs", + query: `query ${id}`, + query_key: id, + streams: ["k8s"], + context: {}, + status: "ok", + starred: false, + run_count: 1, + created_at: at(ranAt), + updated_at: at(ranAt), +}); + +const mountDrawer = () => + mount(RecentSearchesDrawer, { + props: { source: "logs", open: true }, + global: { + plugins: [store, i18n], + stubs: { + ODrawer: { + props: ["open"], + template: `
`, + }, + RecentSearchesDayPicker: { + props: ["modelValue"], + emits: ["update:modelValue"], + template: ``, + }, + OTimeCell: true, + OTooltip: true, + }, + }, + }); + +describe("RecentSearchesDrawer in the configured timezone", () => { + const original = store.state.timezone; + + beforeEach(() => { + vi.clearAllMocks(); + store.state.timezone = "Asia/Kolkata"; + api.count.mockResolvedValue({ data: { count: 0 } }); + // 20:00 UTC is 01:30 the next day in Kolkata; the test runner's own zone is UTC. + api.list.mockResolvedValue({ + data: [row("late", "2025-03-04T20:00:00Z"), row("noon", "2025-03-04T10:00:00Z")], + }); + }); + afterEach(() => { + store.state.timezone = original; + }); + + it("cuts the day headers in the app timezone, not the browser's", async () => { + const wrapper = mountDrawer(); + await flushPromises(); + const days = wrapper.findAll('[data-test="recent-searches-day"]'); + expect(days).toHaveLength(2); + expect(days[0].text()).toContain("Wed, Mar 5"); + expect(days[1].text()).toContain("Tue, Mar 4"); + }); + + it("asks for a Ran on day's bounds in the app timezone", async () => { + const wrapper = mountDrawer(); + await flushPromises(); + await wrapper.find('[data-stub="date"]').setValue("2025-03-05"); + await flushPromises(); + expect(api.list.mock.calls.at(-1)?.[1]).toMatchObject({ + ran_from: at("2025-03-04T18:30:00Z"), + ran_to: at("2025-03-05T18:30:00Z") - 1, + }); + }); +}); diff --git a/web/src/components/queryHistory/RecentSearchesDrawer.vue b/web/src/components/queryHistory/RecentSearchesDrawer.vue new file mode 100644 index 0000000000..e77c4eed90 --- /dev/null +++ b/web/src/components/queryHistory/RecentSearchesDrawer.vue @@ -0,0 +1,653 @@ + + + + + diff --git a/web/src/components/queryHistory/RecentSearchesFocusDropdown.editorBox.spec.ts b/web/src/components/queryHistory/RecentSearchesFocusDropdown.editorBox.spec.ts new file mode 100644 index 0000000000..46b85ec843 --- /dev/null +++ b/web/src/components/queryHistory/RecentSearchesFocusDropdown.editorBox.spec.ts @@ -0,0 +1,210 @@ +// Copyright 2026 OpenObserve Inc. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { enableAutoUnmount, flushPromises, mount } from "@vue/test-utils"; +import { defineComponent, h, nextTick, ref } from "vue"; +import store from "@/test/unit/helpers/store"; +import i18n from "@/locales"; +import { + routeEditorKeyToRecents, + type RecentsKeyTarget, +} from "@/utils/queryHistory/recentSearches"; + +const api = vi.hoisted(() => ({ list: vi.fn(), record: vi.fn(), star: vi.fn(), remove: vi.fn() })); +vi.mock("@/services/query_history", () => ({ default: api })); +vi.mock("@/composables/useChatHistory", () => ({ + useChatHistory: vi.fn(() => ({ + saveToHistory: vi.fn().mockResolvedValue(1), + loadHistory: vi.fn().mockResolvedValue([]), + loadChat: vi.fn().mockResolvedValue(null), + deleteChatById: vi.fn().mockResolvedValue(true), + clearAllHistory: vi.fn().mockResolvedValue(true), + updateChatTitle: vi.fn().mockResolvedValue(true), + })), +})); + +import QueryEditor from "@/components/QueryEditor.vue"; +import RecentSearchesFocusDropdown from "./RecentSearchesFocusDropdown.vue"; +import { placeCaret } from "@/test/unit/helpers/monacoCaret"; + +enableAutoUnmount(afterEach); + +const NOW_US = Date.now() * 1000; +const generateSql = vi.fn().mockResolvedValue(undefined); + +// Stands in for Monaco: the same `.monaco-editor` surface and caret element the real editor renders. +const MonacoStub = defineComponent({ + setup(_, { expose }) { + expose({ handleGenerateSQL: generateSql, getValue: () => "" }); + return () => + h("div", { class: "monaco-editor" }, [ + h("textarea", { class: "inputarea", "data-test": "monaco-input" }), + h("div", { class: "cursors-layer" }, [ + h("div", { class: "cursor", ref: (el) => placeCaret(el as Element | null) }), + ]), + ]); + }, +}); + +const mountEditorBox = async () => { + const open = ref(false); + const box = ref(null); + const dropdown = ref(null); + const emitted: string[] = []; + const Host = defineComponent({ + setup() { + return () => + h("div", [ + h( + "div", + { + ref: box, + onKeydownCapture: (e: KeyboardEvent) => + open.value && + routeEditorKeyToRecents(e, dropdown.value, () => (open.value = false)), + }, + [ + h(QueryEditor, { + query: "", + nlpMode: true, + editorHeight: "200px", + dataTestPrefix: "logs-search-bar", + }), + ], + ), + h(RecentSearchesFocusDropdown, { + ref: dropdown, + source: "logs", + anchor: box.value, + open: open.value, + "onUpdate:open": (value: boolean) => (open.value = value), + onRun: () => emitted.push("run"), + onLoad: () => emitted.push("load"), + }), + ]); + }, + }); + mount(Host, { + attachTo: document.body, + global: { plugins: [store, i18n], stubs: { CodeQueryEditor: MonacoStub, OTimeCell: true } }, + }); + await flushPromises(); + open.value = true; + await vi.waitFor(() => + expect(document.querySelector('[data-test="recent-searches-row-e0"]')).not.toBeNull(), + ); + const aiInput = document.querySelector( + '[data-test^="logs-search-bar-ai-input-field"] input', + )!; + const monacoInput = document.querySelector('[data-test="monaco-input"]')!; + return { open, emitted, aiInput, monacoInput }; +}; + +const keydown = (target: Element, key: string, shiftKey = false) => { + const event = new KeyboardEvent("keydown", { key, shiftKey, bubbles: true, cancelable: true }); + target.dispatchEvent(event); + return event; +}; + +describe("recent-searches dropdown and the controls that share the editor box", () => { + beforeEach(() => { + vi.clearAllMocks(); + store.state.zoConfig.query_history_enabled = true; + api.list.mockResolvedValue({ + data: [ + { + id: "e0", + source: "logs", + query: "level='error'", + query_key: "k", + streams: ["k8s"], + context: {}, + status: "ok", + starred: false, + run_count: 1, + created_at: NOW_US, + updated_at: NOW_US, + }, + ], + }); + }); + afterEach(() => { + delete store.state.zoConfig.query_history_enabled; + document.body.innerHTML = ""; + }); + + it("leaves Enter, Shift+Enter and ArrowDown in the AI prompt to the prompt while the dropdown is open", async () => { + const { open, emitted, aiInput } = await mountEditorBox(); + expect(aiInput).not.toBeNull(); + aiInput.value = "show errors"; + aiInput.dispatchEvent(new Event("input", { bubbles: true })); + await nextTick(); + + const arrow = keydown(aiInput, "ArrowDown"); + const shiftEnter = keydown(aiInput, "Enter", true); + const enter = keydown(aiInput, "Enter"); + await flushPromises(); + + expect(arrow.defaultPrevented).toBe(false); + expect(shiftEnter.defaultPrevented).toBe(false); + expect(enter.defaultPrevented).toBe(false); + expect(document.activeElement?.getAttribute("data-test") ?? "").not.toMatch( + /^recent-searches-row/, + ); + expect(emitted).toEqual([]); + expect(generateSql).toHaveBeenCalledWith("show errors", expect.anything(), expect.anything()); + expect(open.value).toBe(true); + }); + + it("leaves Enter to the editor's newline until a row is hovered, then runs that row", async () => { + const { emitted, monacoInput } = await mountEditorBox(); + const plain = keydown(monacoInput, "Enter"); + await flushPromises(); + expect(plain.defaultPrevented).toBe(false); + expect(emitted).toEqual([]); + + document + .querySelector('[data-test="recent-searches-row-e0"]')! + .dispatchEvent(new PointerEvent("pointerenter")); + await flushPromises(); + const enter = keydown(monacoInput, "Enter"); + await flushPromises(); + expect(enter.defaultPrevented).toBe(true); + expect(emitted).toEqual(["run"]); + }); + + it("closes when focus or a click moves to the AI prompt or a button in the same box", async () => { + const first = await mountEditorBox(); + first.aiInput.focus(); + await nextTick(); + expect(first.open.value).toBe(false); + + document.body.innerHTML = ""; + const second = await mountEditorBox(); + const send = document.querySelector('[data-test="logs-search-bar-ai-send-btn"]')!; + send.dispatchEvent(new MouseEvent("pointerdown", { bubbles: true })); + await nextTick(); + expect(second.open.value).toBe(false); + }); + + it("stays open on a click or focus inside the editor's own surface", async () => { + const { open, monacoInput } = await mountEditorBox(); + monacoInput.dispatchEvent(new MouseEvent("pointerdown", { bubbles: true })); + monacoInput.focus(); + await nextTick(); + expect(open.value).toBe(true); + }); +}); diff --git a/web/src/components/queryHistory/RecentSearchesFocusDropdown.spec.ts b/web/src/components/queryHistory/RecentSearchesFocusDropdown.spec.ts new file mode 100644 index 0000000000..f058d617f4 --- /dev/null +++ b/web/src/components/queryHistory/RecentSearchesFocusDropdown.spec.ts @@ -0,0 +1,376 @@ +// Copyright 2026 OpenObserve Inc. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { enableAutoUnmount, mount } from "@vue/test-utils"; +import { defineComponent, h, nextTick } from "vue"; +import store from "@/test/unit/helpers/store"; +import { FIRST_LINE_CARET, type CaretBox } from "@/test/unit/helpers/monacoCaret"; + +const api = vi.hoisted(() => ({ list: vi.fn(), record: vi.fn(), star: vi.fn(), remove: vi.fn() })); +vi.mock("@/services/query_history", () => ({ default: api })); + +import RecentSearchesFocusDropdown from "./RecentSearchesFocusDropdown.vue"; + +enableAutoUnmount(afterEach); + +const rectOf = + ({ left, top, right, bottom }: CaretBox) => + () => + ({ + left, + top, + right, + bottom, + x: left, + y: top, + width: right - left, + height: bottom - top, + toJSON: () => ({}), + }) as DOMRect; + +// The editor box with Monaco's caret element inside, as the real editor renders it. +const editorWithCaret = (caret: CaretBox) => { + const anchor = document.createElement("div"); + const monaco = document.createElement("div"); + monaco.className = "monaco-editor"; + const layer = document.createElement("div"); + layer.className = "cursors-layer"; + const cursor = document.createElement("div"); + cursor.className = "cursor"; + cursor.getBoundingClientRect = rectOf(caret); + layer.appendChild(cursor); + monaco.appendChild(layer); + anchor.appendChild(monaco); + document.body.appendChild(anchor); + // Monaco moves its caret by rewriting the element's inline style, which is what the panel listens for. + const moveCaret = (to: CaretBox) => { + cursor.getBoundingClientRect = rectOf(to); + cursor.style.top = `${to.top}px`; + cursor.style.left = `${to.left}px`; + }; + // A page scroll or a resize moves the caret on screen with no change inside the editor. + const shiftCaret = (to: CaretBox) => (cursor.getBoundingClientRect = rectOf(to)); + return { anchor, monaco, moveCaret, shiftCaret }; +}; + +const setViewport = (width: number, height: number) => { + Object.defineProperty(window, "innerWidth", { configurable: true, value: width }); + Object.defineProperty(window, "innerHeight", { configurable: true, value: height }); +}; + +const panel = () => + document.querySelector('[data-test="recent-searches-focus-dropdown"]'); + +const px = (value: string) => parseFloat(value); + +const ownsNode = vi.fn((_node: Node | null) => false); +const PopoverStub = defineComponent({ + setup(_, { expose }) { + expose({ + ownsNode, + focusFirst: () => false, + pickActive: () => false, + scrollDetails: () => false, + }); + return () => h("div", { "data-test": "popover-stub" }); + }, +}); + +const mountDropdown = (anchor: HTMLElement | null) => + mount(RecentSearchesFocusDropdown, { + props: { source: "logs", anchor, open: true }, + attachTo: document.body, + global: { plugins: [store], stubs: { RecentSearchesPopover: PopoverStub } }, + }); + +describe("RecentSearchesFocusDropdown placement at the caret", () => { + const settle = () => new Promise((resolve) => setTimeout(resolve, 0)); + + beforeEach(() => { + store.state.zoConfig.query_history_enabled = true; + setViewport(1440, 900); + }); + afterEach(() => { + vi.restoreAllMocks(); + delete store.state.zoConfig.query_history_enabled; + setViewport(1024, 768); + document.body.innerHTML = ""; + }); + + it("opens under the caret line from the caret's x, like the editor's suggest widget", () => { + mountDropdown(editorWithCaret(FIRST_LINE_CARET).anchor); + const el = panel()!; + expect(el.dataset.side).toBe("below"); + expect(px(el.style.top)).toBe(FIRST_LINE_CARET.bottom); + expect(px(el.style.left)).toBe(FIRST_LINE_CARET.left); + expect(el.style.bottom).toBe(""); + expect(px(el.style.maxHeight)).toBe(900 - FIRST_LINE_CARET.bottom - 8); + }); + + it("shifts left to stay on-screen when the caret is near the right edge", async () => { + vi.spyOn(HTMLElement.prototype, "offsetWidth", "get").mockReturnValue(430); + mountDropdown(editorWithCaret({ left: 1300, top: 91, right: 1302, bottom: 109 }).anchor); + window.dispatchEvent(new Event("resize")); + await nextTick(); + expect(px(panel()!.style.left)).toBe(1440 - 430 - 8); + expect(px(panel()!.style.top)).toBe(109); + }); + + it("flips above the caret line only when there is no room below, never covering it", () => { + const caret = { left: 200, top: 800, right: 202, bottom: 818 }; + mountDropdown(editorWithCaret(caret).anchor); + const el = panel()!; + expect(el.dataset.side).toBe("above"); + expect(px(el.style.bottom)).toBe(900 - caret.top); + expect(el.style.top).toBe(""); + expect(px(el.style.left)).toBe(caret.left); + expect(px(el.style.maxHeight)).toBe(caret.top - 8); + }); + + it("stays below with the least room it needs, rather than flipping", () => { + const caret = { left: 200, top: 714, right: 202, bottom: 732 }; + mountDropdown(editorWithCaret(caret).anchor); + expect(panel()!.dataset.side).toBe("below"); + expect(px(panel()!.style.maxHeight)).toBe(160); + }); + + it("stays closed when there is room neither below nor above the caret line", async () => { + setViewport(1440, 200); + const wrapper = mountDropdown( + editorWithCaret({ left: 200, top: 90, right: 202, bottom: 108 }).anchor, + ); + await nextTick(); + expect(panel()).toBeNull(); + expect(wrapper.emitted("update:open")?.at(-1)).toEqual([false]); + }); + + it("stays closed while the editor has no caret on screen", async () => { + const anchor = document.createElement("div"); + document.body.appendChild(anchor); + const wrapper = mountDropdown(anchor); + await nextTick(); + expect(panel()).toBeNull(); + expect(wrapper.emitted("update:open")?.at(-1)).toEqual([false]); + }); + + it("follows the caret as Monaco moves it, without polling", async () => { + const { anchor, moveCaret } = editorWithCaret(FIRST_LINE_CARET); + mountDropdown(anchor); + moveCaret({ left: 420, top: 127, right: 422, bottom: 145 }); + await settle(); + await nextTick(); + expect(px(panel()!.style.top)).toBe(145); + expect(px(panel()!.style.left)).toBe(420); + }); + + it("follows the editor when its page scrolls, and ignores scrolling elsewhere", async () => { + const { anchor, shiftCaret } = editorWithCaret(FIRST_LINE_CARET); + const page = document.createElement("div"); + page.appendChild(anchor); + document.body.appendChild(page); + const elsewhere = document.createElement("div"); + document.body.appendChild(elsewhere); + mountDropdown(anchor); + await settle(); + + shiftCaret({ left: 113, top: 300, right: 115, bottom: 318 }); + elsewhere.dispatchEvent(new Event("scroll")); + await settle(); + await nextTick(); + expect(px(panel()!.style.top)).toBe(109); + + page.dispatchEvent(new Event("scroll")); + await nextTick(); + expect(px(panel()!.style.top)).toBe(318); + }); + + it("follows the editor when a banner mounts above it inside a container that keeps its size", async () => { + const { anchor, shiftCaret } = editorWithCaret(FIRST_LINE_CARET); + const column = document.createElement("div"); + column.appendChild(anchor); + document.body.appendChild(column); + mountDropdown(anchor); + await settle(); + + shiftCaret({ left: 113, top: 141, right: 115, bottom: 159 }); + column.insertBefore(document.createElement("div"), anchor); + await settle(); + expect(px(panel()!.style.top)).toBe(159); + }); + + it("re-places on a synthetic window resize, as Metrics dispatches one after an edit", async () => { + const { anchor, shiftCaret } = editorWithCaret(FIRST_LINE_CARET); + const wrapper = mountDropdown(anchor); + shiftCaret({ left: 461, top: 649, right: 463, bottom: 667 }); + window.dispatchEvent(new Event("resize")); + await nextTick(); + expect(px(panel()!.style.top)).toBe(667); + expect(px(panel()!.style.left)).toBe(461); + expect(wrapper.emitted("update:open")).toBeUndefined(); + }); + + it.each([ + ["above the viewport", { left: 113, top: -60, right: 115, bottom: -42 }], + ["at the viewport top edge", { left: 113, top: -18, right: 115, bottom: 0 }], + ["below the viewport", { left: 113, top: 900, right: 115, bottom: 918 }], + ["far below the viewport", { left: 113, top: 1400, right: 115, bottom: 1418 }], + ])("never opens off-screen with the caret line %s", async (_where, caret) => { + const wrapper = mountDropdown(editorWithCaret(caret).anchor); + await nextTick(); + expect(panel()).toBeNull(); + expect(wrapper.emitted("update:open")?.at(-1)).toEqual([false]); + }); + + it.each([ + ["above", { left: 113, top: -60, right: 115, bottom: -42 }], + ["below", { left: 113, top: 1000, right: 115, bottom: 1018 }], + ])("closes when the page scrolls the caret line %s the viewport", async (_where, caret) => { + const { anchor, moveCaret } = editorWithCaret(FIRST_LINE_CARET); + const wrapper = mountDropdown(anchor); + expect(panel()).not.toBeNull(); + moveCaret(caret); + await settle(); + await nextTick(); + expect(wrapper.emitted("update:open")?.at(-1)).toEqual([false]); + }); + + it("closes once the caret scrolls out of reach", async () => { + const { anchor, moveCaret } = editorWithCaret(FIRST_LINE_CARET); + const wrapper = mountDropdown(anchor); + setViewport(1440, 300); + moveCaret({ left: 113, top: 130, right: 115, bottom: 148 }); + await settle(); + await nextTick(); + expect(wrapper.emitted("update:open")?.at(-1)).toEqual([false]); + }); +}); + +describe("RecentSearchesFocusDropdown outside clicks", () => { + beforeEach(() => { + store.state.zoConfig.query_history_enabled = true; + setViewport(1440, 900); + }); + afterEach(() => { + delete store.state.zoConfig.query_history_enabled; + setViewport(1024, 768); + document.body.innerHTML = ""; + }); + + it("closes on a pointerdown elsewhere and lets that click reach its target", async () => { + const picker = document.createElement("button"); + document.body.appendChild(picker); + const onPicker = vi.fn(); + picker.addEventListener("click", onPicker); + const seenPointerdown = vi.fn(); + document.addEventListener("pointerdown", seenPointerdown); + const wrapper = mountDropdown(editorWithCaret(FIRST_LINE_CARET).anchor); + expect(panel()).not.toBeNull(); + + const down = new MouseEvent("pointerdown", { bubbles: true, cancelable: true }); + picker.dispatchEvent(down); + picker.click(); + await nextTick(); + + expect(wrapper.emitted("update:open")?.at(-1)).toEqual([false]); + expect(down.defaultPrevented).toBe(false); + expect(seenPointerdown).toHaveBeenCalledTimes(1); + expect(onPicker).toHaveBeenCalledTimes(1); + document.removeEventListener("pointerdown", seenPointerdown); + }); + + it("stays open on a pointerdown in the list's details pane, which lives outside the panel", async () => { + const pane = document.createElement("div"); + document.body.appendChild(pane); + ownsNode.mockImplementation((node) => node === pane); + const wrapper = mountDropdown(editorWithCaret(FIRST_LINE_CARET).anchor); + pane.dispatchEvent(new MouseEvent("pointerdown", { bubbles: true })); + pane.dispatchEvent(new FocusEvent("focusin", { bubbles: true })); + await nextTick(); + expect(wrapper.emitted("update:open")).toBeUndefined(); + ownsNode.mockReset(); + ownsNode.mockReturnValue(false); + }); + + it("stays open on a pointerdown in the editor or the panel", async () => { + const { anchor, monaco } = editorWithCaret(FIRST_LINE_CARET); + const wrapper = mountDropdown(anchor); + monaco.dispatchEvent(new MouseEvent("pointerdown", { bubbles: true })); + panel()!.dispatchEvent(new MouseEvent("pointerdown", { bubbles: true })); + await nextTick(); + expect(wrapper.emitted("update:open")).toBeUndefined(); + expect(panel()).not.toBeNull(); + }); +}); + +describe("RecentSearchesFocusDropdown when the editor resizes while open", () => { + const observers: { + callback: ResizeObserverCallback; + targets: Element[]; + boxes: (ResizeObserverBoxOptions | undefined)[]; + }[] = []; + const RealResizeObserver = globalThis.ResizeObserver; + + beforeEach(() => { + store.state.zoConfig.query_history_enabled = true; + setViewport(1440, 900); + observers.length = 0; + globalThis.ResizeObserver = class { + private entry: (typeof observers)[number]; + constructor(callback: ResizeObserverCallback) { + this.entry = { callback, targets: [], boxes: [] }; + observers.push(this.entry); + } + observe(target: Element, options?: ResizeObserverOptions) { + this.entry.targets.push(target); + this.entry.boxes.push(options?.box); + } + unobserve() {} + disconnect() { + this.entry.targets = []; + } + } as unknown as typeof ResizeObserver; + }); + afterEach(() => { + globalThis.ResizeObserver = RealResizeObserver; + delete store.state.zoConfig.query_history_enabled; + setViewport(1024, 768); + document.body.innerHTML = ""; + }); + + const fireResize = () => + observers.forEach((o) => { + if (o.targets.length) o.callback([], {} as ResizeObserver); + }); + + it("stays at the caret as the editor re-lays out, and closes once it would go off-screen", async () => { + const { anchor, moveCaret } = editorWithCaret(FIRST_LINE_CARET); + const wrapper = mountDropdown(anchor); + const active = observers.find((o) => o.targets.length); + expect(active?.targets).toEqual([anchor]); + expect(active?.boxes).toEqual(["border-box"]); + + moveCaret({ left: 113, top: 300, right: 115, bottom: 318 }); + fireResize(); + await nextTick(); + expect(px(panel()!.style.top)).toBe(318); + + setViewport(1440, 240); + moveCaret({ left: 113, top: 100, right: 115, bottom: 118 }); + fireResize(); + await nextTick(); + expect(wrapper.emitted("update:open")?.at(-1)).toEqual([false]); + }); +}); diff --git a/web/src/components/queryHistory/RecentSearchesFocusDropdown.vue b/web/src/components/queryHistory/RecentSearchesFocusDropdown.vue new file mode 100644 index 0000000000..ca6e7a1d05 --- /dev/null +++ b/web/src/components/queryHistory/RecentSearchesFocusDropdown.vue @@ -0,0 +1,219 @@ + + + + + diff --git a/web/src/components/queryHistory/RecentSearchesPopover.spec.ts b/web/src/components/queryHistory/RecentSearchesPopover.spec.ts new file mode 100644 index 0000000000..8f55b1b572 --- /dev/null +++ b/web/src/components/queryHistory/RecentSearchesPopover.spec.ts @@ -0,0 +1,1287 @@ +// Copyright 2026 OpenObserve Inc. +// +// This program is free software: you can redistribute it and/or modify +// it under the terms of the GNU Affero General Public License as published by +// the Free Software Foundation, either version 3 of the License, or +// (at your option) any later version. +// +// This program is distributed in the hope that it will be useful +// but WITHOUT ANY WARRANTY; without even the implied warranty of +// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +// GNU Affero General Public License for more details. +// +// You should have received a copy of the GNU Affero General Public License +// along with this program. If not, see . + +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { enableAutoUnmount, flushPromises, mount } from "@vue/test-utils"; +import { nextTick } from "vue"; +import store from "@/test/unit/helpers/store"; +import type { QueryHistoryEntry } from "@/services/query_history"; +import { b64EncodeUnicode } from "@/utils/formatters"; + +const api = vi.hoisted(() => ({ + list: vi.fn(), + count: vi.fn(), + record: vi.fn(), + star: vi.fn(), + remove: vi.fn(), + clear: vi.fn(), +})); +vi.mock("@/services/query_history", () => ({ default: api })); +const clipboard = vi.hoisted(() => ({ copyToClipboard: vi.fn().mockResolvedValue(true) })); +vi.mock("@/utils/clipboard", () => clipboard); + +import RecentSearchesPopover from "./RecentSearchesPopover.vue"; +import RecentSearchesButton from "./RecentSearchesButton.vue"; +import RecentSearchesFocusDropdown from "./RecentSearchesFocusDropdown.vue"; +import { placeCaret } from "@/test/unit/helpers/monacoCaret"; +import { + routeEditorKeyToRecents, + type RecentsKeyTarget, +} from "@/utils/queryHistory/recentSearches"; + +enableAutoUnmount(afterEach); + +const NOW_US = Date.now() * 1000; + +const setWidths = (el: Element, scrollWidth: number, clientWidth: number) => { + Object.defineProperty(el, "scrollWidth", { configurable: true, value: scrollWidth }); + Object.defineProperty(el, "clientWidth", { configurable: true, value: clientWidth }); +}; +const HOUR_US = 3_600_000_000; + +const setViewport = (width: number, height: number) => { + Object.defineProperty(window, "innerWidth", { configurable: true, value: width }); + Object.defineProperty(window, "innerHeight", { configurable: true, value: height }); +}; + +// The focus dropdown opens at Monaco's caret element, so every editor box here carries one. +const editorBox = () => { + const box = document.createElement("div"); + const monaco = document.createElement("div"); + monaco.className = "monaco-editor"; + const layer = document.createElement("div"); + layer.className = "cursors-layer"; + const cursor = document.createElement("div"); + cursor.className = "cursor"; + placeCaret(cursor); + layer.appendChild(cursor); + monaco.appendChild(layer); + box.appendChild(monaco); + document.body.appendChild(box); + return { box, monaco }; +}; + +const row = (i: number, overrides: Partial = {}): QueryHistoryEntry => ({ + id: `e${i}`, + source: "logs", + query: `q${i}`, + query_key: "k", + streams: ["k8s"], + context: {}, + status: "ok", + starred: false, + run_count: 1, + created_at: NOW_US - i * HOUR_US, + updated_at: NOW_US - i * HOUR_US, + ...overrides, +}); + +const mountPopover = (props: Record = {}) => + mount(RecentSearchesPopover, { + props: { source: "logs", active: true, ...props }, + global: { plugins: [store], stubs: { OTimeCell: true, OTooltip: true } }, + }); + +const rowIds = (wrapper: ReturnType) => + wrapper + .findAll('[data-test^="recent-searches-row-"]') + .map((r) => r.attributes("data-test")?.replace("recent-searches-row-", "")); + +describe("RecentSearchesPopover", () => { + beforeEach(() => { + vi.clearAllMocks(); + api.count.mockResolvedValue({ data: { count: 0 } }); + store.state.zoConfig.query_history_enabled = true; + }); + afterEach(() => { + delete store.state.zoConfig.query_history_enabled; + }); + + it("asks for this page's recents and shows 8 rows", async () => { + api.list.mockResolvedValue({ data: Array.from({ length: 12 }, (_, i) => row(i)) }); + const wrapper = mountPopover(); + await flushPromises(); + expect(api.list).toHaveBeenCalledWith("default", { source: "logs", status: "ok", limit: 8 }); + expect(rowIds(wrapper)).toEqual(["e0", "e1", "e2", "e3", "e4", "e5", "e6", "e7"]); + }); + + it("downloads only the 8 successful rows and the newest run, and counts failures on the server", async () => { + api.list.mockImplementation((_org: string, params: { status?: string; limit: number }) => + Promise.resolve({ + data: params.status + ? Array.from({ length: params.limit }, (_, i) => row(i + 1)) + : [row(0, { status: "error" })], + }), + ); + api.count.mockResolvedValue({ data: { count: 4 } }); + const wrapper = mountPopover(); + await flushPromises(); + expect(api.list.mock.calls.map(([, params]) => params)).toEqual([ + { source: "logs", status: "ok", limit: 8 }, + { source: "logs", limit: 1 }, + ]); + expect(api.count).toHaveBeenCalledWith("default", { source: "logs", status: "error" }); + expect(wrapper.find('[data-test="recent-searches-pinned-failure"]').text()).toContain("q0"); + expect(rowIds(wrapper)).toEqual(["e1", "e2", "e3", "e4", "e5", "e6", "e7"]); + expect(wrapper.find('[data-test="recent-searches-view-failed"]').text()).toBe("3 failed"); + }); + + it("fetches nothing while inactive", async () => { + mountPopover({ active: false }); + await flushPromises(); + expect(api.list).not.toHaveBeenCalled(); + expect(api.count).not.toHaveBeenCalled(); + }); + + it("pins the latest failure with its error and Load to fix, and counts older failures", async () => { + api.list.mockResolvedValue({ + data: [ + row(0, { status: "error", context: { error: "syntax error near level" } }), + row(1), + row(2, { status: "error" }), + ], + }); + api.count.mockResolvedValue({ data: { count: 2 } }); + const wrapper = mountPopover(); + await flushPromises(); + const pinned = wrapper.find('[data-test="recent-searches-pinned-failure"]'); + expect(pinned.text()).toContain("q0"); + expect(pinned.text()).toContain("syntax error near level"); + expect(rowIds(wrapper)).toEqual(["e1"]); + + await wrapper.find('[data-test="recent-searches-load-to-fix"]').trigger("click"); + expect(wrapper.emitted("load")?.[0]?.[0]).toMatchObject({ id: "e0" }); + + await wrapper.find('[data-test="recent-searches-view-failed"]').trigger("click"); + expect(wrapper.emitted("view-all")?.[0]).toEqual(["error"]); + }); + + it("keeps the key hint and the privacy/retention note behind the header info icon", async () => { + api.list.mockResolvedValue({ data: [row(0)] }); + store.state.zoConfig.query_history_retention_days = 14; + const wrapper = mountPopover(); + await flushPromises(); + expect(wrapper.find('[data-test="recent-searches-key-hints"]').exists()).toBe(false); + const about = wrapper.find('[data-test="recent-searches-about"]').attributes("aria-label")!; + expect(about).toContain("Shift+Enter load only"); + expect(about).toContain("Enter run"); + expect(about).not.toMatch(/[⇧⇥]|Tab/); + expect(about).toContain("Only your searches in default, visible only to you"); + expect(about).toContain("Kept 14 days"); + expect(wrapper.text()).not.toContain("visible only to you"); + expect(wrapper.find('[data-test="recent-searches-view-all"]').exists()).toBe(true); + delete store.state.zoConfig.query_history_retention_days; + }); + + it.each([ + [1, "Kept 1 day;"], + [7, "Kept 7 days;"], + [0, "Kept until deleted."], + [-1, "Kept until deleted."], + ])("states a retention of %s as %s", async (days, text) => { + api.list.mockResolvedValue({ data: [row(0)] }); + store.state.zoConfig.query_history_retention_days = days; + const wrapper = mountPopover(); + await flushPromises(); + const about = wrapper.find('[data-test="recent-searches-about"]').attributes("aria-label")!; + expect(about).toContain(text); + expect(about).not.toMatch(/\b1 days\b|\b0 days\b/); + delete store.state.zoConfig.query_history_retention_days; + }); + + it("shows the hint and the note in the tooltip on hover", async () => { + api.list.mockResolvedValue({ data: [row(0)] }); + const wrapper = mount(RecentSearchesPopover, { + props: { source: "logs", active: true }, + global: { plugins: [store], stubs: { OTimeCell: true } }, + attachTo: document.body, + }); + await flushPromises(); + vi.useFakeTimers(); + const info = wrapper.find('[data-test="recent-searches-about"]').element; + [info, ...info.querySelectorAll("*")].forEach((el) => + el.dispatchEvent(new MouseEvent("mouseenter")), + ); + vi.advanceTimersByTime(700); + await nextTick(); + await nextTick(); + const bubble = document.body.querySelector('[data-test="o-tooltip-content"]'); + expect(bubble?.textContent).toContain("Esc close"); + expect(bubble?.textContent).toContain("visible only to you"); + vi.useRealTimers(); + wrapper.unmount(); + }); + + describe("the pinned failure's error line", () => { + const message = `Search field not found: Valid fields are ${"f, ".repeat(80)}`; + const mountWithRealTooltip = async () => { + api.list.mockResolvedValue({ + data: [row(0, { status: "error", context: { error: message } })], + }); + const wrapper = mount(RecentSearchesPopover, { + props: { source: "logs", active: true }, + global: { plugins: [store], stubs: { OTimeCell: true } }, + attachTo: document.body, + }); + await flushPromises(); + return wrapper.find('[data-test="recent-searches-pinned-error"]').element as HTMLElement; + }; + const hover = async (el: HTMLElement) => { + el.dispatchEvent(new MouseEvent("mouseenter")); + vi.advanceTimersByTime(700); + await nextTick(); + await nextTick(); + }; + const bubble = () => document.body.querySelector('[data-test="o-tooltip-content"]'); + + afterEach(() => vi.useRealTimers()); + + it("shows the whole message on hover when the line is cut", async () => { + const line = await mountWithRealTooltip(); + vi.useFakeTimers(); + setWidths(line, 900, 300); + await hover(line); + expect(bubble()?.textContent).toContain(message.trim()); + }); + + it("shows no tooltip when the message fits", async () => { + const line = await mountWithRealTooltip(); + vi.useFakeTimers(); + setWidths(line, 300, 300); + await hover(line); + expect(bubble()).toBeNull(); + }); + }); + + it("runs on click and only loads on shift-click", async () => { + api.list.mockResolvedValue({ data: [row(0)] }); + const wrapper = mountPopover(); + await flushPromises(); + await wrapper.find('[data-test="recent-searches-row-e0"]').trigger("click"); + expect(wrapper.emitted("run")?.[0]).toEqual([expect.objectContaining({ id: "e0" }), "now"]); + await wrapper.find('[data-test="recent-searches-row-e0"]').trigger("click", { shiftKey: true }); + expect(wrapper.emitted("load")?.[0]?.[0]).toMatchObject({ id: "e0" }); + }); + + it("runs on Enter, loads on Shift+Enter, and leaves Ctrl/Cmd+Enter to the page's run shortcut", async () => { + api.list.mockResolvedValue({ data: [row(0)] }); + const wrapper = mountPopover(); + await flushPromises(); + const rowBtn = wrapper.find('[data-test="recent-searches-row-e0"]'); + await rowBtn.trigger("keydown", { key: "Enter", ctrlKey: true }); + await rowBtn.trigger("keydown", { key: "Enter", metaKey: true }); + expect(wrapper.emitted("run")).toBeUndefined(); + expect(wrapper.emitted("load")).toBeUndefined(); + await rowBtn.trigger("keydown", { key: "Enter" }); + expect(wrapper.emitted("run")).toHaveLength(1); + await rowBtn.trigger("keydown", { key: "Enter", shiftKey: true }); + expect(wrapper.emitted("load")).toHaveLength(1); + expect(wrapper.emitted("run")).toHaveLength(1); + }); + + it("re-runs a row from an earlier day at its original time", async () => { + api.list.mockResolvedValue({ data: [row(0, { updated_at: NOW_US - 72 * HOUR_US })] }); + const wrapper = mountPopover(); + await flushPromises(); + await wrapper.find('[data-test="recent-searches-row-e0"]').trigger("click"); + expect(wrapper.emitted("run")?.[0]?.[1]).toBe("original"); + }); + + it("shows the first-time empty state with examples", async () => { + api.list.mockResolvedValue({ data: [] }); + const wrapper = mountPopover({ examples: ["match_all('error')", "match_all('timeout')"] }); + await flushPromises(); + expect(wrapper.find('[data-test="recent-searches-empty"]').exists()).toBe(true); + await wrapper.find('[data-test="recent-searches-example-1"]').trigger("click"); + expect(wrapper.emitted("example")?.[0]).toEqual(["match_all('timeout')"]); + }); + + it("offers a retry when the list fails to load", async () => { + api.list.mockRejectedValueOnce(new Error("down")).mockResolvedValue({ data: [row(0)] }); + const wrapper = mountPopover(); + await flushPromises(); + expect(wrapper.find('[data-test="recent-searches-error"]').exists()).toBe(true); + }); +}); + +describe("the editor's suggest widget look", () => { + beforeEach(() => { + vi.clearAllMocks(); + store.state.zoConfig.query_history_enabled = true; + }); + afterEach(() => { + delete store.state.zoConfig.query_history_enabled; + }); + + it("renders rows at the suggest widget's height with an icon, a monospace one-line query and the time at the end", async () => { + api.list.mockResolvedValue({ + data: [row(0, { query: "SELECT *\n FROM k8s" }), row(1, { starred: true })], + }); + const wrapper = mountPopover(); + await flushPromises(); + const first = wrapper.find('[data-test="recent-searches-row-e0"]'); + expect(first.classes()).toEqual(expect.arrayContaining(["h-6", "flex", "items-center"])); + const parts = first.element.children; + expect(parts[0].className).toContain("size-4"); + expect(parts[0].children).toHaveLength(1); + expect(parts[1].className).toContain("font-mono"); + expect(parts[1].textContent?.trim()).toBe("SELECT * FROM k8s"); + expect(parts[parts.length - 1].tagName.toLowerCase()).toBe("o-time-cell-stub"); + expect(parts[parts.length - 1].getAttribute("mode")).toBe("relative"); + expect(wrapper.find('[data-test="recent-searches-list"]').classes()).toEqual( + expect.arrayContaining(["max-h-72", "overflow-auto"]), + ); + }); +}); + +describe("the details pane for the active row", () => { + const QUERY = [ + "SELECT kubernetes_namespace_name,", + " count(*) AS hits", + ' FROM "k8s_logs"', + " -- only errors", + " WHERE level = 'error'", + " GROUP BY 1", + ].join("\n"); + const details = () => + document.querySelector('[data-test="recent-searches-details"]'); + const detailsQuery = () => + document.querySelector('[data-test="recent-searches-details-query"]'); + const rowEl = (id: string) => + document.querySelector(`[data-test="recent-searches-row-${id}"]`)!; + const hover = async (id: string) => { + rowEl(id).dispatchEvent(new PointerEvent("pointerenter")); + await flushPromises(); + }; + const press = (target: Element, key: string) => + target.dispatchEvent(new KeyboardEvent("keydown", { key, bubbles: true })); + + beforeEach(() => { + vi.clearAllMocks(); + store.state.zoConfig.query_history_enabled = true; + setViewport(1440, 900); + api.list.mockResolvedValue({ + data: [ + row(0, { + query: QUERY, + streams: ["k8s_logs"], + run_count: 3, + context: { time: { type: "relative", period: "1h", start_time: 0, end_time: 0 } }, + }), + row(1, { query: "level = 'warn'\n AND code = 500" }), + ], + }); + }); + afterEach(() => { + delete store.state.zoConfig.query_history_enabled; + setViewport(1024, 768); + document.body.innerHTML = ""; + }); + + const mountAttached = async () => { + const wrapper = mount(RecentSearchesPopover, { + props: { source: "logs", active: true }, + global: { plugins: [store], stubs: { OTimeCell: true, OTooltip: true } }, + attachTo: document.body, + }); + await flushPromises(); + return wrapper; + }; + + it("renders nothing until a row is hovered or focused", async () => { + await mountAttached(); + expect(details()).toBeNull(); + }); + + it("shows the hovered row's full query exactly as stored, with its line breaks and comments", async () => { + await mountAttached(); + await hover("e0"); + expect(detailsQuery()!.tagName.toLowerCase()).toBe("pre"); + expect(detailsQuery()!.textContent).toBe(QUERY); + expect(detailsQuery()!.className).toEqual(expect.stringContaining("whitespace-pre")); + expect(detailsQuery()!.className).toEqual(expect.stringContaining("font-mono")); + expect(detailsQuery()!.className).toEqual(expect.stringContaining("overflow-auto")); + const meta = document.querySelector('[data-test="recent-searches-details-meta"]')!.textContent; + expect(meta).toContain("k8s_logs"); + expect(meta).toContain("Last 1h"); + expect(meta).toContain("ran 3x"); + expect(document.querySelectorAll('[data-test="recent-searches-details"]')).toHaveLength(1); + }); + + it("wraps long lines inside the query's own scroll, with the meta and error in an opaque footer outside it", async () => { + const longLine = `GROUP BY ${"k8s_namespace_name, k8s_pod_name, level, status, ".repeat(4)}`; + api.list.mockResolvedValue({ + data: [ + row(0, { + status: "error", + query: `SELECT *\n FROM k8s_logs\n${longLine}\nORDER BY 1`, + context: { error: "Search field not found" }, + }), + ], + }); + await mountAttached(); + document + .querySelector('[data-test="recent-searches-pinned-failure"]')! + .dispatchEvent(new PointerEvent("pointerenter")); + await flushPromises(); + const pane = details()!; + const query = detailsQuery()!; + const footer = document.querySelector( + '[data-test="recent-searches-details-footer"]', + )!; + expect(query.className.split(" ")).toEqual( + expect.arrayContaining([ + "whitespace-pre-wrap", + "wrap-anywhere", + "overflow-auto", + "min-h-0", + "flex-1", + ]), + ); + expect(query.className.split(" ")).not.toContain("whitespace-pre"); + expect(query.textContent).toBe(`SELECT *\n FROM k8s_logs\n${longLine}\nORDER BY 1`); + expect(pane.className.split(" ")).toEqual( + expect.arrayContaining(["flex-col", "overflow-hidden"]), + ); + expect(pane.children).toHaveLength(2); + expect(pane.children[0].contains(query)).toBe(true); + expect(pane.children[1]).toBe(footer); + expect(query.contains(footer)).toBe(false); + expect(footer.className.split(" ")).toEqual( + expect.arrayContaining(["shrink-0", "bg-select-content-bg", "border-t"]), + ); + expect(footer.querySelector('[data-test="recent-searches-details-meta"]')).not.toBeNull(); + expect(footer.querySelector('[data-test="recent-searches-details-error"]')).not.toBeNull(); + }); + + it("shows a failed row's status and error message", async () => { + api.list.mockResolvedValue({ + data: [row(0, { status: "error", query: "level =\n", context: { error: "syntax error" } })], + }); + await mountAttached(); + document + .querySelector('[data-test="recent-searches-pinned-failure"]')! + .dispatchEvent(new PointerEvent("pointerenter")); + await flushPromises(); + expect(detailsQuery()!.textContent).toBe("level =\n"); + expect( + document.querySelector('[data-test="recent-searches-details-meta"]')!.textContent, + ).toContain("Failed"); + expect( + document.querySelector('[data-test="recent-searches-details-error"]')!.textContent?.trim(), + ).toBe("syntax error"); + }); + + it("follows the keyboard-active row on ArrowDown and ArrowUp", async () => { + const wrapper = await mountAttached(); + (wrapper.vm as unknown as { focusFirst: () => boolean }).focusFirst(); + await flushPromises(); + expect(detailsQuery()!.textContent).toBe(QUERY); + + press(document.activeElement!, "ArrowDown"); + await flushPromises(); + expect(document.activeElement).toBe(rowEl("e1")); + expect(detailsQuery()!.textContent).toBe("level = 'warn'\n AND code = 500"); + + press(document.activeElement!, "ArrowUp"); + await flushPromises(); + expect(detailsQuery()!.textContent).toBe(QUERY); + }); + + describe("hover across the gap to the pane", () => { + afterEach(() => vi.useRealTimers()); + const list = () => document.querySelector('[data-test="recent-searches-list"]')!; + const leave = (from: Element, to: Element | null) => + from.dispatchEvent(new PointerEvent("pointerleave", { relatedTarget: to })); + + it("keeps the pane while the pointer crosses the gap and enters it", async () => { + await mountAttached(); + await hover("e0"); + vi.useFakeTimers(); + leave(list(), document.body); + vi.advanceTimersByTime(80); + details()!.dispatchEvent(new PointerEvent("pointerenter")); + vi.advanceTimersByTime(1000); + await nextTick(); + expect(detailsQuery()!.textContent).toBe(QUERY); + }); + + it("keeps the pane when the pointer goes straight from the list into it", async () => { + await mountAttached(); + await hover("e0"); + vi.useFakeTimers(); + leave(list(), details()); + vi.advanceTimersByTime(1000); + await nextTick(); + expect(details()).not.toBeNull(); + }); + + it("drops the pane shortly after the pointer leaves both for elsewhere", async () => { + await mountAttached(); + await hover("e0"); + vi.useFakeTimers(); + leave(list(), document.body); + vi.advanceTimersByTime(80); + await nextTick(); + expect(details()).not.toBeNull(); + vi.advanceTimersByTime(200); + await nextTick(); + expect(details()).toBeNull(); + }); + + it("drops the pane after the pointer leaves it for elsewhere", async () => { + await mountAttached(); + await hover("e0"); + vi.useFakeTimers(); + details()!.dispatchEvent(new PointerEvent("pointerenter")); + leave(details()!, document.body); + vi.advanceTimersByTime(300); + await nextTick(); + expect(details()).toBeNull(); + }); + }); + + describe("placement beside the list", () => { + const placeRoot = async (box: { left: number; top: number; right: number; bottom: number }) => { + await mountAttached(); + const root = document.querySelector('[data-test="recent-searches-popover"]')!; + root.getBoundingClientRect = () => + ({ + ...box, + x: box.left, + y: box.top, + width: box.right - box.left, + height: box.bottom - box.top, + }) as DOMRect; + await hover("e0"); + const style = details()!.style; + return { + left: parseFloat(style.left), + top: parseFloat(style.top), + width: parseFloat(style.width), + maxHeight: parseFloat(style.maxHeight), + }; + }; + + it("docks to the right of the list, top-aligned, with its own capped height", async () => { + const at = await placeRoot({ left: 113, top: 110, right: 543, bottom: 400 }); + expect(at).toEqual({ left: 547, top: 110, width: 400, maxHeight: 320 }); + }); + + it("flips to the left when there is no room on the right", async () => { + const at = await placeRoot({ left: 900, top: 110, right: 1330, bottom: 400 }); + expect(at.left).toBe(900 - 4 - 400); + expect(at.top).toBe(110); + }); + + it.each([ + ["a tall pane rises just enough to stay on-screen", 320, 900 - 8 - 320], + ["a short pane that fits stays level with the list", 73, 668], + ])("beside a list low on the screen, %s", async (_case, height, top) => { + vi.spyOn(HTMLElement.prototype, "offsetHeight", "get").mockReturnValue(height); + try { + const at = await placeRoot({ left: 461, top: 668, right: 891, bottom: 794 }); + expect(at.left).toBe(895); + expect(at.top).toBe(top); + expect(at.maxHeight).toBe(320); + } finally { + vi.restoreAllMocks(); + } + }); + + it("drops below the list when neither side has room, and caps its height to the viewport", async () => { + setViewport(800, 700); + const at = await placeRoot({ left: 200, top: 110, right: 630, bottom: 400 }); + expect(at.left).toBe(200); + expect(at.top).toBe(404); + expect(at.maxHeight).toBe(700 - 404 - 8); + }); + }); +}); + +describe("long queries in the details pane", () => { + const LONG = Array.from({ length: 40 }, (_, i) => + i === 0 ? "SELECT" : i === 39 ? "LIMIT 100" : ` col_${i},`, + ).join("\n"); + const details = () => + document.querySelector('[data-test="recent-searches-details"]'); + const query = () => + document.querySelector('[data-test="recent-searches-details-query"]')!; + const rowEl = (id: string) => + document.querySelector(`[data-test="recent-searches-row-${id}"]`)!; + const hover = async (id: string) => { + rowEl(id).dispatchEvent(new PointerEvent("pointerenter")); + await flushPromises(); + }; + const key = (target: Element, name: string) => { + const event = new KeyboardEvent("keydown", { key: name, bubbles: true, cancelable: true }); + target.dispatchEvent(event); + return event; + }; + const setScroll = (el: HTMLElement, clientHeight: number, scrollHeight: number) => { + Object.defineProperty(el, "clientHeight", { configurable: true, value: clientHeight }); + Object.defineProperty(el, "scrollHeight", { configurable: true, value: scrollHeight }); + }; + const placeRoot = (box: { left: number; top: number; right: number; bottom: number }) => { + const root = document.querySelector('[data-test="recent-searches-popover"]')!; + root.getBoundingClientRect = () => + ({ + ...box, + x: box.left, + y: box.top, + width: box.right - box.left, + height: box.bottom - box.top, + }) as DOMRect; + }; + const mountAttached = async () => { + const wrapper = mount(RecentSearchesPopover, { + props: { source: "logs", active: true }, + global: { plugins: [store], stubs: { OTimeCell: true, OTooltip: true } }, + attachTo: document.body, + }); + await flushPromises(); + return wrapper; + }; + + beforeEach(() => { + vi.clearAllMocks(); + store.state.zoConfig.query_history_enabled = true; + setViewport(1440, 900); + api.list.mockResolvedValue({ + data: [row(0, { query: LONG }), row(1, { query: "level='error'" })], + }); + }); + afterEach(() => { + vi.unstubAllGlobals(); + delete store.state.zoConfig.query_history_enabled; + setViewport(1024, 768); + document.body.innerHTML = ""; + }); + + it("shows the query's line count in the footer, singular for one line", async () => { + await mountAttached(); + await hover("e0"); + const lines = () => + document.querySelector('[data-test="recent-searches-details-lines"]')!.textContent?.trim(); + expect(lines()).toBe("40 lines"); + await hover("e1"); + expect(lines()).toBe("1 line"); + }); + + it("fades the query's bottom edge only while more of it is below", async () => { + await mountAttached(); + await hover("e0"); + const fade = () => document.querySelector('[data-test="recent-searches-details-fade"]'); + setScroll(query(), 280, 640); + query().dispatchEvent(new Event("scroll")); + await nextTick(); + expect(fade()).not.toBeNull(); + query().scrollTop = 360; + query().dispatchEvent(new Event("scroll")); + await nextTick(); + expect(fade()).toBeNull(); + }); + + it("copies the full query with the drawer's copy and toast", async () => { + await mountAttached(); + await hover("e0"); + const copy = document.querySelector('[data-test="recent-searches-details-copy"]')!; + expect(copy.getAttribute("aria-label")).toBe("Copy query"); + copy.click(); + expect(clipboard.copyToClipboard).toHaveBeenCalledWith(LONG, expect.any(Function), { + successMessage: "Query copied", + }); + }); + + it("pages the query with PageDown and PageUp on the active row, while the arrows keep moving rows", async () => { + const wrapper = await mountAttached(); + (wrapper.vm as unknown as { focusFirst: () => boolean }).focusFirst(); + await flushPromises(); + setScroll(query(), 280, 640); + const down = key(document.activeElement!, "PageDown"); + expect(down.defaultPrevented).toBe(true); + expect(query().scrollTop).toBe(264); + key(document.activeElement!, "PageUp"); + expect(query().scrollTop).toBe(0); + expect(document.activeElement).toBe(rowEl("e0")); + key(document.activeElement!, "ArrowDown"); + await flushPromises(); + expect(document.activeElement).toBe(rowEl("e1")); + expect(query().textContent).toBe("level='error'"); + }); + + describe("a keyboard-selected row's pane, reached without hovering the row", () => { + const copyButton = () => + document.querySelector('[data-test="recent-searches-details-copy"]')!; + const focusFirstRow = async (wrapper: Awaited>) => { + (wrapper.vm as unknown as { focusFirst: () => boolean }).focusFirst(); + await flushPromises(); + expect(document.activeElement).toBe(rowEl("e0")); + }; + + it("stays while Tab moves focus to Copy, and Copy then works from the keyboard", async () => { + const wrapper = await mountAttached(); + await focusFirstRow(wrapper); + const tab = key(rowEl("e0"), "Tab"); + await flushPromises(); + expect(tab.defaultPrevented).toBe(true); + expect(document.activeElement).toBe(copyButton()); + expect(query().textContent).toBe(LONG); + copyButton().click(); + expect(clipboard.copyToClipboard).toHaveBeenCalledWith(LONG, expect.any(Function), { + successMessage: "Query copied", + }); + expect(details()).not.toBeNull(); + }); + + it("stays when focus moves straight to Copy, as a click on it does", async () => { + const wrapper = await mountAttached(); + await focusFirstRow(wrapper); + copyButton().focus(); + await flushPromises(); + expect(document.activeElement).toBe(copyButton()); + expect(query().textContent).toBe(LONG); + }); + + it("stays when the pointer enters the pane and the click takes focus nowhere", async () => { + const wrapper = await mountAttached(); + await focusFirstRow(wrapper); + details()!.dispatchEvent(new PointerEvent("pointerenter")); + rowEl("e0").blur(); + await flushPromises(); + expect(query().textContent).toBe(LONG); + copyButton().click(); + expect(clipboard.copyToClipboard).toHaveBeenCalledWith(LONG, expect.any(Function), { + successMessage: "Query copied", + }); + }); + + it("goes back to the row on Shift+Tab from Copy, and closes the list on Escape", async () => { + const wrapper = await mountAttached(); + await focusFirstRow(wrapper); + key(rowEl("e0"), "Tab"); + await flushPromises(); + const back = new KeyboardEvent("keydown", { + key: "Tab", + shiftKey: true, + bubbles: true, + cancelable: true, + }); + copyButton().dispatchEvent(back); + await flushPromises(); + expect(back.defaultPrevented).toBe(true); + expect(document.activeElement).toBe(rowEl("e0")); + key(rowEl("e0"), "Tab"); + await flushPromises(); + key(copyButton(), "Escape"); + expect(wrapper.emitted("close")).toHaveLength(1); + }); + + it("goes away once focus leaves the pane for elsewhere with the pointer outside it", async () => { + const wrapper = await mountAttached(); + await focusFirstRow(wrapper); + key(rowEl("e0"), "Tab"); + await flushPromises(); + const elsewhere = document.createElement("button"); + document.body.appendChild(elsewhere); + elsewhere.focus(); + await flushPromises(); + expect(details()).toBeNull(); + }); + + it("leaves Tab alone when focus is not on a row", async () => { + await mountAttached(); + const viewAll = document.querySelector( + '[data-test="recent-searches-view-all"]', + )!; + const tab = key(viewAll, "Tab"); + expect(tab.defaultPrevented).toBe(false); + }); + }); + + it("shows a row's VRL function in its own block under the query, with its own copy", async () => { + const vrl = '.severity = if .status >= 500 {\n "critical"\n} else {\n "info"\n}'; + api.list.mockResolvedValue({ + data: [ + row(0, { + query: "SELECT 1", + context: { page: { stream: "k8s", functionContent: b64EncodeUnicode(vrl) ?? "" } }, + }), + row(1), + ], + }); + await mountAttached(); + await hover("e0"); + const block = document.querySelector('[data-test="recent-searches-details-function"]')!; + expect(query().contains(block)).toBe(false); + const text = block.querySelector('[data-test="recent-searches-details-function-text"]')!; + expect(text.textContent).toBe(vrl); + expect(text.className.split(" ")).toEqual( + expect.arrayContaining([ + "whitespace-pre-wrap", + "wrap-anywhere", + "font-mono", + "overflow-auto", + ]), + ); + block + .querySelector('[data-test="recent-searches-details-copy-function"]')! + .click(); + expect(clipboard.copyToClipboard).toHaveBeenLastCalledWith(vrl, expect.any(Function), { + successMessage: "Function copied", + }); + document.querySelector('[data-test="recent-searches-details-copy"]')!.click(); + expect(clipboard.copyToClipboard).toHaveBeenLastCalledWith("SELECT 1", expect.any(Function), { + successMessage: "Query copied", + }); + + await hover("e1"); + expect(document.querySelector('[data-test="recent-searches-details-function"]')).toBeNull(); + }); + + it("leaves PageDown to its owner when no row is active", async () => { + const wrapper = await mountAttached(); + const vm = wrapper.vm as unknown as { + handleOutsideKey: (event: KeyboardEvent) => boolean; + scrollDetails: (step: 1 | -1) => boolean; + }; + const event = new KeyboardEvent("keydown", { key: "PageDown", cancelable: true }); + expect(vm.handleOutsideKey(event)).toBe(false); + expect(event.defaultPrevented).toBe(false); + expect(vm.scrollDetails(1)).toBe(false); + }); + + it("widens to 560px for a query over 18 lines when the viewport has room, and stays 400px for short ones", async () => { + await mountAttached(); + placeRoot({ left: 113, top: 110, right: 543, bottom: 400 }); + await hover("e0"); + expect(details()!.style.width).toBe("560px"); + expect(details()!.style.maxHeight).toBe("320px"); + await hover("e1"); + expect(details()!.style.width).toBe("400px"); + }); + + it("widens only into the room the viewport has, keeping the pane on-screen", async () => { + setViewport(1100, 900); + await mountAttached(); + placeRoot({ left: 113, top: 110, right: 543, bottom: 400 }); + await hover("e0"); + expect(details()!.style.left).toBe("547px"); + expect(details()!.style.width).toBe(`${1100 - 8 - 547}px`); + }); + + it("follows the list when it moves while the same row stays active", async () => { + const holder = document.createElement("div"); + document.body.appendChild(holder); + const wrapper = mount(RecentSearchesPopover, { + props: { source: "logs", active: true }, + global: { plugins: [store], stubs: { OTimeCell: true, OTooltip: true } }, + attachTo: holder, + }); + await flushPromises(); + placeRoot({ left: 113, top: 110, right: 543, bottom: 400 }); + (wrapper.vm as unknown as { focusFirst: () => boolean }).focusFirst(); + await flushPromises(); + expect(details()!.style.top).toBe("110px"); + placeRoot({ left: 420, top: 146, right: 850, bottom: 436 }); + holder.style.top = "36px"; + await flushPromises(); + expect(details()!.style.left).toBe("854px"); + expect(details()!.style.top).toBe("146px"); + expect(query().textContent).toBe(LONG); + }); +}); + +describe("when history is disabled", () => { + beforeEach(() => { + vi.clearAllMocks(); + store.state.zoConfig.query_history_enabled = false; + }); + afterEach(() => { + delete store.state.zoConfig.query_history_enabled; + }); + + it("hides the clock button", () => { + const wrapper = mount(RecentSearchesButton, { + props: { source: "logs" }, + global: { plugins: [store] }, + }); + expect(wrapper.find('[data-test="recent-searches-btn"]').exists()).toBe(false); + }); + + it("never opens the focus dropdown and never fetches", async () => { + const wrapper = mount(RecentSearchesFocusDropdown, { + props: { source: "logs", anchor: null, open: true }, + global: { plugins: [store] }, + attachTo: document.body, + }); + await flushPromises(); + expect(document.querySelector('[data-test="recent-searches-focus-dropdown"]')).toBeNull(); + expect(api.list).not.toHaveBeenCalled(); + wrapper.unmount(); + }); +}); + +describe("keyboard: arrows then Enter or Shift+Enter", () => { + const press = (target: Element, key: string, shiftKey = false) => + target.dispatchEvent(new KeyboardEvent("keydown", { key, shiftKey, bubbles: true })); + const activeTest = () => document.activeElement?.getAttribute("data-test"); + + beforeEach(() => { + vi.clearAllMocks(); + store.state.zoConfig.query_history_enabled = true; + api.list.mockResolvedValue({ data: [row(0), row(1)] }); + }); + afterEach(() => { + delete store.state.zoConfig.query_history_enabled; + }); + + const openFromButton = async () => { + const wrapper = mount(RecentSearchesButton, { + props: { source: "logs" }, + global: { plugins: [store], stubs: { OTimeCell: true } }, + attachTo: document.body, + }); + const trigger = wrapper.find('[data-test="recent-searches-btn"]'); + await trigger.trigger("click"); + await flushPromises(); + await vi.waitFor(() => + expect(document.querySelector('[data-test="recent-searches-row-e1"]')).not.toBeNull(), + ); + return { wrapper, trigger: trigger.element as HTMLElement }; + }; + + it("loads the row Shift+Enter is pressed on, even when focus was left on the clock button", async () => { + const { wrapper, trigger } = await openFromButton(); + trigger.focus(); + press(trigger, "ArrowDown"); + press(document.activeElement!, "ArrowDown"); + expect(activeTest()).toBe("recent-searches-row-e1"); + + press(document.activeElement!, "Enter", true); + await flushPromises(); + expect(wrapper.emitted("load")?.[0]?.[0]).toMatchObject({ id: "e1" }); + expect(wrapper.emitted("run")).toBeUndefined(); + }); + + it("gives the keyboard-focused row, or the hovered one, the menu's active state", async () => { + const { trigger } = await openFromButton(); + trigger.focus(); + press(trigger, "ArrowDown"); + press(document.activeElement!, "ArrowDown"); + await nextTick(); + const row = (id: string) => + document.querySelector(`[data-test="recent-searches-row-${id}"]`)!; + expect(row("e1").getAttribute("aria-selected")).toBe("true"); + expect(row("e1").hasAttribute("data-highlighted")).toBe(true); + expect(row("e0").getAttribute("aria-selected")).toBe("false"); + expect(row("e0").hasAttribute("data-highlighted")).toBe(false); + expect(row("e1").className).toContain("data-[highlighted]:bg-select-item-selected-bg"); + expect(row("e1").className).toContain("data-[highlighted]:border-accent"); + + row("e0").dispatchEvent(new PointerEvent("pointerenter")); + await nextTick(); + expect(row("e0").hasAttribute("data-highlighted")).toBe(true); + expect(row("e1").hasAttribute("data-highlighted")).toBe(false); + vi.useFakeTimers(); + try { + document + .querySelector('[data-test="recent-searches-list"]')! + .dispatchEvent(new PointerEvent("pointerleave", { relatedTarget: document.body })); + vi.advanceTimersByTime(300); + await nextTick(); + expect(row("e1").hasAttribute("data-highlighted")).toBe(true); + } finally { + vi.useRealTimers(); + } + }); + + describe("Enter on a hovered row after opening from the clock button", () => { + const hoverRow = async (id: string) => { + document + .querySelector(`[data-test="recent-searches-row-${id}"]`)! + .dispatchEvent(new PointerEvent("pointerenter")); + await flushPromises(); + }; + const enter = (target: Element, shiftKey = false) => { + const event = new KeyboardEvent("keydown", { + key: "Enter", + shiftKey, + bubbles: true, + cancelable: true, + }); + target.dispatchEvent(event); + return event; + }; + + it("focuses the list itself, so Enter runs the hovered row and never reaches the page", async () => { + const pageSaw = vi.fn(); + document.addEventListener("keydown", pageSaw); + try { + const { wrapper } = await openFromButton(); + const list = document.querySelector('[data-test="recent-searches-popover"]')!; + expect(document.activeElement).toBe(list); + await hoverRow("e1"); + const event = enter(list); + await flushPromises(); + expect(event.defaultPrevented).toBe(true); + expect(pageSaw).not.toHaveBeenCalled(); + expect(wrapper.emitted("run")?.[0]?.[0]).toMatchObject({ id: "e1" }); + } finally { + document.removeEventListener("keydown", pageSaw); + } + }); + + it("loads the hovered row on Shift+Enter", async () => { + const { wrapper } = await openFromButton(); + await hoverRow("e1"); + enter(document.querySelector('[data-test="recent-searches-popover"]')!, true); + await flushPromises(); + expect(wrapper.emitted("load")?.[0]?.[0]).toMatchObject({ id: "e1" }); + expect(wrapper.emitted("run")).toBeUndefined(); + }); + + it("does nothing on Enter when no row is hovered", async () => { + const { wrapper } = await openFromButton(); + const event = enter(document.querySelector('[data-test="recent-searches-popover"]')!); + await flushPromises(); + expect(event.defaultPrevented).toBe(false); + expect(wrapper.emitted("run")).toBeUndefined(); + }); + + it("runs the hovered row on Enter even with focus left on the clock button", async () => { + const { wrapper, trigger } = await openFromButton(); + trigger.focus(); + await hoverRow("e1"); + const event = enter(trigger); + await flushPromises(); + expect(event.defaultPrevented).toBe(true); + expect(wrapper.emitted("run")?.[0]?.[0]).toMatchObject({ id: "e1" }); + }); + }); + + it("runs the row plain Enter is pressed on", async () => { + const { wrapper, trigger } = await openFromButton(); + trigger.focus(); + press(trigger, "ArrowDown"); + press(document.activeElement!, "Enter"); + await flushPromises(); + expect(wrapper.emitted("run")?.[0]?.[0]).toMatchObject({ id: "e0" }); + expect(wrapper.emitted("load")).toBeUndefined(); + }); + + it("loads from the editor's focus dropdown after its ArrowDown handoff", async () => { + const wrapper = mount(RecentSearchesFocusDropdown, { + props: { source: "logs", anchor: editorBox().box, open: true }, + global: { plugins: [store], stubs: { OTimeCell: true } }, + attachTo: document.body, + }); + await vi.waitFor(() => + expect(document.querySelector('[data-test="recent-searches-row-e0"]')).not.toBeNull(), + ); + expect((wrapper.vm as unknown as { focusFirst: () => boolean }).focusFirst()).toBe(true); + press(document.activeElement!, "Enter", true); + await flushPromises(); + expect(wrapper.emitted("load")?.[0]?.[0]).toMatchObject({ id: "e0" }); + }); + + const mountWithEditor = async () => { + const { box, monaco } = editorBox(); + const editor = document.createElement("textarea"); + editor.addEventListener("keydown", (e) => { + if (e.key === "Enter") editor.value += "\n"; + }); + monaco.appendChild(editor); + const wrapper = mount(RecentSearchesFocusDropdown, { + props: { source: "logs", anchor: box, open: true }, + global: { plugins: [store], stubs: { OTimeCell: true } }, + attachTo: document.body, + }); + const handle = wrapper.vm as unknown as RecentsKeyTarget; + box.addEventListener( + "keydown", + (e) => routeEditorKeyToRecents(e, handle, () => wrapper.setProps({ open: false })), + true, + ); + await vi.waitFor(() => + expect(document.querySelector('[data-test="recent-searches-row-e0"]')).not.toBeNull(), + ); + editor.focus(); + return { wrapper, editor }; + }; + + const hoverRow = async (id: string) => { + document + .querySelector(`[data-test="recent-searches-row-${id}"]`)! + .dispatchEvent(new PointerEvent("pointerenter")); + await flushPromises(); + }; + + it("leaves Enter and Shift+Enter to the editor when no row is highlighted", async () => { + const { wrapper, editor } = await mountWithEditor(); + press(editor, "Enter", true); + press(editor, "Enter"); + await flushPromises(); + expect(wrapper.emitted("load")).toBeUndefined(); + expect(wrapper.emitted("run")).toBeUndefined(); + expect(editor.value).toBe("\n\n"); + }); + + it("loads the hovered row on Shift+Enter from the editor", async () => { + const { wrapper, editor } = await mountWithEditor(); + await hoverRow("e1"); + press(editor, "Enter", true); + await flushPromises(); + expect(wrapper.emitted("load")?.[0]?.[0]).toMatchObject({ id: "e1" }); + expect(wrapper.emitted("run")).toBeUndefined(); + expect(editor.value).toBe(""); + }); + + it("runs the hovered row on Enter from the editor", async () => { + const { wrapper, editor } = await mountWithEditor(); + await hoverRow("e1"); + press(editor, "Enter"); + await flushPromises(); + expect(wrapper.emitted("run")?.[0]?.[0]).toMatchObject({ id: "e1" }); + expect(editor.value).toBe(""); + }); + + it("pages the hovered row's details on PageDown from the editor, and leaves PageDown to the editor with no row active", async () => { + const { wrapper, editor } = await mountWithEditor(); + const pageDown = () => { + const event = new KeyboardEvent("keydown", { + key: "PageDown", + bubbles: true, + cancelable: true, + }); + editor.dispatchEvent(event); + return event; + }; + expect(pageDown().defaultPrevented).toBe(false); + + document + .querySelector('[data-test="recent-searches-row-e0"]')! + .dispatchEvent(new PointerEvent("pointerenter")); + await flushPromises(); + const query = document.querySelector( + '[data-test="recent-searches-details-query"]', + )!; + Object.defineProperty(query, "clientHeight", { configurable: true, value: 200 }); + Object.defineProperty(query, "scrollHeight", { configurable: true, value: 900 }); + expect(pageDown().defaultPrevented).toBe(true); + expect(query.scrollTop).toBe(184); + expect(document.activeElement).toBe(editor); + expect(wrapper.emitted("update:open")).toBeUndefined(); + }); + + it("keeps the clock popover open for a click in the details pane, and closes it for a click elsewhere", async () => { + const { wrapper } = await openFromButton(); + document + .querySelector('[data-test="recent-searches-row-e0"]')! + .dispatchEvent(new PointerEvent("pointerenter")); + await flushPromises(); + const copy = document.querySelector('[data-test="recent-searches-details-copy"]')!; + copy.dispatchEvent(new PointerEvent("pointerdown", { bubbles: true })); + copy.dispatchEvent(new FocusEvent("focusin", { bubbles: true })); + await flushPromises(); + expect(document.querySelector('[data-test="recent-searches-popover"]')).not.toBeNull(); + + const elsewhere = document.createElement("button"); + document.body.appendChild(elsewhere); + elsewhere.dispatchEvent(new PointerEvent("pointerdown", { bubbles: true })); + await flushPromises(); + expect(document.querySelector('[data-test="recent-searches-popover"]')).toBeNull(); + wrapper.unmount(); + }); + + it("keeps the clock popover open while Tab takes a keyboard-selected row to its details' Copy", async () => { + const { wrapper, trigger } = await openFromButton(); + trigger.focus(); + press(trigger, "ArrowDown"); + await flushPromises(); + expect(activeTest()).toBe("recent-searches-row-e0"); + press(document.activeElement!, "Tab"); + await flushPromises(); + expect(activeTest()).toBe("recent-searches-details-copy"); + expect(document.querySelector('[data-test="recent-searches-popover"]')).not.toBeNull(); + expect(document.querySelector('[data-test="recent-searches-details-query"]')?.textContent).toBe( + "q0", + ); + wrapper.unmount(); + }); + + it("fits a short dropdown: header and footer keep their rows and the list scrolls the rest", async () => { + Object.defineProperty(window, "innerHeight", { configurable: true, value: 300 }); + api.list.mockResolvedValue({ data: Array.from({ length: 8 }, (_, i) => row(i)) }); + try { + const { box } = editorBox(); + const cursor = box.querySelector(".cursor")!; + placeCaret(cursor, { left: 113, top: 91, right: 115, bottom: 109 }); + mount(RecentSearchesFocusDropdown, { + props: { source: "logs", anchor: box, open: true }, + global: { plugins: [store], stubs: { OTimeCell: true } }, + attachTo: document.body, + }); + await vi.waitFor(() => + expect(document.querySelector('[data-test="recent-searches-row-e7"]')).not.toBeNull(), + ); + const panel = document.querySelector( + '[data-test="recent-searches-focus-dropdown"]', + )!; + expect(panel.style.maxHeight).toBe(`${300 - 109 - 8}px`); + expect(panel.className.split(" ")).toEqual( + expect.arrayContaining(["flex", "flex-col", "overflow-hidden"]), + ); + const root = panel.querySelector('[data-test="recent-searches-popover"]')!; + expect(root.className.split(" ")).toEqual( + expect.arrayContaining(["flex", "flex-col", "min-h-0"]), + ); + const list = root.querySelector('[data-test="recent-searches-list"]')!; + expect(list.className.split(" ")).toEqual( + expect.arrayContaining(["min-h-0", "overflow-auto"]), + ); + const [header, , footer] = [...root.children] as HTMLElement[]; + expect(header.className.split(" ")).toContain("shrink-0"); + expect(footer.className.split(" ")).toContain("shrink-0"); + expect(footer.querySelector('[data-test="recent-searches-view-all"]')).not.toBeNull(); + } finally { + Object.defineProperty(window, "innerHeight", { configurable: true, value: 768 }); + } + }); + + it("leaves Enter to the editor when the list has no rows", async () => { + api.list.mockResolvedValue({ data: [] }); + const { box, monaco } = editorBox(); + const editor = document.createElement("textarea"); + editor.addEventListener("keydown", (e) => { + if (e.key === "Enter") editor.value += "\n"; + }); + monaco.appendChild(editor); + const wrapper = mount(RecentSearchesFocusDropdown, { + props: { source: "logs", anchor: box, open: true }, + global: { plugins: [store], stubs: { OTimeCell: true } }, + attachTo: document.body, + }); + box.addEventListener( + "keydown", + (e) => routeEditorKeyToRecents(e, wrapper.vm as unknown as RecentsKeyTarget, () => {}), + true, + ); + await flushPromises(); + press(editor, "Enter", true); + expect(editor.value).toBe("\n"); + expect(wrapper.emitted("load")).toBeUndefined(); + }); +}); diff --git a/web/src/components/queryHistory/RecentSearchesPopover.vue b/web/src/components/queryHistory/RecentSearchesPopover.vue new file mode 100644 index 0000000000..48c103cf2f --- /dev/null +++ b/web/src/components/queryHistory/RecentSearchesPopover.vue @@ -0,0 +1,435 @@ + + + + + diff --git a/web/src/components/rum/VideoPlayer.spec.ts b/web/src/components/rum/VideoPlayer.spec.ts index 46af1d8abf..1705a4f582 100644 --- a/web/src/components/rum/VideoPlayer.spec.ts +++ b/web/src/components/rum/VideoPlayer.spec.ts @@ -1490,6 +1490,301 @@ describe("VideoPlayer", () => { }); }); + describe("Full screen", () => { + let fullscreenEl: Element | null; + let requestSpy: ReturnType; + let exitSpy: ReturnType; + let removeSpy: ReturnType | undefined; + const mounted: ReturnType[] = []; + + const fireChange = async (el: Element | null) => { + fullscreenEl = el; + document.dispatchEvent(new Event("fullscreenchange")); + await flushPromises(); + }; + + const mountSupported = async () => { + const local = mountComponent(); + mounted.push(local); + await flushPromises(); + return local; + }; + + const mountKept = async () => { + const Host = defineComponent({ + components: { VideoPlayer }, + props: { show: { type: Boolean, default: true } }, + setup: () => ({ events: mockEvents }), + template: `
`, + }); + const host = mount(Host, { + global: { + plugins: [i18n], + provide: { store }, + stubs: { + OIcon: { + template: '', + props: ["name", "size"], + }, + }, + }, + }); + mounted.push(host); + await flushPromises(); + return host; + }; + + const deferRequest = () => { + let settle: () => void = () => {}; + requestSpy.mockImplementation( + () => + new Promise((resolve) => { + settle = resolve; + }), + ); + return () => settle(); + }; + + const button = (local: ReturnType) => + local.find('[data-test="video-player-fullscreen-btn"]'); + + const isPinned = (local: ReturnType) => + local.find('[data-test="video-player-root"]').classes().includes("fixed"); + + beforeEach(() => { + fullscreenEl = null; + requestSpy = vi.fn(() => Promise.resolve()); + exitSpy = vi.fn(() => Promise.resolve()); + Object.defineProperty(document, "fullscreenEnabled", { configurable: true, value: true }); + Object.defineProperty(document, "fullscreenElement", { + configurable: true, + get: () => fullscreenEl, + }); + Object.defineProperty(document, "exitFullscreen", { configurable: true, value: exitSpy }); + Object.defineProperty(document.documentElement, "requestFullscreen", { + configurable: true, + value: requestSpy, + }); + }); + + afterEach(() => { + mounted.splice(0).forEach((w) => { + if (!w.vm.$.isUnmounted) w.unmount(); + }); + removeSpy?.mockRestore(); + removeSpy = undefined; + delete (document as any).fullscreenEnabled; + delete (document as any).fullscreenElement; + delete (document as any).exitFullscreen; + delete (document.documentElement as any).requestFullscreen; + }); + + it("renders the button when the browser supports full screen", async () => { + const local = await mountSupported(); + expect(button(local).exists()).toBe(true); + expect(button(local).attributes("aria-label")).toBe("Full screen"); + expect(button(local).find('[data-test="OIcon"]').attributes("data-name")).toBe("fullscreen"); + }); + + it("hides the button when the browser does not allow full screen", async () => { + Object.defineProperty(document, "fullscreenEnabled", { configurable: true, value: false }); + const local = await mountSupported(); + expect(button(local).exists()).toBe(false); + }); + + it("requests full screen on the page, and exits it on a click while active", async () => { + const local = await mountSupported(); + await button(local).trigger("click"); + await flushPromises(); + expect(requestSpy).toHaveBeenCalledTimes(1); + expect(requestSpy.mock.contexts[0]).toBe(document.documentElement); + + await fireChange(document.documentElement); + await button(local).trigger("click"); + await flushPromises(); + expect(exitSpy).toHaveBeenCalledTimes(1); + }); + + it("pins the player and swaps icon and label on entry, and unpins on a browser exit", async () => { + const local = await mountSupported(); + await button(local).trigger("click"); + await fireChange(document.documentElement); + + expect(isPinned(local)).toBe(true); + expect(button(local).attributes("aria-label")).toBe("Exit full screen"); + expect(button(local).find('[data-test="OIcon"]').attributes("data-name")).toBe( + "fullscreen-exit", + ); + + await fireChange(null); + expect(isPinned(local)).toBe(false); + expect(button(local).attributes("aria-label")).toBe("Full screen"); + expect(button(local).find('[data-test="OIcon"]').attributes("data-name")).toBe("fullscreen"); + }); + + it("does not pin the player for a full screen another component started", async () => { + const local = await mountSupported(); + await fireChange(document.documentElement); + expect(isPinned(local)).toBe(false); + + await fireChange(null); + await fireChange(document.createElement("div")); + expect(isPinned(local)).toBe(false); + }); + + it("claims a page full screen another component owns instead of exiting it", async () => { + const local = await mountSupported(); + await fireChange(document.documentElement); + expect(isPinned(local)).toBe(false); + + await button(local).trigger("click"); + await flushPromises(); + expect(exitSpy).not.toHaveBeenCalled(); + expect(requestSpy).not.toHaveBeenCalled(); + expect(isPinned(local)).toBe(true); + expect(button(local).attributes("aria-label")).toBe("Exit full screen"); + }); + + it("ignores a second click while the first request is pending, then pins on entry", async () => { + const settle = deferRequest(); + const local = await mountSupported(); + await button(local).trigger("click"); + await button(local).trigger("click"); + expect(requestSpy).toHaveBeenCalledTimes(1); + + await fireChange(document.documentElement); + settle(); + await flushPromises(); + expect(isPinned(local)).toBe(true); + expect(button(local).attributes("aria-label")).toBe("Exit full screen"); + expect(exitSpy).not.toHaveBeenCalled(); + }); + + it("leaves the player unchanged when the browser denies the request", async () => { + requestSpy.mockImplementation(() => Promise.reject(new Error("denied"))); + const local = await mountSupported(); + await button(local).trigger("click"); + await flushPromises(); + expect(isPinned(local)).toBe(false); + expect(button(local).attributes("aria-label")).toBe("Full screen"); + + await fireChange(document.documentElement); + expect(isPinned(local)).toBe(false); + }); + + it("exits full screen and drops its listener on unmount while active", async () => { + removeSpy = vi.spyOn(document, "removeEventListener"); + const local = await mountSupported(); + await button(local).trigger("click"); + await fireChange(document.documentElement); + + local.unmount(); + expect(exitSpy).toHaveBeenCalledTimes(1); + expect(removeSpy).toHaveBeenCalledWith("fullscreenchange", expect.any(Function)); + }); + + it("exits full screen when deactivated by keep-alive while active", async () => { + const host = await mountKept(); + await host.find('[data-test="video-player-fullscreen-btn"]').trigger("click"); + await fireChange(document.documentElement); + + await host.setProps({ show: false }); + await flushPromises(); + expect(exitSpy).toHaveBeenCalledTimes(1); + }); + + it("exits a full screen that lands after the player unmounted mid-request", async () => { + const settle = deferRequest(); + const local = await mountSupported(); + await button(local).trigger("click"); + local.unmount(); + + fullscreenEl = document.documentElement; + settle(); + await flushPromises(); + expect(exitSpy).toHaveBeenCalledTimes(1); + }); + + it("exits a full screen that lands after keep-alive deactivated the player mid-request", async () => { + const settle = deferRequest(); + const host = await mountKept(); + await host.find('[data-test="video-player-fullscreen-btn"]').trigger("click"); + await host.setProps({ show: false }); + await flushPromises(); + expect(exitSpy).not.toHaveBeenCalled(); + + fullscreenEl = document.documentElement; + settle(); + await flushPromises(); + expect(exitSpy).toHaveBeenCalledTimes(1); + }); + + describe("through the ResizeObserver", () => { + let OriginalResizeObserver: typeof ResizeObserver; + let callbacks: ResizeObserverCallback[]; + + beforeEach(() => { + OriginalResizeObserver = global.ResizeObserver; + callbacks = []; + function MockRO(this: any, cb: ResizeObserverCallback) { + callbacks.push(cb); + } + MockRO.prototype.observe = vi.fn(); + MockRO.prototype.disconnect = vi.fn(); + MockRO.prototype.unobserve = vi.fn(); + global.ResizeObserver = MockRO as unknown as typeof ResizeObserver; + }); + + afterEach(() => { + global.ResizeObserver = OriginalResizeObserver; + }); + + it("re-fits the frame and the bar through the ResizeObserver after entering and exiting", async () => { + const local = await mountSupported(); + await local.setProps({ segments: mockSegments }); + await flushPromises(); + const instance = playerSpies.instances.at(-1); + const fire = () => callbacks.forEach((cb) => cb([], {} as ResizeObserver)); + const setSize = (width: number, height: number) => { + Object.defineProperty(HTMLElement.prototype, "clientWidth", { + configurable: true, + value: width, + }); + Object.defineProperty(HTMLElement.prototype, "clientHeight", { + configurable: true, + value: height, + }); + }; + const markerLeft = () => + parseFloat( + (local.find('[data-test="video-player-event-marker"]').element as HTMLElement).style + .left, + ); + fire(); + await flushPromises(); + const leftBefore = markerLeft(); + expect(leftBefore).toBeGreaterThan(0); + + setSize(1600, 1200); + await button(local).trigger("click"); + await fireChange(document.documentElement); + fire(); + await flushPromises(); + expect((local.vm as any).playerState.width).toBe(1600); + expect(instance.$set).toHaveBeenLastCalledWith(expect.objectContaining({ width: 1600 })); + expect(markerLeft()).toBeCloseTo(leftBefore * 2); + + setSize(800, 600); + await fireChange(null); + fire(); + await flushPromises(); + expect((local.vm as any).playerState.width).toBe(800); + expect(instance.$set).toHaveBeenLastCalledWith(expect.objectContaining({ width: 800 })); + expect(markerLeft()).toBeCloseTo(leftBefore); + }); + }); + }); + describe("Error and empty states", () => { it("shows the load error with a Retry that reaches the parent", async () => { const local = mountComponent({ loadState: "error" }); diff --git a/web/src/components/rum/VideoPlayer.vue b/web/src/components/rum/VideoPlayer.vue index cc8e8cf5c1..9688fd3057 100644 --- a/web/src/components/rum/VideoPlayer.vue +++ b/web/src/components/rum/VideoPlayer.vue @@ -15,7 +15,11 @@ along with this program. If not, see . -->