starred/openobserve
1
0
Fork 0
mirror of https://github.com/openobserve/openobserve.git synced 2026-11-05 00:09:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'main' into perf/ui-unit-test-runtime

Browse Source
This commit is contained in:
Ashish Kolhe
2026-10-31 14:43:16 +05:30
committed by GitHub
parent 2df43dc32f 7a6b1bbe90
commit 87b5b3aeba
503 changed files with 57643 additions and 7521 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/engineering-guidelines.md
+231
View File
@@ -0,0 +1,231 @@
# How to build things here
These rules apply in every chat and every repo. They are not a style preference. They exist because
an agent can write thousands of lines in an hour, some of them invented, and a human then has to
review all of it. Every rule below either cuts invention or makes the code easier to review.
**Precedence.** A repo's own instruction file (`AGENTS.md` / `CLAUDE.md`) wins on everything
mechanical: formatting, item order, comment rules, lint thresholds, schema rules, PR conventions.
This file governs judgment: what to build, how much of it, and how to prove it works. If the two
conflict, follow the repo file and say so in your report.
The definitions and tests apply to every language in the repo. The examples in §2 and §3 are written
for the Rust backend; a frontend list belongs next to them when the frontend owners write one.
## 0. Pick the tier first
Every task is one of three sizes. Say which one you are in before touching code; it decides how
much of the rest applies.
| Tier | What it is | Spec gate | Tests | Review |
|---|---|---|---|---|
| **Patch** | One cause, one place. No new public API, table, flag or dependency. Typically under ~50 changed lines. | None. One sentence: cause → fix. | One failing test that reproduces the cause, then green. | Self-review against the three tests in §2, then the gates. |
| **Change** | New behaviour inside the existing architecture: a new branch of logic, a new field, a new error path. | Numbered rules and open questions (§5.2), a few lines each. | One per rule. | Full loop (§5). |
| **Feature** | A new component, endpoint, table, background job or config flag. | Design doc first, per the repo's PR rules; the numbered rules derive from it. | One per rule, plus a failure-path test for every `[strong]` / `[weak]` function. | Full loop, security depth where §5 says so. |
If a Patch outgrows its definition while you work, stop, say so, and re-enter as a Change. Do not
keep going.
## 1. Never invent facts
- **A claim about existing code needs a `path:line` you opened in this session.** A function,
table, config flag, env var or endpoint you have not opened does not exist yet. Search for it
before you use it. This applies to specs, reviews and reports, not to every sentence in chat.
- **A claim about an outside service** (Twilio, Slack, a cloud API) needs a link to the vendor's
docs. "Twilio probably returns X" is not allowed.
- **"I don't know" is a correct answer.** Write it down as an open question. Do not fill the gap
with something that sounds right.
- **If the spec does not decide something, stop and ask.** An invented decision is how a feature
turns into an endless fix loop. Anything new you discover while building becomes a new numbered
rule, and the human decides whether it is in scope.
## 2. The three things we refuse
Each one has a definition, a test you can apply mechanically, and examples. The cold reviewer (§5)
applies the same three tests, so there is one vocabulary for writer and reviewer.
### 2.1 Hacks
**Definition.** Code that makes a symptom disappear while the underlying statement about the system
stays false.
**Test.** Delete the new code: is the thing it "fixed" still wrong, for the same reason? Then it was
a hack. Second test: does the fix only work because of a value, name, order or timing that is not
part of any contract?
**Examples, all banned:**
- A branch keyed on one input, test name, org id or environment so that one case passes.
- Turning an error into success where the caller needed to know: `.ok()`, `.unwrap_or_default()`,
`let _ = …`, `if let Ok(x)` with no `else`, a handler that logs at debug and continues.
- A sleep, retry or re-ordering that makes a race rarer instead of removing it.
- Widening a type (`String`, `serde_json::Value`, `i64` for everything) so the compiler stops
complaining, instead of modelling the value.
- `#[allow(clippy::…)]`, `#[ignore]`, a weakened assertion, or a raised lint threshold to get green.
- A flag that defaults the new behaviour off so the bug stops reproducing.
- A "temporary" anything with no issue number and no removal condition.
**Not a hack:** a fallback the spec names and that is visible when it fires (a `warn!` with context,
a metric, an error field in the response). A default is a decision; a silent default is a hack.
**When the real cause is out of scope:** stop and report it. Do not patch the symptom to finish the
task.
### 2.2 Over-engineering
**Definition.** Structure that exists for a future the spec does not contain.
**Test.** Name the second concrete user that exists today, or that the spec names. If you cannot,
inline it.
**Examples, all banned until the second user exists:**
- A trait with one impl; a generic with one instantiation; a type parameter "for testing".
- A config option, env var or feature flag nobody asked for ("in case someone wants to tune it").
- A newtype or wrapper that enforces no invariant.
- Layers that only forward: `handler → service → manager → repository` with logic in one of them.
- A builder for a struct with three fields; a registry or plugin point with one entry.
- An error enum with variants nothing matches on; one variant per call site when every caller
just logs.
- A new module or file for one function.
**Counts as a second user:** another call site in the same diff, another provider the spec names,
or an existing trait that every implementation in the repo already follows. **Does not count:** "the
UI will need it", "we may add Postgres", or a test that is the only other caller.
### 2.3 Over-complication
**Definition.** Control flow or state a reviewer cannot predict from the signature and the first
screen of the body. Over-engineering is too much structure around the code; over-complication is
too much happening inside one piece of it.
**Test.** Cover everything below the first screen of the function. Can you say what it returns for
each kind of input? If not, split it.
**Examples, each needs a split or a name:**
- A `bool` parameter that switches behaviour (`send(msg, true, false)`): two functions, or an enum.
- More than ~5 parameters: a named struct.
- Nesting deeper than three: an early return or a helper.
- `Option<Option<T>>`, `Result<Option<Result<…>>>`, `(bool, bool, Option<String>)` as a return
type: a named enum.
- An iterator chain whose closures contain branches: a loop.
- A function that is only correct if another one ran first: return the value and pass it in, or
make the ordering a type.
- `Arc<Mutex<_>>` to share what could be passed by value; `async` or `spawn` for work that is not
I/O; a macro where a function works.
- A `&str` or `i32` that is really an enum (`status == "running"`).
- A mutable local that is reassigned across more than one screen.
**Default when unsure:** the version with fewer names and fewer branches wins. If two versions both
meet the rules, ship the shorter diff.
## 3. Build in logical components with predictable behaviour
Split each Change or Feature into **logical components**. A logical component is one unit with one
job, such as "decide who to page", "render the SMS text" or "send through the provider".
- **One line, no exceptions.** You must be able to explain each component in one plain sentence
that has no "except when". If you cannot, it is two components.
- **Same input, same output.** Keep the decision logic pure: no I/O, and the clock (`now: i64`),
randomness and config are passed in as arguments. Side effects (database, network, queue) live in
a thin outer layer that only carries out what the pure core decided. In practice: the core takes
the config *values* it needs; the outer layer is the one that reads the global config.
- **Deep is fine, wide is not.** A function may run many steps one after another. It should not
fan out into many branches. Prefer an exhaustive `match` on a project enum (no catch-all `_`
hiding a future variant; `_` is only for `#[non_exhaustive]` foreign enums), early returns, and
lookup tables over chains of `if/else`. If a decision has more than about four outcomes, give the
outcome a name: make it an enum.
- **Write the guarantee in the type wherever the type can say it.** Adapted from Audacity's
safety guarantees (https://audacity.gitbook.io/dev/getting-started/coding-standards#safety_guarantees_defined):
- *pure*: the signature says it. Takes values or `&`, returns a value, not `async`, touches no
global. No tag needed.
- *no-fail*: returns `T`, not `Result<T>`. Any `unwrap`, `expect`, index or unchecked arithmetic
inside needs a one-line comment naming the invariant that makes failure impossible.
- `[strong]`, tagged at the end of the doc comment: either it fully succeeds, or it fails and
nothing changed. For example, one database transaction.
- `[weak: <what may be left behind>]`, tagged: a failure can leave partial effects, but
everything stays consistent. Name what is left behind and what makes a retry safe. For example:
`[weak: the ledger row may exist without a send; a retry is a no-op]`.
Only the outer I/O layer carries `[strong]` / `[weak]`. A tag is a promise: each one needs a test
of its failure path.
## 4. TDD, without the bloat
- **Test first, and see it fail.** A test written after the code, or one you never saw fail,
proves nothing. Run the narrowest target that contains it red, then make it green.
Exempt: a pure refactor (behaviour unchanged, the existing tests are the net and you do not edit
them) and a pure deletion.
- **One test per numbered rule or per real branch.** Use table-driven tests (inputs → expected
outputs) for pure components. Do not test the framework, getters or private helpers on their
own. Do not repeat an assertion that another test already makes.
- **Pass values, do not mock.** If a function needs a mock to be tested, its I/O is in the wrong
layer (§3). Mocks are for the thin outer layer only, and only when no real local dependency is
available.
- **Never change a test's expectation to get green** unless the PR states why the old expectation
was wrong.
- **Done = every numbered rule has a passing test and the repo's gates pass.** "It looks
complete" is not done.
- **Size each diff by rule, not by subsystem.** One rule (or a few small ones), its test and its
code, small enough for a person to review in one sitting.
## 5. The working loop
In repos that have the `o2-loop` skill (openobserve, o2-enterprise), the skill *is* steps 3–6
below: it owns the round cap and the findings ledger. Elsewhere, run the steps by hand.
1. **The main chat is the monitor.** It holds the spec, the rule list and the progress. It
delegates the work and does not do bulk reading or writing itself.
2. **Spec gate** (Change and Feature only). Before any code, write the numbered rules, the
components with their one-line jobs and guarantee tags, and the open questions. Ask the human
about every open question.
3. **Tests.** A subagent writes the tests for the next rule-sized slice. Before it reports, it
checks them once against this list: does each test map to a rule, did it fail first, is any
test a duplicate, is anything invented.
4. **Cold review.** A fresh reviewer (one that did not write the code) gets only the spec path
and the diff, not the conversation, and applies the three tests in §2. Each finding must give
`path:line` and a concrete failure: "input X gives wrong output Y". A finding with no concrete
failure is dropped.
5. **Confirm each finding before acting on it.** Read the cited lines, or run a test that shows
the failure. Reviewers make things up too. Fix only the findings you confirmed, and record the
ones you rejected along with the reason.
6. **Repeat step 4** until a round produces no confirmed finding. If the round cap is reached and
real problems remain, the spec is the problem: go back to the human.
7. **Run the repo's gates (fmt, clippy, tests) before every review round** so reviewers do not
spend tokens on what a tool would catch.
**Review depth follows risk.** Pure logic with good table tests needs one cold round. Full depth
for anything that touches: unauthenticated endpoints, authorization, signature verification,
secrets, money, data deletion, migrations, anything that pages a human.
**Stop and ask the human when:**
- the spec does not decide something (§1);
- the only green path you can find is a hack (§2.1);
- the fix needs to touch files or layers outside the stated scope;
- an abstraction needs a second user and the spec has none (§2.2);
- a test can only pass by changing its expectation;
- the gates fail for a reason that predates your change.
**Keeping the token cost down.** Give a subagent file paths and a narrow question, not pasted
contents, and ask for conclusions with citations, capped in length. Use read-only search agents
for searching. Continue an existing agent instead of starting one that has to re-read everything.
Do not re-read files you already read, and do not re-derive facts you already settled.
## 6. UI
- Follow the repo's UI rules. In openobserve that is the `ui-architect` skill: O2 components,
rem not px, design tokens. Mock the flow on the real product UI before building it.
- **Zero-config for 90% of users.** Defaults that work without any setup, controls whose meaning
is obvious from the screen, and no step that needs the docs. Advanced options stay out of the way
until someone needs them.
## 7. How to write for the human
- Write for a reader who has not followed the conversation. Define each term the first time you
use it, use plain words, and make the conclusion the first thing they read.
- When agents reach a decision, present it as: what we will build, how it behaves for the user,
what we decided not to do, and what still needs a human answer.
- A PR description states the tier (§0), lists the numbered rules with the test that covers each,
and lists the rejected review findings with their reasons.
.claude/agents/o2-coder.md
+2 -2
View File
@@ -10,7 +10,7 @@ When the brief names a checkout path, every file you read or edit and every comm
## Ground rules
- Follow the repo's `CLAUDE.md` exactly: item ordering, one-line comments, tests module last, clippy thresholds.
- Follow the repo's `AGENTS.md` exactly: item ordering, one-line comments, tests module last, clippy thresholds.
- Never run `git commit`, `git add`, `git reset`, `git stash`, or `git push`. The orchestrator's script freezes your working tree as a WIP commit; you only edit files.
- Never spawn agents or split your work; if the spec needs more than one coder, that is the orchestrator's decision, and you say so in a `question:` line.
- Never write to the ledger except the files named below, and never edit `verdict.json`.
@@ -27,7 +27,7 @@ When the brief names a checkout path, every file you read or edit and every comm
cargo fmt --all
cargo clippy --workspace --all-targets -- -W clippy::too_many_lines -W clippy::cognitive_complexity -W clippy::excessive_nesting -D warnings
```
If the change touches `#[cfg(feature = "enterprise")]` code, follow `CLAUDE.md`'s enterprise verification rule or say in the evidence that it was not checked.
If the change touches `#[cfg(feature = "enterprise")]` code, follow `AGENTS.md`'s enterprise verification rule or say in the evidence that it was not checked.
4. Write `<ledger>/round-1/evidence.md`: the exact commands, pass or fail, test names, tails of any failure output, and anything skipped.
5. Reply to the orchestrator with: files changed, gates run, anything you deviated from in the spec and why.
.claude/skills/o2-loop/prompts/backend-codex.md
+1 -1
View File
@@ -1,2 +1,2 @@
## Notes for the Codex reviewer process
You run in Codex's read-only sandbox inside a disposable checkout of the commit under review. There is no `AGENTS.md` in this repository; `CLAUDE.md` at the checkout root is the rules file, so do not search for anything else. The ledger paths given below are absolute and readable; `rg` is not installed, use `grep`, `cat`, and `git`. Do not run cargo; the evidence section already carries the build and test results.
You run in Codex's read-only sandbox inside a disposable checkout of the commit under review. There is no `CLAUDE.md` in this repository; `AGENTS.md` at the checkout root is the rules file, so do not search for anything else. The ledger paths given below are absolute and readable; `rg` is not installed, use `grep`, `cat`, and `git`. Do not run cargo; the evidence section already carries the build and test results.
.claude/skills/o2-loop/prompts/review.md
+5 -3
View File
@@ -1,8 +1,9 @@
You are the second reviewer in a two-AI loop: the coder (a Claude agent) wrote this change, you (an independent reviewer process) review it, the coder fixes, you verify. Your job is to find real defects that a careful senior Rust engineer would block a PR on. Be precise, not exhaustive.
## Before you look at the diff
1. Read `CLAUDE.md` at the repo root. Its rules on item ordering, comments, and clippy thresholds are hard requirements for this repo.
2. Read the evidence file listed below. It contains the build, clippy, and test results the coder already ran. Do not re-run cargo; the sandbox is read-only.
1. Read `AGENTS.md` at the repo root. Its rules on item ordering, comments, and clippy thresholds are hard requirements for this repo.
2. Read §2 of `.agents/engineering-guidelines.md`: the definitions and tests for a hack, over-engineering and over-complication. Apply those tests, not your own taste, when you report one.
3. Read the evidence file listed below. It contains the build, clippy, and test results the coder already ran. Do not re-run cargo; the sandbox is read-only.
## What to review
Only the change set described in the "Change set" section. Read surrounding code as needed to judge it, but do not report pre-existing problems outside the diff unless the diff makes them worse.
@@ -14,7 +15,8 @@ Look for, in priority order:
- Error handling: swallowed errors, `unwrap` on fallible paths, wrong status codes, silent truncation of results.
- API and compatibility contracts: wire-format changes, enterprise `cfg` gates, behaviour differences between OSS and enterprise builds.
- Performance regressions on hot paths: extra allocations or clones in loops, O(n²), needless full scans, metrics with high-cardinality labels.
- Repo conventions from `CLAUDE.md`: multi-line or narrating comments, functions placed above constants or types, tests module not last, `pub` and private functions interleaved.
- Hacks, over-engineering and over-complication as `.agents/engineering-guidelines.md` §2 defines them: a symptom patch, an abstraction with one user, a function whose result cannot be predicted from its first screen.
- Repo conventions from `AGENTS.md`: multi-line or narrating comments, functions placed above constants or types, tests module not last, `pub` and private functions interleaved.
- Missing tests for new branches, or tests that cannot fail.
Do not report: formatting, naming taste, anything `cargo fmt` or `cargo clippy` already enforces, or speculative issues you cannot point to a concrete line for.
.claude/skills/o2-loop/prompts/verify.md
+3 -2
View File
@@ -1,7 +1,8 @@
You are the second reviewer in a two-AI loop: the coder (a Claude agent) wrote this change, an independent reviewer process (possibly you) reviewed it in a previous round, the coder responded, and now you verify. You have no memory of the previous round; everything you need is in the files listed below.
## Before you look at the code
1. Read `CLAUDE.md` at the repo root. Its rules are hard requirements for this repo.
1. Read `AGENTS.md` at the repo root. Its rules are hard requirements for this repo.
Apply §2 of `.agents/engineering-guidelines.md` (hack, over-engineering, over-complication) by its tests, not by taste.
2. Read the evidence file for this round (build, clippy, and test results the coder ran after fixing). Do not re-run cargo; the sandbox is read-only.
3. Read every earlier round's findings and the coder's per-finding responses, all listed below in order. A finding's severity, location, and defect description live in the round that first reported it; later rounds only carry its id and status.
@@ -14,7 +15,7 @@ A prior finding is any id from any earlier round whose latest status is not `res
Do not repeat a prior finding in `findings`; its status lives in `prior_findings`.
## Step 2: review what changed since the previous round
The delta between the previous round's commit and this round's commit is listed below. Review only that delta for new defects, using the same priorities as a first-round review: correctness, concurrency, security, error handling, API contracts, performance, `CLAUDE.md` conventions, missing tests. New findings get fresh ids continuing after the highest id used in any earlier round.
The delta between the previous round's commit and this round's commit is listed below. Review only that delta for new defects, using the same priorities as a first-round review: correctness, concurrency, security, error handling, API contracts, performance, `AGENTS.md` conventions, missing tests. New findings get fresh ids continuing after the highest id used in any earlier round.
## Output rules
- `verdict` is `approve` only if no prior finding is `still_open` with an original severity of critical, high, or medium, and no new finding is critical, high, or medium.
.gitignore
+1
View File
@@ -68,6 +68,7 @@ tests/ui-testing/playwright-tests/utils/auth/user.json
# ---------- other agents ----------
/.agents/*
!/.agents/engineering-guidelines.md
alert_mock
# Local verification artifacts (browser screenshots, per-crate scratch DBs)
AGENTS.md
+7
View File
@@ -1,5 +1,12 @@
# OpenObserve — project rules
## Engineering guidelines
- [.agents/engineering-guidelines.md](.agents/engineering-guidelines.md) governs judgment: task
tiers, what counts as a hack / over-engineering / over-complication, TDD and the review loop.
Read it before any change bigger than a one-cause patch; reviewers apply its §2 tests. On
anything mechanical (ordering, comments, lint thresholds, schema, PR rules) this file wins.
## Build & workflow
- Build with `cargo build` (debug). Never use `--release` unless explicitly asked.
Cargo.toml
+6
View File
@@ -51,8 +51,14 @@ cloud = [
]
mimalloc = ["dep:mimalloc"]
jemalloc = ["dep:tikv-jemallocator"]
cpu-profiling = [
"openobserve-api-http/cpu-profiling",
"openobserve-jobs/cpu-profiling",
"openobserve-core/cpu-profiling",
]
profiling = [
"jemalloc",
"cpu-profiling",
"openobserve-api-http/profiling",
"openobserve-jobs/profiling",
"openobserve-core/profiling",
deploy/build/Dockerfile.profiling.amd64
+1 -1
View File
@@ -23,7 +23,7 @@ COPY . /openobserve
COPY --from=webbuilder /web/dist web/dist
RUN mkdir -p /openobserve/target/release/
RUN --mount=type=cache,target=/root/.cache/sccache cargo build --profile release-profiling --features mimalloc,profiling --target x86_64-unknown-linux-gnu \
RUN --mount=type=cache,target=/root/.cache/sccache cargo build --profile release-profiling --features mimalloc,cpu-profiling --target x86_64-unknown-linux-gnu \
&& sccache --show-stats
RUN mv /openobserve/target/x86_64-unknown-linux-gnu/release-profiling/openobserve /openobserve/target/release/openobserve
scripts/translations/.translation_state.json
+580 -93
View File
File diff suppressed because it is too large Load Diff
src/api/common/src/auth/jwt.rs
+308 -143
View File
@@ -13,11 +13,6 @@
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
#[cfg(feature = "cloud")]
use openobserve_core::{
organization::list_org_users_by_user,
self_reporting::cloud_events::{CloudEvent, EventType, enqueue_cloud_event},
};
#[cfg(all(feature = "enterprise", not(feature = "cloud")))]
use {
crate::common::meta::user::RoleOrg,
@@ -36,6 +31,25 @@ use {
std::collections::HashMap,
std::sync::LazyLock as Lazy,
};
#[cfg(feature = "cloud")]
use {
config::get_config,
config::{DEFAULT_ORG, meta::user::UserRole, utils::rand::generate_random_string},
db::org_status,
db::{org_users, organization::get_org_setting},
o2_enterprise::enterprise::cloud::org_invites,
o2_enterprise::enterprise::{cloud::billing_group, domain_management::meta::AccessDecision},
o2_openfga::authorizer::authz::get_add_user_to_org_tuples,
o2_openfga::authorizer::roles::{
check_and_get_crole_tuple_for_new_user, get_roles_for_org_user,
get_user_crole_removal_tuples,
},
openobserve_core::org_domain_ownership::get_cached_org_for_domain,
openobserve_core::organization::list_org_users_by_user,
openobserve_core::self_reporting::cloud_events::{CloudEvent, EventType, enqueue_cloud_event},
openobserve_core::users,
std::{collections::HashSet, str::FromStr as _},
};
#[cfg(all(feature = "enterprise", not(feature = "cloud")))]
use {
config::meta::user::{UserOrg, UserRole},
@@ -45,19 +59,6 @@ use {
},
std::str::FromStr,
};
#[cfg(feature = "cloud")]
use {
config::{DEFAULT_ORG, META_ORG_ID, meta::user::UserRole, utils::rand::generate_random_string},
db::org_status,
db::{org_users, organization::get_org_setting},
o2_enterprise::enterprise::cloud::org_invites,
o2_openfga::authorizer::authz::get_add_user_to_org_tuples,
o2_openfga::authorizer::roles::{
check_and_get_crole_tuple_for_new_user, get_roles_for_org_user,
get_user_crole_removal_tuples,
},
std::{collections::HashSet, str::FromStr as _},
};
#[cfg(feature = "cloud")]
use crate::common::meta::{
@@ -124,9 +125,9 @@ pub async fn process_token(
let mut custom_roles: Vec<String> = vec![];
let mut tuples_to_add = HashMap::new();
// Try custom claim parsing first
// Try custom claim parsing first, always from _meta
if let Some((parsed_orgs, parsed_roles)) =
process_custom_claim_parsing(&dec_token.claims).await
process_custom_claim_parsing(&dec_token.claims, "_meta").await
{
log::info!(
"Using custom claim parsing results: {} orgs, {} custom roles",
@@ -879,141 +880,262 @@ pub async fn process_domain_org_mapping(
claims: HashMap<String, Value>,
is_new_user: bool,
) -> Result<bool, anyhow::Error> {
let meta_settings = get_org_setting(META_ORG_ID).await?;
let mappings = meta_settings.domain_org_mappings;
// split email to get the domain
if let Some((_, domain)) = user_email.to_lowercase().split_once("@") {
if let Some(mapped) = mappings
.into_iter()
.find(|m| m.domain.to_lowercase() == domain)
{
match org_status::read_active(&mapped.org_id).await {
Ok(true) => {}
Ok(false) => {
log::warn!(
"domain org mapping for user {user_email} points at inactive org {}, skipping",
mapped.org_id
);
return Ok(false);
let Some((_, domain)) = user_email.split_once("@") else {
log::warn!(
"user email {user_email} could not be split correctly at @, skipping org domain mapping"
);
return Ok(false);
};
let domain = domain.trim().to_lowercase();
let Some(org_id) = get_cached_org_for_domain(&domain).await else {
log::info!("no cached domain org mapping found for user {user_email}, continuing normally");
return Ok(false);
};
let org_settings = get_org_setting(&org_id).await?;
let mappings = org_settings.domain_org_mappings;
let mut allowed_orgs: HashSet<_> = billing_group::list_billing_group_members_of(&org_id)
.await?
.into_iter()
.map(|v| v.member_org_id)
.collect();
allowed_orgs.insert(org_id.clone());
let decision = org_settings.domain_management_config.evaluate(user_email);
if matches!(decision, AccessDecision::Deny) {
if is_new_user {
// for new user who is blocked via domain, we return false, so
// the normal flow can take place
return Ok(false);
}
let orgs = org_users::list_orgs_by_user(user_email).await?;
let mut orgs_to_remove = Vec::new();
for org in orgs {
if allowed_orgs.contains(&org.org_id) {
orgs_to_remove.push(org.org_id);
}
}
for org in orgs_to_remove {
match users::remove_user_from_org(
&org,
&user_email,
&get_config().auth.root_user_email,
users::RemovalTarget::User,
)
.await
{
Ok(v) => {
if !v.status().is_success() {
log::error!(
"error user {user_email} was blocked from org {org_id}, but failed to remove from org {org} with status : {}",
v.status().as_str()
);
}
}
Err(e) => {
log::warn!(
"domain org mapping for user {user_email}: cannot read the status of org {}, skipping: {e}",
mapped.org_id
log::error!(
"error user {user_email} was blocked from org {org_id}, but failed to remove from org {org} : {e}"
);
return Ok(false);
}
}
}
log::info!("found domain org mapping for user {user_email}, processing");
let base_role = UserRole::from_str(&mapped.base_role).map_err(|e| {
anyhow::anyhow!(
"error parsing domain org role {} to user role : {e:?}",
mapped.base_role
)
})?;
log::info!("user {user_email} blocked by domain management config of org {org_id}");
return Ok(true);
}
let need_to_add_user = org_users::get_optional(&mapped.org_id, user_email)
.await?
.is_none();
let mapped_orgs: Vec<_> = mappings
.into_iter()
.filter(|m| m.domain.to_lowercase() == domain && allowed_orgs.contains(&m.org_id))
.collect();
if is_new_user || need_to_add_user {
let mut new_tuples = Vec::new();
get_add_user_to_org_tuples(
&mapped.org_id,
user_email,
&mapped.base_role,
&mut new_tuples,
if mapped_orgs.is_empty() {
log::info!(
"domain {domain} is owned by {org_id} but no mapping found, processing user {user_email} normally"
);
return Ok(false);
}
let mut added = 0;
for mapped in mapped_orgs {
match org_status::read_active(&mapped.org_id).await {
Ok(true) => {}
Ok(false) => {
log::warn!(
"domain org mapping for user {user_email} points at inactive org {}, skipping",
mapped.org_id
);
update_tuples(new_tuples, vec![]).await?;
org_users::add(
&mapped.org_id,
user_email,
base_role,
&generate_random_string(16),
Some(format!("rum{}", generate_random_string(16))),
)
.await?;
continue;
}
if let Some(claim_name) = mapped.role_claim_name
&& let Some(v) = claims.get(&claim_name)
&& let Some(arr) = v.as_array()
{
let mut roles = HashSet::new();
for claim in arr {
if let Some(role) = claim.as_str() {
roles.insert(format_role_name_only(role));
}
}
let existing_roles = get_roles_for_org_user(&mapped.org_id, user_email).await;
Err(e) => {
log::warn!(
"domain org mapping for user {user_email}: cannot read the status of org {}, skipping: {e}",
mapped.org_id
);
continue;
}
}
let mut add_roles = Vec::new();
let mut add_tuples = Vec::new();
let mut remove_tuples = Vec::new();
if org_status::is_blocked(&mapped.org_id) {
log::info!(
"user {user_email} supposed to be mapped to org {} but org status is blocked, so skipping",
mapped.org_id
);
continue;
}
log::info!(
"found domain org mapping {} for user {user_email}, processing",
mapped.org_id
);
let base_role = UserRole::from_str(&mapped.base_role).map_err(|e| {
anyhow::anyhow!(
"error parsing domain org role {} to user role : {e:?}",
mapped.base_role
)
})?;
for role in &existing_roles {
if !roles.contains(role) {
let name = format_role_name(&mapped.org_id, role);
get_user_crole_removal_tuples(user_email, &name, &mut remove_tuples);
}
}
for role in &roles {
if !existing_roles.contains(role) {
add_roles.push(format_role_name(&mapped.org_id, role))
}
}
if !add_roles.is_empty() {
check_and_get_crole_tuple_for_new_user(
user_email,
&mapped.org_id,
add_roles,
mapped.create_missing_roles,
&mut add_tuples,
)
.await;
}
let need_to_add_user = org_users::get_optional(&mapped.org_id, user_email)
.await?
.is_none();
if (!add_tuples.is_empty() || !remove_tuples.is_empty())
&& let Err(e) = update_tuples(add_tuples, remove_tuples).await
{
if is_new_user || need_to_add_user {
let mut new_tuples = Vec::new();
get_add_user_to_org_tuples(
&mapped.org_id,
user_email,
&mapped.base_role,
&mut new_tuples,
);
update_tuples(new_tuples, vec![]).await?;
org_users::add(
&mapped.org_id,
user_email,
base_role,
&generate_random_string(16),
Some(format!("rum{}", generate_random_string(16))),
)
.await?;
}
if let Some(claim_name) = mapped.role_claim_name
&& let Some(v) = claims.get(&claim_name)
&& let Some(arr) = v.as_array()
{
let mut roles = HashSet::new();
for claim in arr {
if let Some(role) = claim.as_str() {
roles.insert(format_role_name_only(role));
}
}
let existing_roles = get_roles_for_org_user(&mapped.org_id, user_email).await;
let mut add_roles = Vec::new();
let mut add_tuples = Vec::new();
let mut remove_tuples = Vec::new();
for role in &existing_roles {
if !roles.contains(role) {
let name = format_role_name(&mapped.org_id, role);
get_user_crole_removal_tuples(user_email, &name, &mut remove_tuples);
}
}
for role in &roles {
if !existing_roles.contains(role) {
add_roles.push(format_role_name(&mapped.org_id, role))
}
}
if !add_roles.is_empty() {
check_and_get_crole_tuple_for_new_user(
user_email,
&mapped.org_id,
add_roles,
mapped.create_missing_roles,
&mut add_tuples,
)
.await;
}
if !add_tuples.is_empty() || !remove_tuples.is_empty() {
if let Err(e) = update_tuples(add_tuples, remove_tuples).await {
log::error!(
"error updating claim based role tuples for user {user_email} org {} : {e}",
mapped.org_id
);
}
}
}
log::info!(
"user {user_email} mapped to org {} successfully via domain org mapping",
mapped.org_id
);
if need_to_add_user {
log::info!(
"user {user_email} mapped to org {} successfully via domain org mapping",
"deleting any invites for {user_email} in org {} as added via org mapping",
mapped.org_id
);
if need_to_add_user {
log::info!(
"deleting any invites for {user_email} in org {} as added via org mapping",
if let Err(e) =
org_invites::delete_invites_for_user(&mapped.org_id, &user_email.to_lowercase())
.await
{
log::error!(
"error in deleting invites for user {user_email} for org {} after joining via domain mapping : {e}",
mapped.org_id
);
if let Err(e) =
org_invites::delete_invites_for_user(&mapped.org_id, &user_email.to_lowercase())
.await
{
log::error!(
"error in deleting invites for user {user_email} for org {} after joining via domain mapping : {e}",
mapped.org_id
);
}
}
log::info!("domain org mapping for user {user_email} successfully processed");
Ok(true)
} else {
log::info!("no domain org mapping found for user {user_email}, continuing normally");
Ok(false)
}
} else {
log::warn!(
"user email {user_email} could not be split correctly at @, skipping org domain mapping"
);
Ok(false)
added += 1;
}
let mut removed = 0;
// if added = 0, it means no viable orgs were there for mapping,
// so removing the user would just keep recreating new orgs at each login
if org_settings.enforce_allowed_orgs_only && !is_new_user && added > 0 {
let orgs = org_users::list_orgs_by_user(user_email).await?;
let mut orgs_to_remove = Vec::new();
for org in orgs {
if !allowed_orgs.contains(&org.org_id) {
orgs_to_remove.push(org.org_id);
}
}
removed = orgs_to_remove.len();
for org in orgs_to_remove {
match users::remove_user_from_org(
&org,
&user_email,
&get_config().auth.root_user_email,
users::RemovalTarget::User,
)
.await
{
Ok(v) => {
if !v.status().is_success() {
log::error!(
"error user {user_email} was domain mapped from org {org_id}, but failed to remove from extra org {org} with status : {}",
v.status().as_str()
);
}
}
Err(e) => log::error!(
"error user {user_email} was domain mapped from org {org_id}, but failed to remove from extra org {org} : {e}"
),
}
}
}
if added == 0 && removed == 0 {
log::info!(
"no mapped orgs were viable, so skipped processing domain mapping for user {user_email} from org {org_id}"
);
return Ok(false);
}
log::info!("domain org mapping for user {user_email} successfully processed");
Ok(true)
}
#[cfg(feature = "cloud")]
@@ -1040,8 +1162,8 @@ pub async fn check_and_add_to_org(
vec![]
}
};
let pending_invites = pending_invites
.into_iter()
let mut has_pending_invites = pending_invites
.iter()
.any(|invite| invite.status == OrgInviteStatus::Pending && invite.expires_at > now);
if db_user.is_none() {
is_new_user = true;
@@ -1069,7 +1191,7 @@ pub async fn check_and_add_to_org(
}
Err(e) => {
log::error!("Error adding user to the database: {}", e);
return Ok((is_new_user, pending_invites));
return Ok((is_new_user, has_pending_invites));
}
}
}
@@ -1078,8 +1200,48 @@ pub async fn check_and_add_to_org(
match process_domain_org_mapping(user_email, claims, is_new_user).await {
Ok(processed) => {
if processed {
// domain based adding is done, so do not treat as a new user
return Ok((false, pending_invites));
let user_record = db::user::get_user_by_email(user_email).await;
if user_record.is_none() {
is_new_user = true;
has_pending_invites = false;
match create_new_user(DBUser {
email: user_email.to_owned(),
first_name: first_name.to_owned(),
last_name: last_name.to_owned(),
password: "".to_owned(),
salt: "".to_owned(),
organizations: vec![], // No org
is_external: true,
password_ext: Some("".to_owned()),
})
.await
{
Ok(_) => {
let mut tuples = vec![];
get_new_user_creation_tuple(user_email, &mut tuples);
tuples_to_add.insert(user_email.to_string(), tuples);
log::info!("User added to the database");
}
Err(e) => {
log::error!(
"Error adding user to the database after domain mapping: {}",
e
);
return Ok((is_new_user, has_pending_invites));
}
}
} else {
// domain based adding is done, so do not treat as a new user
let orgs = org_users::list_orgs_by_user(user_email).await?;
if orgs.is_empty() {
is_new_user = true;
has_pending_invites = pending_invites.iter().any(|invite| {
invite.status == OrgInviteStatus::Pending && invite.expires_at > now
});
} else {
return Ok((false, has_pending_invites));
}
}
}
}
Err(e) => log::error!(
@@ -1108,8 +1270,8 @@ pub async fn check_and_add_to_org(
existing_orgs[0].role.to_string(),
),
// if there are some pending invites, we do not want to create any orgs for the user
_ if pending_invites => {
return Ok((is_new_user, pending_invites));
_ if has_pending_invites => {
return Ok((is_new_user, has_pending_invites));
}
_ => {
// Create a default org for the user
@@ -1209,7 +1371,7 @@ pub async fn check_and_add_to_org(
.await;
}
Ok((is_new_user, pending_invites))
Ok((is_new_user, has_pending_invites))
}
/// Process custom claim parsing and populate source_orgs and custom_roles
@@ -1219,6 +1381,7 @@ pub async fn check_and_add_to_org(
#[cfg(all(feature = "enterprise", not(feature = "cloud")))]
async fn process_custom_claim_parsing(
claims: &HashMap<String, Value>,
org_id: &str,
) -> Option<(Vec<UserOrg>, Vec<String>)> {
use o2_openfga::config::get_config as get_openfga_config;
@@ -1234,8 +1397,9 @@ async fn process_custom_claim_parsing(
// Prepare callback functions for claim parser
let iam_settings_fn = || {
Box::pin(async {
match db::organization::get_org_setting("_meta").await {
let org_id = org_id.to_owned();
Box::pin(async move {
match db::organization::get_org_setting(&org_id).await {
Ok(settings) => settings.claim_parser_function,
Err(_) => "claim_parser".to_string(),
}
@@ -1257,11 +1421,12 @@ async fn process_custom_claim_parsing(
let function_loader_fn = |function_name: &str| {
let function_name = function_name.to_string();
let org_id = org_id.to_owned();
Box::pin(async move {
match db::functions::get("_meta", &function_name).await {
match db::functions::get(&org_id, &function_name).await {
Ok(func) => Ok(func), // Return full Transform object with trans_type
Err(e) => Err(anyhow::anyhow!(
"Failed to load function '{}' from _meta org: {}",
"Failed to load function '{}' from {org_id} org: {}",
function_name,
e
)),
@@ -1279,7 +1444,7 @@ async fn process_custom_claim_parsing(
// Use String as compiled program type for Send compatibility
let vrl_compile_fn = |function_content: &str| -> Result<String, anyhow::Error> {
// Just validate that it compiles, but return the source for execution
transform::compile_vrl_function(function_content, "_meta")
transform::compile_vrl_function(function_content, org_id)
.map_err(|e| anyhow::anyhow!("VRL compilation failed: {}", e))?;
Ok(function_content.to_string())
};
@@ -1289,7 +1454,7 @@ async fn process_custom_claim_parsing(
use vrl::compiler::{TargetValue, runtime::Runtime};
// Compile the VRL function
let vrl_config = transform::compile_vrl_function(function_content, "_meta")
let vrl_config = transform::compile_vrl_function(function_content, org_id)
.map_err(|e| anyhow::anyhow!("VRL compilation failed: {}", e))?;
let mut runtime = Runtime::default();
@@ -1319,12 +1484,12 @@ async fn process_custom_claim_parsing(
let js_execute_fn = |function_content: &str, claims: Value| -> Result<Value, anyhow::Error> {
// Compile the JavaScript function
let js_config = transform::js::compile_js_function(function_content, "_meta")
let js_config = transform::js::compile_js_function(function_content, org_id)
.map_err(|e| anyhow::anyhow!("JavaScript compilation failed: {}", e))?;
// Execute JavaScript with claims as input
let (result, error) =
transform::js::apply_js_fn(&js_config, claims, "_meta", &[String::new()]);
transform::js::apply_js_fn(&js_config, claims, org_id, &[String::new()]);
if let Some(err) = error {
return Err(anyhow::anyhow!("JavaScript execution failed: {}", err));
src/api/http/Cargo.toml
+2 -1
View File
@@ -42,7 +42,8 @@ vectorscan = [
"openobserve-core/vectorscan",
"stream/vectorscan",
]
profiling = ["openobserve-api-management/profiling"]
cpu-profiling = ["openobserve-api-management/cpu-profiling"]
profiling = ["cpu-profiling", "openobserve-api-management/profiling"]
[dependencies]
common.workspace = true
src/api/http/src/handler/http/router/mod.rs
+45 -6
View File
@@ -26,7 +26,9 @@ use axum::{
use config::get_config;
use openobserve_api_common::{X_O2_ASSISTANT_SESSION_ID, X_O2_ASSISTANT_TURN_ID};
use openobserve_api_ingest::request::{clusters, logs, metrics, profiles, rum};
#[cfg(feature = "profiling")]
#[cfg(feature = "cloud")]
use openobserve_api_management::request::org_domains;
#[cfg(feature = "cpu-profiling")]
use openobserve_api_management::request::profiling;
use openobserve_api_management::request::{
alerts, announcements, authz, dashboards, db_monitoring, downtimes, folders, ingest_rejections,
@@ -952,12 +954,14 @@ pub fn basic_routes() -> Router {
router = router.nest("/node", node_routes);
// Debug/profiling routes with auth
#[cfg(feature = "profiling")]
#[cfg(feature = "cpu-profiling")]
{
let debug_routes = Router::new()
let debug_routes = Router::new().route("/profile/cpu", get(profiling::cpu_profile));
#[cfg(feature = "profiling")]
let debug_routes = debug_routes
.route("/profile/memory", get(profiling::memory_profile))
.route("/profile/stats", get(profiling::jemalloc_stats))
.route("/profile/cpu", get(profiling::cpu_profile))
.route("/profile/stats", get(profiling::jemalloc_stats));
let debug_routes = debug_routes
.layer(middleware::from_fn(password_policy_middleware))
.layer(middleware::from_fn(auth_middleware));
@@ -1376,7 +1380,7 @@ pub fn service_routes() -> Router {
.route("/{org_id}/savedviews", get(search::saved_view::get_views).post(search::saved_view::create_view))
.route("/{org_id}/savedviews/{view_id}", get(search::saved_view::get_view).put(search::saved_view::update_view).delete(search::saved_view::delete_view))
.route("/{org_id}/query_history", get(query_history::list).post(query_history::record).layer(DefaultBodyLimit::max(query_history::MAX_BODY_BYTES)))
.route("/{org_id}/query_history", get(query_history::list).post(query_history::record).delete(query_history::clear).layer(DefaultBodyLimit::max(query_history::MAX_BODY_BYTES)))
.route("/{org_id}/query_history/{id}", patch(query_history::star).delete(query_history::delete).layer(DefaultBodyLimit::max(query_history::MAX_BODY_BYTES)))
// Functions
@@ -2171,6 +2175,18 @@ pub fn service_routes() -> Router {
get(billing_group::check_membership).delete(billing_group::leave_group),
)
.route("/{org_id}/billing_group/members", get(billing_group::check_members))
.route(
"/{org_id}/settings/domains",
get(org_domains::get_linked_domain).post(org_domains::link_domain)
)
.route("/{org_id}/settings/domains/requests",post(org_domains::raise_request).get(org_domains::list_requests))
.route("/{org_id}/settings/domain/request/{id}/approve",post(org_domains::approve_request))
.route("/{org_id}/settings/domain/request/{id}/reject",post(org_domains::reject_request))
.route("/{org_id}/settings/domains/{domain}",delete(org_domains::delete_linked_domain))
.route(
"/{org_id}/settings/domains/verify/{domain}",
post(org_domains::verify_domain),
)
.route(
"/{org_id}/billing_group/members/{member_org_id}",
delete(billing_group::remove_member),
@@ -4356,6 +4372,29 @@ mod tests {
);
}
#[test]
fn query_history_list_publishes_both_response_shapes() {
let spec = serde_json::to_value(super::openapi::ApiDoc::openapi()).unwrap();
let ok = &spec["paths"]["/api/{org_id}/query_history"]["get"]["responses"]["200"];
assert_eq!(
ok["content"]["application/json"]["schema"]["$ref"],
"#/components/schemas/QueryHistoryListResponse"
);
let shapes = spec["components"]["schemas"]["QueryHistoryListResponse"]["oneOf"]
.as_array()
.expect("the list response is a oneOf");
let shapes = serde_json::to_string(shapes).unwrap();
assert!(
shapes.contains("#/components/schemas/QueryHistoryEntry"),
"{shapes}"
);
assert!(shapes.contains("\"type\":\"array\""), "{shapes}");
assert!(
shapes.contains("#/components/schemas/QueryHistoryCountResponse"),
"{shapes}"
);
}
#[cfg(feature = "enterprise")]
#[test]
fn experiment_lifecycle_actions_are_published_in_the_openapi_surface() {
src/api/http/src/handler/http/router/openapi.rs
+5
View File
@@ -120,6 +120,7 @@ use crate::{
openobserve_api_search::search::saved_view::update_view,
openobserve_api_management::request::query_history::record,
openobserve_api_management::request::query_history::list,
openobserve_api_management::request::query_history::clear,
openobserve_api_management::request::query_history::star,
openobserve_api_management::request::query_history::delete,
openobserve_api_management::request::folders::delete_folder,
@@ -698,6 +699,9 @@ use crate::{
openobserve_api_management::request::query_history::QueryHistoryRequest,
openobserve_api_management::request::query_history::QueryHistoryStarRequest,
openobserve_api_management::request::query_history::QueryHistoryEntry,
openobserve_api_management::request::query_history::QueryHistoryCountResponse,
openobserve_api_management::request::query_history::QueryHistoryListResponse,
openobserve_api_management::request::query_history::QueryHistoryClearResponse,
meta::user::UpdateUser,
meta::user::UserRoleRequest,
meta::user::PostUserRequest,
@@ -1618,6 +1622,7 @@ mod tests {
for (path, method) in [
("/api/{org_id}/query_history", "post"),
("/api/{org_id}/query_history", "get"),
("/api/{org_id}/query_history", "delete"),
("/api/{org_id}/query_history/{id}", "patch"),
("/api/{org_id}/query_history/{id}", "delete"),
] {
src/api/management/Cargo.toml
+2 -1
View File
@@ -42,7 +42,8 @@ vectorscan = [
"search_service/vectorscan",
"stream/vectorscan",
]
profiling = ["openobserve-core/profiling"]
cpu-profiling = ["openobserve-core/cpu-profiling"]
profiling = ["cpu-profiling", "openobserve-core/profiling"]
[dependencies]
anyhow.workspace = true
src/api/management/src/models/alerts/mod.rs
+6
View File
@@ -241,6 +241,10 @@ pub struct Alert {
#[serde(default)]
#[schema(example = 0)]
pub keep_firing_for: i64,
/// Notify once per state change instead of on every firing evaluation. Enterprise only.
#[serde(default)]
pub suppress_repeats: bool,
}
/// Configuration for when and how an alert should be triggered.
@@ -638,6 +642,7 @@ impl From<(meta_alerts::alert::Alert, Option<Trigger>)> for Alert {
pending_period_sec: alert.pending_period_sec,
notify_on_recovery: alert.notify_on_recovery,
keep_firing_for: alert.keep_firing_for,
suppress_repeats: alert.suppress_repeats,
}
}
}
@@ -847,6 +852,7 @@ impl From<Alert> for meta_alerts::alert::Alert {
alert.notify_on_recovery = value.notify_on_recovery;
alert.recovery_destinations = value.recovery_destinations;
alert.keep_firing_for = value.keep_firing_for.max(0);
alert.suppress_repeats = value.suppress_repeats;
alert.normalize_promql_rule_mode();
alert
src/api/management/src/models/dashboards.rs
+8
View File
@@ -558,6 +558,14 @@ mod tests {
"alert_list_folder": "",
"flame_graph_stack_field": "folded",
"flame_graph_value_field": "value",
"boxplot_whisker": "minmax",
"logs_message_field": "log",
"logs_level_field": "severity",
"logs_time_field": "_timestamp",
"logs_wrap": false,
"logs_prettify_json": true,
"logs_show_time": false,
"logs_order": "newest",
"override_config": [{
"field": {"matchBy": "series", "value": "errors"},
"config": [
src/api/management/src/request/alerts/history.rs
+27 -1
View File
@@ -42,12 +42,13 @@ use crate::common::{
};
/// Columns a stream gains only once a row carried them, so [history_columns] adds each one present.
const OPTIONAL_HISTORY_COLUMNS: [&str; 5] = [
const OPTIONAL_HISTORY_COLUMNS: [&str; 6] = [
"downtime_id",
"acked_by",
"ack_action",
"ack_source",
"taken_over_from",
"repeat_suppressed",
];
/// Every column the data query reads but [OPTIONAL_HISTORY_COLUMNS].
@@ -144,6 +145,9 @@ pub struct AlertHistoryEntry {
pub ack_source: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub taken_over_from: Option<String>,
/// `suppress_repeats` withheld this run because it repeated the last delivered state.
#[serde(skip_serializing_if = "Option::is_none")]
pub repeat_suppressed: Option<bool>,
}
#[derive(Debug, Clone, Serialize, Deserialize, ToSchema)]
@@ -437,6 +441,7 @@ pub async fn get_alert_history(
ack_action: None,
ack_source: None,
taken_over_from: None,
repeat_suppressed: None,
}
})
.collect();
@@ -845,6 +850,7 @@ pub async fn get_alert_history(
ack_action: hit_text(&hit, "ack_action"),
ack_source: hit_text(&hit, "ack_source"),
taken_over_from: hit_text(&hit, "taken_over_from"),
repeat_suppressed: hit.get("repeat_suppressed").and_then(|v| v.as_bool()),
});
}
@@ -1256,6 +1262,7 @@ mod tests {
ack_action: None,
ack_source: None,
taken_over_from: None,
repeat_suppressed: None,
};
assert_eq!(entry.alert_name, "test_alert");
@@ -1315,6 +1322,7 @@ mod tests {
ack_action: None,
ack_source: None,
taken_over_from: None,
repeat_suppressed: None,
};
let response = AlertHistoryResponse {
@@ -1363,6 +1371,7 @@ mod tests {
ack_action: None,
ack_source: None,
taken_over_from: None,
repeat_suppressed: None,
};
assert_eq!(entry.status, "error");
@@ -1593,6 +1602,7 @@ mod tests {
ack_action: None,
ack_source: None,
taken_over_from: None,
repeat_suppressed: None,
};
let json = serde_json::to_string(&entry).unwrap();
@@ -1637,6 +1647,7 @@ mod tests {
ack_action: None,
ack_source: None,
taken_over_from: None,
repeat_suppressed: None,
};
let response = AlertHistoryResponse {
@@ -1688,6 +1699,7 @@ mod tests {
ack_action: None,
ack_source: None,
taken_over_from: None,
repeat_suppressed: None,
};
let json = serde_json::to_value(&entry).unwrap();
let obj = json.as_object().unwrap();
@@ -1729,6 +1741,7 @@ mod tests {
ack_action: None,
ack_source: None,
taken_over_from: None,
repeat_suppressed: None,
};
let json = serde_json::to_value(&entry).unwrap();
let obj = json.as_object().unwrap();
@@ -1826,4 +1839,17 @@ mod tests {
assert_eq!(hit_text(&hit, "acked_by").as_deref(), Some("dana@acme.com"));
assert_eq!(hit_text(&hit, "ack_action"), None);
}
#[test]
fn the_repeat_suppressed_column_is_read_once_the_stream_has_it() {
use arrow_schema::{DataType, Field};
let mut fields: Vec<Field> = HISTORY_COLUMNS
.split(", ")
.map(|name| Field::new(name, DataType::Utf8, true))
.collect();
assert!(!history_columns(&Schema::new(fields.clone())).contains("repeat_suppressed"));
fields.push(Field::new("repeat_suppressed", DataType::Boolean, true));
assert!(history_columns(&Schema::new(fields)).ends_with(", repeat_suppressed"));
}
}
src/api/management/src/request/cloud/org_usage.rs
+2 -7
View File
@@ -18,15 +18,10 @@ use axum::{
response::Response,
};
use hashbrown::HashMap;
use o2_enterprise::enterprise::cloud::{
billing_group,
billings::{self, MeteringProvider},
};
use o2_enterprise::enterprise::cloud::billings::MeteringProvider;
use super::IntoHttpResponse;
use crate::{
common::meta::http::HttpResponse as MetaHttpResponse,
models::billings::GetOrgUsageResponseBody, service::org_usage,
common::meta::http::HttpResponse as MetaHttpResponse, models::billings::GetOrgUsageResponseBody,
};
/// GetUsageData
src/api/management/src/request/fleet/mod.rs
+8 -4
View File
@@ -1609,7 +1609,11 @@ pub async fn list_audit(
if !allowed_on(FLEET_MANAGEMENT, &org_id, &user_email.user_id, "GET").await {
return MetaHttpResponse::forbidden("Forbidden");
}
match openobserve_core::fleet::list_audit(&org_id, query).await {
let permitted = match permitted_fleets(&org_id, &user_email.user_id).await {
Ok(permitted) => permitted,
Err(refused) => return refused,
};
match openobserve_core::fleet::list_audit(&org_id, query, permitted.as_ref()).await {
Ok(mut list) => {
list.total = list.items.len();
MetaHttpResponse::json(list)
@@ -1808,7 +1812,7 @@ async fn desired_request(
}
}
/// D19c: the fleets whose items an org-wide list or aggregate shows; `None` is unfiltered.
/// D19c: the fleets an org-wide list or aggregate shows, whatever the list setting; `None` is all.
#[cfg(feature = "enterprise")]
async fn permitted_fleets(
org_id: &str,
@@ -1821,7 +1825,7 @@ async fn permitted_fleets(
if let Some(objects) = tests::listed_objects(org_id, user_id) {
return Ok(permitted_from(org_id, key, objects));
}
match db::authz::list_objects_for_user(org_id, user_id, "GET", key).await {
match db::authz::list_permitted_objects(org_id, user_id, "GET", key).await {
Ok(objects) => Ok(permitted_from(org_id, key, objects)),
Err(e) => Err(MetaHttpResponse::forbidden(e)),
}
@@ -2272,7 +2276,7 @@ mod tests {
assert_eq!(
permitted_fleets(ORG, custom).await.ok(),
Some(None),
"OpenFGA off, or the list flag off: unfiltered"
"OpenFGA off: unfiltered"
);
for (listed, want) in [
(None, None),
src/api/management/src/request/mod.rs
+4 -1
View File
@@ -58,7 +58,7 @@ pub mod organization;
pub mod password_policy;
#[cfg(feature = "enterprise")]
pub mod playground;
#[cfg(feature = "profiling")]
#[cfg(feature = "cpu-profiling")]
pub mod profiling;
#[cfg(feature = "enterprise")]
pub mod prompts;
@@ -90,4 +90,7 @@ pub mod users;
#[cfg(feature = "enterprise")]
pub mod workflows;
#[cfg(feature = "cloud")]
pub mod org_domains;
use openobserve_api_common::request::{BulkDeleteRequest, BulkDeleteResponse};
src/api/management/src/request/org_domains/mod.rs
+341
View File
@@ -0,0 +1,341 @@
// Copyright 2026 OpenObserve Inc.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use axum::{Json, extract::Path, response::Response};
use common::meta::http::HttpResponse as MetaHttpResponse;
use config::{META_ORG_ID, meta::status_pages::CreateDomainRequest, utils::str::is_valid_domain};
use infra::table::org_domain_requests::Status;
use openobserve_api_common::extractors::Headers;
use openobserve_core::{org_domain_ownership, org_domain_requests};
use serde::Deserialize;
use crate::service::auth::UserEmail;
#[derive(Deserialize)]
pub struct RequestDecision {
reason: String,
}
pub async fn get_linked_domain(Path(org_id): Path<String>) -> Response {
match org_domain_ownership::get_domains_for_org(&org_id).await {
Ok(v) => MetaHttpResponse::json(v),
Err(e) => {
log::error!("error listing domains for org {org_id} : {e}");
MetaHttpResponse::internal_error(format!("error listing domains : {e}"))
}
}
}
pub async fn link_domain(
Path(org_id): Path<String>,
Headers(user): Headers<UserEmail>,
Json(body): Json<CreateDomainRequest>,
) -> Response {
let domain = body.domain.trim().to_lowercase();
if !is_valid_domain(&domain) {
return MetaHttpResponse::bad_request(format!(
"domain {domain} is not a valid domain to link"
));
}
match org_domain_ownership::get_verified_org_for_domain(&domain).await {
Ok(Some(v)) => {
log::info!(
"org {org_id} tried to link domain {domain} but it is already verified by org {v}"
);
let org_info = if v == org_id {
"to this org"
} else {
"by some other org"
};
return MetaHttpResponse::bad_request(format!(
"this domain is already linked {org_info}"
));
}
Err(e) => {
return MetaHttpResponse::internal_error(format!(
"error getting org domain linking : {e}"
));
}
_ => {}
}
let existing = match org_domain_requests::get_req_by_org_domain(&org_id, &domain).await {
Ok(v) => v,
Err(e) => {
return MetaHttpResponse::internal_error(format!(
"error getting org domain requests : {e}"
));
}
};
if existing.is_some() {
return MetaHttpResponse::bad_request(
"there is already a request open for this domain by this org",
);
}
let attempt = match org_domain_ownership::get_domain_org_record(&org_id, &domain).await {
Ok(v) => v,
Err(e) => {
return MetaHttpResponse::internal_error(format!(
"error getting org domain requests : {e}"
));
}
};
if attempt.is_some() {
return MetaHttpResponse::bad_request(format!(
"there is already a linking created for this domain by this org"
));
}
match org_domain_ownership::save_org_domain_mapping(&org_id, &domain).await {
Ok(_) => {
log::info!(
"successfully created org domain ownership entry for {org_id} domain {domain} by user {}",
user.user_id
);
MetaHttpResponse::ok("successfully created mapping request")
}
Err(e) => {
log::error!("error saving domain {domain} link to org {org_id} : {e}",);
MetaHttpResponse::internal_error(format!("error saving org domain linking : {e}"))
}
}
}
pub async fn raise_request(
Path(org_id): Path<String>,
Headers(user): Headers<UserEmail>,
Json(body): Json<CreateDomainRequest>,
) -> Response {
let domain = body.domain.trim().to_lowercase();
if !is_valid_domain(&domain) {
return MetaHttpResponse::bad_request(format!("domain {domain} is not a valid domain",));
}
match org_domain_ownership::get_verified_org_for_domain(&domain).await {
Ok(Some(v)) => {
log::info!(
"org {org_id} tried to link domain {domain} but it is already verified by org {v}"
);
return MetaHttpResponse::bad_request("this domain is already linked by some org");
}
Err(e) => {
return MetaHttpResponse::internal_error(format!(
"error getting org domain linking : {e}"
));
}
_ => {}
}
match org_domain_requests::get_req_by_org_domain(&org_id, &domain).await {
Err(e) => {
log::error!("error getting org domain request for {org_id} domain {domain} : {e}");
return MetaHttpResponse::internal_error(e);
}
Ok(Some(v)) => {
return MetaHttpResponse::bad_request(format!(
"org {org_id} already has a request for domain {domain} with status {:?}",
Status::from(v.status)
));
}
_ => {}
}
match org_domain_requests::save_org_domain_request(&org_id, &domain, &user.user_id).await {
Ok(_) => {
log::info!(
"successfully created org domain ownership request for {org_id} domain {domain} by user {}",
user.user_id
);
MetaHttpResponse::ok("successfully created ownership request")
}
Err(e) => {
log::error!("error saving domain {domain} link request to org {org_id} : {e}",);
MetaHttpResponse::internal_error(format!(
"error saving org domain ownership request : {e}"
))
}
}
}
pub async fn list_requests(Path(org_id): Path<String>) -> Response {
let requests = if org_id == META_ORG_ID {
org_domain_requests::list_all_req().await
} else {
org_domain_requests::list_by_org(&org_id).await
};
match requests {
Ok(v) => MetaHttpResponse::json(v),
Err(e) => {
log::error!("error listing org domain mapping requests for org {org_id} : {e}");
MetaHttpResponse::internal_error(e)
}
}
}
pub async fn approve_request(
Path((org_id, id)): Path<(String, String)>,
Headers(user): Headers<UserEmail>,
Json(body): Json<RequestDecision>,
) -> Response {
if org_id != META_ORG_ID {
return MetaHttpResponse::forbidden("operation not allowed");
}
let reason = body.reason.trim().to_string();
let req = match org_domain_requests::get_req_by_id(&id).await {
Ok(Some(v)) => v,
Ok(None) => {
return MetaHttpResponse::not_found("request with given id not found");
}
Err(e) => {
log::error!("error getting org domain request by id {id} for approval : {e}");
return MetaHttpResponse::internal_error(e);
}
};
if req.status != Status::Pending as i32 {
return MetaHttpResponse::bad_request("This request is not pending, cannot be approved");
}
match org_domain_ownership::get_verified_org_for_domain(&req.domain).await {
Ok(Some(v)) => {
return MetaHttpResponse::bad_request(format!(
"this domain is already linked to org {v}, request cannot be processed"
));
}
Err(e) => {
return MetaHttpResponse::internal_error(format!(
"error getting org domain linking : {e}"
));
}
_ => {}
}
if let Err(e) = org_domain_ownership::link_from_meta(&req.org_id, &req.domain).await {
log::error!("error linking the domain to org after request approval : {e}");
return MetaHttpResponse::internal_error(format!(
"error linking domain from req {} : {e}",
req.id
));
}
match org_domain_requests::update_status(&id, &user.user_id, Status::Approved, reason).await {
Ok(_) => MetaHttpResponse::ok("request approved successfully"),
Err(e) => {
log::error!("error approving domain ownership request {id} : {e}",);
MetaHttpResponse::internal_error(e)
}
}
}
pub async fn reject_request(
Path((org_id, id)): Path<(String, String)>,
Headers(user): Headers<UserEmail>,
Json(body): Json<RequestDecision>,
) -> Response {
if org_id != META_ORG_ID {
return MetaHttpResponse::forbidden("operation not allowed");
}
let reason = body.reason.trim().to_string();
let req = match org_domain_requests::get_req_by_id(&id).await {
Ok(Some(v)) => v,
Ok(None) => {
return MetaHttpResponse::not_found("request with given id not found");
}
Err(e) => {
log::error!("error getting org domain request by id {id} for approval : {e}");
return MetaHttpResponse::internal_error(e);
}
};
if req.status != Status::Pending as i32 {
return MetaHttpResponse::bad_request("This request is not pending, cannot be rejected");
}
if let Err(e) =
org_domain_requests::update_status(&id, &user.user_id, Status::Rejected, reason).await
{
log::error!("error rejecting domain ownership request {id} : {e}",);
return MetaHttpResponse::internal_error(format!(
"error approving org domain ownership request : {e}"
));
}
log::info!(
"org domain request {id} for org {} successfully rejected by user {}",
req.org_id,
user.user_id
);
MetaHttpResponse::ok("request rejected successfully")
}
pub async fn delete_linked_domain(
Path((org_id, did)): Path<(String, String)>,
Headers(user): Headers<UserEmail>,
) -> Response {
let domain = did.trim().to_lowercase();
let record = match org_domain_ownership::get_domain_org_record(&org_id, &domain).await {
Ok(v) => v,
Err(e) => {
return MetaHttpResponse::internal_error(format!(
"error getting org domain linking : {e}"
));
}
};
let Some(model) = record else {
return MetaHttpResponse::bad_request("org domain link not found");
};
match org_domain_ownership::delete_linked_domain(model).await {
Ok(_) => {
log::info!(
"removed domain link of {domain} from org {org_id} by user {}",
user.user_id
);
MetaHttpResponse::ok("successfully deleted mapping")
}
Err(e) => {
log::error!(
"error removing domain link of {domain} from org {org_id} by user {} : {e}",
user.user_id
);
MetaHttpResponse::internal_error(format!("error deleting org domain linking : {e}"))
}
}
}
pub async fn verify_domain(
Path((org_id, did)): Path<(String, String)>,
Headers(user): Headers<UserEmail>,
) -> Response {
let domain = did.trim().to_lowercase();
match org_domain_ownership::verify_domain_now(&org_id, &domain).await {
Ok(record) => MetaHttpResponse::json(record),
Err(e) => {
log::error!(
"error verifying domain {domain} for org {org_id} by user {}: {e}",
user.user_id
);
MetaHttpResponse::bad_request(e)
}
}
}
src/api/management/src/request/organization/billing_group.rs
+34
View File
@@ -316,6 +316,23 @@ pub async fn remove_member(
if let Err(e) = billing_group::remove_billing_member(&org_id, &member_org_id).await {
return HttpResponse::bad_request(format!("error in removing member: {e}"));
}
let org_id_copy = member_org_id.clone();
if let Err(e) = db::organization::update_org_setting(&org_id, move |settings| {
let before = settings.domain_org_mappings.len();
settings
.domain_org_mappings
.retain(|m| m.org_id != member_org_id);
settings.domain_org_mappings.len() != before
})
.await
{
log::error!(
"error updating domain mapping for {org_id_copy} after removing the membership from {org_id} : {e}"
);
};
HttpResponse::ok("successfully removed")
}
@@ -366,6 +383,23 @@ pub async fn leave_group(
if let Err(e) = billing_group::remove_billing_member(&membership.payer_org_id, &org_id).await {
return HttpResponse::bad_request(format!("error in leaving billing group: {e}"));
}
let org_id_copy = org_id.clone();
if let Err(e) =
db::organization::update_org_setting(&membership.payer_org_id, move |settings| {
let before = settings.domain_org_mappings.len();
settings.domain_org_mappings.retain(|m| m.org_id != org_id);
settings.domain_org_mappings.len() != before
})
.await
{
log::error!(
"error updating domain mapping for {} after removing the membership of {org_id_copy} : {e}",
&membership.payer_org_id
);
};
HttpResponse::ok("successfully left")
}
src/api/management/src/request/organization/settings.rs
+104 -21
View File
@@ -19,6 +19,8 @@ use axum::{
http::StatusCode,
response::{IntoResponse, Response},
};
#[cfg(feature = "cloud")]
use config::utils::str::is_valid_domain;
use db::organization::{get_org_setting, set_org_setting};
use infra::errors::{DbError, Error};
#[cfg(feature = "enterprise")]
@@ -28,7 +30,6 @@ use {
};
#[cfg(feature = "cloud")]
use {
config::META_ORG_ID,
o2_enterprise::enterprise::cloud::{
billing_group::list_billing_group_members_of, billings::MeteringProvider, customer_billings,
},
@@ -78,13 +79,19 @@ pub async fn create(
#[cfg(feature = "cloud")]
let settings = match check_domain_org_mappings(&org_id, settings).await {
Ok(settings) => settings,
Err(refusal) => return refusal,
Err(refusal) => return MetaHttpResponse::bad_request(refusal),
};
#[cfg(feature = "cloud")]
if let Err(response) = check_budget_config(&org_id, &settings).await {
return response;
}
#[cfg(feature = "cloud")]
let settings = match check_domain_management_config(settings).await {
Ok(settings) => settings,
Err(msg) => return MetaHttpResponse::bad_request(msg),
};
match db::organization::update_org_setting(&org_id, |data| {
apply_settings_payload(data, settings)
})
@@ -262,64 +269,130 @@ fn validate_settings_payload(settings: &OrganizationSettingPayload) -> Result<()
async fn check_domain_org_mappings(
org_id: &str,
mut settings: OrganizationSettingPayload,
) -> Result<OrganizationSettingPayload, Response> {
) -> Result<OrganizationSettingPayload, String> {
use hashbrown::HashSet;
let Some(mut mappings) = settings.domain_org_mappings.take() else {
return Ok(settings);
};
if org_id != META_ORG_ID {
return Ok(settings);
}
let child_orgs =
o2_enterprise::enterprise::cloud::billing_group::list_billing_group_members_of(&org_id)
.await
.unwrap_or_default();
let mut allowed_orgs: HashSet<_> = child_orgs.into_iter().map(|v| v.member_org_id).collect();
allowed_orgs.insert(org_id.to_string());
let mut seen_domains = HashSet::new();
for mapping in &mut mappings {
use openobserve_core::org_domain_ownership;
let domain = mapping.domain.trim().to_lowercase();
if !is_valid_domain(&domain) {
return Err(format!("domain {domain} is not a valid domain",));
}
if seen_domains.contains(&(domain.clone(), mapping.org_id.clone())) {
return Err(format!(
"domain {domain} is linked to {} multiple times",
mapping.org_id
));
}
seen_domains.insert((domain.clone(), mapping.org_id.clone()));
if !allowed_orgs.contains(&mapping.org_id) {
return Err(format!(
"Not allowed to setup a mapping for org {}",
mapping.org_id
));
}
if openobserve_core::organization::get_org(&mapping.org_id)
.await
.is_none()
{
return Err(format!("No org with org id {} found", mapping.org_id));
}
// A deleted org keeps its row as a tombstone, so the row alone does not make it a target.
match db::org_status::read_active(&mapping.org_id).await {
Ok(true) => {}
Ok(false) => return Err(inactive_mapping_target(&mapping.org_id).await),
Err(e) => {
return Err(MetaHttpResponse::internal_error(format!(
return Err(format!(
"cannot read the status of org {}: {e}",
mapping.org_id
)));
));
}
}
if mapping.domain.is_empty() || mapping.domain.contains(' ') {
return Err(MetaHttpResponse::bad_request(format!(
if domain.is_empty() || domain.contains(' ') {
return Err(format!(
"domain cannot have space or be empty, bad domain '{}'",
mapping.domain
)));
));
}
if !matches!(
mapping.base_role.as_str(),
"admin" | "editor" | "viewer" | "allowed_user"
) {
return Err(MetaHttpResponse::bad_request(format!(
return Err(format!(
"base role {} for org {} is not a valid base role - only admin, editor, viewer, allowed_user are supported",
mapping.base_role, mapping.org_id
)));
));
}
if let Some(claim) = mapping.role_claim_name.as_mut() {
if claim.trim().is_empty() {
return Err(MetaHttpResponse::bad_request(
"role claim name must not be empty",
));
return Err("role claim name must not be empty".into());
}
*claim = claim.trim().to_string();
}
mapping.domain = mapping.domain.to_lowercase();
match org_domain_ownership::get_domain_org_record(&org_id, &domain).await {
Ok(None) => {
return Err("This domain is not linked yet".to_string());
}
Err(e) => {
return Err(format!("error getting domain mappings : {e}"));
}
Ok(Some(v)) => {
use infra::table::org_domain_ownership::OwnershipState;
if v.verification_state != OwnershipState::Verified as i32 {
return Err("This domain is not verified yet".to_string());
}
}
}
mapping.domain = domain;
}
settings.domain_org_mappings = Some(mappings);
Ok(settings)
}
#[cfg(feature = "cloud")]
async fn check_domain_management_config(
mut settings: OrganizationSettingPayload,
) -> Result<OrganizationSettingPayload, String> {
if let Some(mut config) = settings.domain_management_config {
config.blocked_emails = config
.blocked_emails
.into_iter()
.map(|v| v.trim().to_lowercase())
.collect();
settings.domain_management_config = Some(config);
}
Ok(settings)
}
/// The refusal for a mapping whose target org is not active: missing, blocked, or unreadable.
#[cfg(feature = "cloud")]
async fn inactive_mapping_target(org_id: &str) -> Response {
async fn inactive_mapping_target(org_id: &str) -> String {
match infra::table::organizations::find(org_id).await {
Ok(None) => MetaHttpResponse::bad_request(format!("No org with org id {org_id} found")),
Ok(Some(_)) => MetaHttpResponse::bad_request(format!("org {org_id} is not active")),
Err(e) => MetaHttpResponse::internal_error(format!("cannot read org {org_id}: {e}")),
Ok(None) => format!("No org with org id {org_id} found"),
Ok(Some(_)) => format!("org {org_id} is not active"),
Err(e) => format!("cannot read org {org_id}: {e}"),
}
}
@@ -449,6 +522,16 @@ fn apply_settings_payload(
field_found = true;
data.budget_config = configs;
}
#[cfg(feature = "cloud")]
if let Some(enforce) = settings.enforce_allowed_orgs_only {
field_found = true;
data.enforce_allowed_orgs_only = enforce;
}
#[cfg(feature = "cloud")]
if let Some(config) = settings.domain_management_config {
field_found = true;
data.domain_management_config = config;
}
// this will always be taken from orgs table, never from settings
data.free_trial_expiry = None;
field_found
src/api/management/src/request/profiling.rs
+8 -3
View File
@@ -15,11 +15,14 @@
//! HTTP handlers for profiling endpoints
use axum::{Json, extract::Query};
#[cfg(feature = "profiling")]
use axum::Json;
use axum::extract::Query;
use config::get_config;
use openobserve_core::self_profiles::dump_cpu_profile;
#[cfg(feature = "profiling")]
use openobserve_core::self_profiles::{
dump_cpu_profile, dump_memory_flamegraph, dump_memory_pprof,
jemalloc_stats as read_jemalloc_stats,
dump_memory_flamegraph, dump_memory_pprof, jemalloc_stats as read_jemalloc_stats,
};
use serde::Deserialize;
@@ -29,6 +32,7 @@ use serde::Deserialize;
/// The profile can be analyzed with jeprof or converted to flamegraph.
///
/// Returns: Binary profile data
#[cfg(feature = "profiling")]
pub async fn memory_profile() -> Result<String, String> {
let pprof_data = dump_memory_pprof().await?;
let timestamp = chrono::Utc::now().format("%Y%m%d_%H%M%S");
@@ -58,6 +62,7 @@ pub async fn memory_profile() -> Result<String, String> {
/// - allocated: Current allocated memory (bytes)
/// - resident: Current resident memory (bytes)
/// - metadata: Metadata overhead (bytes)
#[cfg(feature = "profiling")]
pub async fn jemalloc_stats() -> Result<Json<serde_json::Value>, String> {
Ok(Json(read_jemalloc_stats()?))
}
src/api/management/src/request/public_dashboards/public.rs
+27 -2
View File
@@ -19,7 +19,7 @@ use std::{sync::LazyLock, time::Duration};
use axum::{
Json,
extract::{Extension, Path, Query},
extract::{Extension, Path, Query, rejection::QueryRejection},
http::{HeaderMap, HeaderValue, StatusCode, header},
response::{IntoResponse, Response},
};
@@ -182,7 +182,7 @@ pub async fn config(ip: Option<Extension<RealIp>>, Path(slug): Path<String>) ->
pub async fn data(
ip: Option<Extension<RealIp>>,
Path(slug): Path<String>,
Query(params): Query<DataParams>,
query: Result<Query<DataParams>, QueryRejection>,
headers: HeaderMap,
) -> Response {
let if_none_match = headers
@@ -191,6 +191,9 @@ pub async fn data(
if rate_limited(ip.map(|e| e.0)) || shed_early(&slug, if_none_match) {
return with_headers(too_many_requests(READ_WINDOW));
}
let Ok(Query(params)) = query else {
return with_headers(invalid_range());
};
with_headers(serve_data(&slug, params.range.as_deref(), if_none_match).await)
}
@@ -430,6 +433,11 @@ fn expired() -> Response {
(StatusCode::GONE, "expired").into_response()
}
// The extractor's own text names serde internals, e.g. a repeated `range`.
fn invalid_range() -> Response {
(StatusCode::BAD_REQUEST, "invalid range").into_response()
}
// A failed read is temporary, so the viewer retries instead of treating the link as gone.
fn server_error() -> Response {
(StatusCode::INTERNAL_SERVER_ERROR, "Something went wrong.").into_response()
@@ -833,6 +841,7 @@ mod tests {
StatusCode::OK.into_response(),
StatusCode::NOT_FOUND.into_response(),
expired(),
invalid_range(),
unavailable(),
too_many_requests(READ_WINDOW),
server_error(),
@@ -846,6 +855,22 @@ mod tests {
}
}
#[tokio::test]
async fn a_query_the_extractor_refuses_is_a_plain_invalid_range() {
let uri: axum::http::Uri = "/data?range=r3600&range=r60".parse().unwrap();
let resp = data(
None,
Path("Qz8Zt2mR7aB3cD4eF5gH6Q".into()),
Query::try_from_uri(&uri),
HeaderMap::new(),
)
.await;
assert_eq!(resp.status(), StatusCode::BAD_REQUEST);
assert_eq!(resp.headers()[header::CACHE_CONTROL], "no-store");
let body = axum::body::to_bytes(resp.into_body(), 1024).await.unwrap();
assert_eq!(&body[..], b"invalid range");
}
#[test]
fn gate12_a_missing_settings_row_means_enabled_and_a_disabled_row_stops_serving() {
assert_eq!(org_gate(None), (true, 0));
src/api/management/src/request/query_history.rs
+622 -98
View File
@@ -21,7 +21,10 @@ use axum::{
};
use infra::table::{
entity::query_history::Model,
query_history::{self, MAX_STARRED_PER_USER, SetStarred},
query_history::{
self, DEFAULT_SOURCE, ListFilter, MAX_STARRED_PER_USER, NewRun, SOURCES, STATUS_ERROR,
STATUS_OK, SetStarred,
},
};
use openobserve_api_common::extractors::Headers;
use openobserve_core::auth::UserEmail;
@@ -34,14 +37,27 @@ use crate::common::meta::http::HttpResponse as MetaHttpResponse;
pub const MAX_BODY_BYTES: usize = 128 * 1024;
const MAX_QUERY_BYTES: usize = 16 * 1024;
const MAX_CONTEXT_BYTES: usize = 64 * 1024;
const MAX_STREAMS: usize = 50;
const DEFAULT_LIMIT: u64 = 50;
const MAX_LIMIT: u64 = 200;
#[derive(Deserialize, Serialize, ToSchema)]
pub struct QueryHistoryRequest {
/// `logs`, `traces`, `metrics`, `dbm` or `profiles`; defaults to `metrics`.
#[serde(default)]
pub source: Option<String>,
/// The query text, at most 16 KB and not blank.
pub query: String,
/// Time range, step, chart type and the panel data that reopens the query, at most 64 KB.
/// Runs with the same source and key merge into one entry; defaults to the query text.
#[serde(default)]
pub query_key: Option<String>,
/// Streams the query ran on, at most 50; each name non-empty and without a comma.
#[serde(default)]
pub streams: Vec<String>,
/// `ok` or `error`; defaults to `ok`.
#[serde(default)]
pub status: Option<String>,
/// What the page needs to restore the run, at most 64 KB; stored as given.
pub context: serde_json::Value,
}
@@ -55,24 +71,43 @@ pub struct QueryHistoryStarRequest {
pub struct QueryHistoryEntry {
/// Entry id.
pub id: String,
/// Page the query ran on.
pub source: String,
/// The query text.
pub query: String,
/// Time range, step, chart type and the panel data that reopens the query.
/// Hex sha256 of the merge key; null on entries recorded before keys existed.
pub query_key: Option<String>,
/// Streams the query ran on.
pub streams: Vec<String>,
/// What the page needs to restore the run.
pub context: serde_json::Value,
/// `ok` or `error`, from the latest run; `ok` on entries recorded by v1.1.x.
pub status: String,
/// Starred entries are kept until deleted.
pub starred: bool,
/// When the query was last run, in microseconds since the epoch.
/// How many runs merged into this entry; 1 on entries recorded by v1.1.x.
pub run_count: i64,
/// First run, in microseconds since the epoch; the last run on entries recorded by v1.1.x.
pub created_at: i64,
/// Latest run, in microseconds since the epoch.
pub updated_at: i64,
}
impl From<Model> for QueryHistoryEntry {
fn from(m: Model) -> Self {
Self {
streams: query_history::decode_streams(&m.streams),
id: m.id,
source: m.source,
query: m.query,
query_key: m.query_key,
context: m.context,
status: m.status,
starred: m.starred,
run_count: m.run_count,
created_at: m.created_at,
// A v1.1.x node in a rolling upgrade writes updated_at = 0; only the response is fixed.
updated_at: m.updated_at.max(m.created_at),
}
}
}
@@ -80,18 +115,55 @@ impl From<Model> for QueryHistoryEntry {
#[derive(Deserialize, IntoParams)]
#[into_params(parameter_in = Query)]
pub struct QueryHistoryListParams {
/// Comma list of sources (logs, traces, metrics, dbm, profiles); omitted means metrics.
pub source: Option<String>,
/// `ok` or `error`.
pub status: Option<String>,
/// Exact stream name the entry ran on.
pub stream: Option<String>,
/// Case-insensitive substring of the query text or the stream names.
pub q: Option<String>,
/// Only starred (`true`) or only unstarred (`false`) entries.
pub starred: Option<bool>,
/// Substring match on the query text.
pub q: Option<String>,
/// Entries last run at or after this time, in microseconds.
pub ran_from: Option<i64>,
/// Entries last run at or before this time, in microseconds.
pub ran_to: Option<i64>,
/// Defaults to 50, at most 200.
pub limit: Option<u64>,
/// Entries to skip, for paging. Defaults to 0.
/// Entries to skip, for paging. Defaults to 0, at most 9223372036854775807.
pub offset: Option<u64>,
/// Answer `{"count": n}`, the number of entries the filters match, instead of the entries.
pub count_only: Option<bool>,
}
fn list_limit(limit: Option<u64>) -> u64 {
limit.unwrap_or(DEFAULT_LIMIT).clamp(1, MAX_LIMIT)
#[derive(Deserialize, IntoParams)]
#[into_params(parameter_in = Query)]
pub struct QueryHistoryClearParams {
/// One source or a comma-separated list; all sources when omitted.
pub source: Option<String>,
/// Also delete starred entries. Defaults to false.
pub include_starred: Option<bool>,
}
#[derive(Deserialize, Serialize, ToSchema)]
pub struct QueryHistoryCountResponse {
/// Entries matching the filters, ignoring limit and offset.
pub count: u64,
}
/// The entries, or with `count_only=true` how many entries match.
#[derive(Serialize, ToSchema)]
#[serde(untagged)]
pub enum QueryHistoryListResponse {
Entries(Vec<QueryHistoryEntry>),
Count(QueryHistoryCountResponse),
}
#[derive(Deserialize, Serialize, ToSchema)]
pub struct QueryHistoryClearResponse {
/// Number of entries deleted.
pub deleted: u64,
}
/// RecordQueryHistory
@@ -103,7 +175,7 @@ fn list_limit(limit: Option<u64>) -> u64 {
tag = "Query History",
operation_id = "RecordQueryHistory",
summary = "Record a query history entry",
description = "Records a query the caller ran. Running the same query as the caller's latest entry refreshes that entry instead of adding one. Stores nothing and answers 204 when ZO_QUERY_HISTORY_ENABLED is false.",
description = "Records a query the caller ran. A run with the same source and query_key as an existing entry of the caller merges into it: the run count goes up and the latest run time, context and status replace the old ones. Stores nothing and answers 204 when ZO_QUERY_HISTORY_ENABLED is false.",
security(
("Authorization"= [])
),
@@ -114,7 +186,7 @@ fn list_limit(limit: Option<u64>) -> u64 {
responses(
(status = 200, description = "Success", content_type = "application/json", body = QueryHistoryEntry),
(status = 204, description = "Query history is disabled; nothing was stored"),
(status = 400, description = "Query empty, or query or context too large", content_type = "application/json", body = ()),
(status = 400, description = "Unknown source or status, query empty, query or context too large, too many streams, or an empty stream name or one with a comma (the message lists them)", content_type = "application/json", body = ()),
(status = 413, description = "Request body larger than 128 KB"),
(status = 500, description = "Failure", content_type = "application/json", body = ()),
),
@@ -146,7 +218,7 @@ pub async fn record(
tag = "Query History",
operation_id = "ListQueryHistory",
summary = "List query history",
description = "Lists the caller's query history, newest first. Empty when ZO_QUERY_HISTORY_ENABLED is false.",
description = "Lists the caller's query history, most recently run first. With count_only=true it answers {\"count\": n} (QueryHistoryCountResponse), the number of entries the filters match. Empty, or a count of 0, when ZO_QUERY_HISTORY_ENABLED is false.",
security(
("Authorization"= [])
),
@@ -155,7 +227,8 @@ pub async fn record(
QueryHistoryListParams,
),
responses(
(status = 200, description = "Success", content_type = "application/json", body = Vec<QueryHistoryEntry>),
(status = 200, description = "The entries, or with count_only=true the count", content_type = "application/json", body = QueryHistoryListResponse),
(status = 400, description = "Unknown source or status, or an offset above 9223372036854775807", content_type = "application/json", body = ()),
(status = 500, description = "Failure", content_type = "application/json", body = ()),
),
extensions(
@@ -177,6 +250,49 @@ pub async fn list(
.await
}
/// ClearQueryHistory
#[utoipa::path(
delete,
path = "/{org_id}/query_history",
context_path = "/api",
tag = "Query History",
operation_id = "ClearQueryHistory",
summary = "Clear the caller's query history",
description = "Deletes the caller's query history entries for the given sources, or all sources. Starred entries are kept unless include_starred is true.",
security(
("Authorization"= [])
),
params(
("org_id" = String, Path, description = "Organization name"),
QueryHistoryClearParams,
),
responses(
(status = 200, description = "Success", content_type = "application/json", body = QueryHistoryClearResponse),
(status = 400, description = "Unknown source", content_type = "application/json", body = ()),
(status = 500, description = "Failure", content_type = "application/json", body = ()),
),
extensions(
("x-o2-ratelimit" = json!({"module": "Query History", "operation": "delete"})),
("x-o2-mcp" = json!({"enabled": false}))
)
)]
pub async fn clear(
Path(org_id): Path<String>,
Headers(user_email): Headers<UserEmail>,
Query(params): Query<QueryHistoryClearParams>,
) -> Response {
let sources = match parse_sources(params.source.as_deref()) {
Ok(sources) => sources,
Err(e) => return MetaHttpResponse::bad_request(e),
};
let include_starred = params.include_starred.unwrap_or(false);
match query_history::clear(&org_id, &user_email.user_id, &sources, include_starred).await {
Ok(deleted) => MetaHttpResponse::json(QueryHistoryClearResponse { deleted }),
Err(e) => MetaHttpResponse::internal_error(e),
}
}
/// StarQueryHistory
#[utoipa::path(
@@ -186,7 +302,7 @@ pub async fn list(
tag = "Query History",
operation_id = "StarQueryHistory",
summary = "Star or unstar a query history entry",
description = "Starred entries are kept until deleted. A user can star at most 200 entries per organization.",
description = "Starred entries are kept until deleted. A user can star at most 200 entries per organization, shared across every source (logs, traces, metrics, dbm, profiles).",
security(
("Authorization"= [])
),
@@ -213,10 +329,10 @@ pub async fn star(
Json(req): Json<QueryHistoryStarRequest>,
) -> Response {
match query_history::set_starred(&org_id, &user_email.user_id, &id, req.starred).await {
Ok(SetStarred::Updated(entry)) => MetaHttpResponse::json(QueryHistoryEntry::from(entry)),
Ok(SetStarred::Updated(entry)) => MetaHttpResponse::json(QueryHistoryEntry::from(*entry)),
Ok(SetStarred::NotFound) => MetaHttpResponse::not_found("query history entry not found"),
Ok(SetStarred::StarredCapReached) => MetaHttpResponse::bad_request(format!(
"at most {MAX_STARRED_PER_USER} query history entries can be starred; unstar one first"
"at most {MAX_STARRED_PER_USER} query history entries can be starred across all sources; unstar one first"
)),
Err(e) => MetaHttpResponse::internal_error(e),
}
@@ -260,6 +376,81 @@ pub async fn delete(
}
}
fn list_limit(limit: Option<u64>) -> u64 {
limit.unwrap_or(DEFAULT_LIMIT).clamp(1, MAX_LIMIT)
}
/// The databases bind OFFSET as a signed 64-bit integer.
fn list_offset(offset: Option<u64>) -> Result<u64, String> {
let offset = offset.unwrap_or(0);
if offset > i64::MAX as u64 {
return Err(format!("offset exceeds {}", i64::MAX));
}
Ok(offset)
}
/// Splits a comma list of sources; empty means all, an unknown name is an error.
fn parse_sources(sources: Option<&str>) -> Result<Vec<String>, String> {
let mut parsed = Vec::new();
for source in sources.unwrap_or("").split(',').map(str::trim) {
if source.is_empty() {
continue;
}
if !SOURCES.contains(&source) {
return Err(format!("unknown source: {source}"));
}
parsed.push(source.to_string());
}
Ok(parsed)
}
fn parse_status(status: Option<&str>) -> Result<Option<String>, String> {
match status.map(str::trim).filter(|s| !s.is_empty()) {
None => Ok(None),
Some(s) if s == STATUS_OK || s == STATUS_ERROR => Ok(Some(s.to_string())),
Some(s) => Err(format!("unknown status: {s}")),
}
}
/// The source and status a record stores, or why the request is refused.
fn validate_record(req: &QueryHistoryRequest) -> Result<(String, String), String> {
let source = req
.source
.as_deref()
.map(str::trim)
.filter(|s| !s.is_empty())
.unwrap_or(DEFAULT_SOURCE);
if !SOURCES.contains(&source) {
return Err(format!("unknown source: {source}"));
}
let status = parse_status(req.status.as_deref())?.unwrap_or_else(|| STATUS_OK.to_string());
if req.query.trim().is_empty() {
return Err("query is empty".to_string());
}
if req.query.len() > MAX_QUERY_BYTES {
return Err("query exceeds 16 KB".to_string());
}
if req.context.to_string().len() > MAX_CONTEXT_BYTES {
return Err("context exceeds 64 KB".to_string());
}
if req.streams.len() > MAX_STREAMS {
return Err(format!("at most {MAX_STREAMS} streams"));
}
let invalid: Vec<String> = req
.streams
.iter()
.filter(|s| s.is_empty() || s.contains(','))
.map(|s| format!("{s:?}"))
.collect();
if !invalid.is_empty() {
return Err(format!(
"stream names must be non-empty and contain no comma: {}",
invalid.join(", ")
));
}
Ok((source.to_string(), status))
}
async fn record_entry(
enabled: bool,
org_id: &str,
@@ -269,24 +460,22 @@ async fn record_entry(
if !enabled {
return StatusCode::NO_CONTENT.into_response();
}
if req.query.trim().is_empty() {
return MetaHttpResponse::bad_request("query is empty");
}
if req.query.len() > MAX_QUERY_BYTES {
return MetaHttpResponse::bad_request("query exceeds 16 KB");
}
if req.context.to_string().len() > MAX_CONTEXT_BYTES {
return MetaHttpResponse::bad_request("context exceeds 64 KB");
}
match query_history::record(
let (source, status) = match validate_record(&req) {
Ok(valid) => valid,
Err(e) => return MetaHttpResponse::bad_request(e),
};
let run = NewRun {
org_id,
user_email,
&req.query,
req.context,
config::utils::time::now_micros(),
)
.await
{
source: &source,
query: &req.query,
query_key: req.query_key.as_deref(),
streams: &req.streams,
status: &status,
context: req.context,
now_us: config::utils::time::now_micros(),
};
match query_history::record(run).await {
Ok(entry) => MetaHttpResponse::json(QueryHistoryEntry::from(entry)),
Err(e) => MetaHttpResponse::internal_error(e),
}
@@ -298,45 +487,81 @@ async fn list_entries(
user_email: &str,
params: QueryHistoryListParams,
) -> Response {
let count_only = params.count_only.unwrap_or(false);
if !enabled {
return MetaHttpResponse::json(Vec::<QueryHistoryEntry>::new());
return MetaHttpResponse::json(if count_only {
QueryHistoryListResponse::Count(QueryHistoryCountResponse { count: 0 })
} else {
QueryHistoryListResponse::Entries(Vec::new())
});
}
let (filter, offset) =
match list_filter(&params).and_then(|f| Ok((f, list_offset(params.offset)?))) {
Ok(valid) => valid,
Err(e) => return MetaHttpResponse::bad_request(e),
};
if count_only {
return match query_history::count(org_id, user_email, &filter).await {
Ok(count) => {
MetaHttpResponse::json(QueryHistoryListResponse::Count(QueryHistoryCountResponse {
count,
}))
}
Err(e) => MetaHttpResponse::internal_error(e),
};
}
match query_history::list(
org_id,
user_email,
params.starred,
params.q.as_deref(),
&filter,
list_limit(params.limit),
params.offset.unwrap_or(0),
offset,
)
.await
{
Ok(entries) => MetaHttpResponse::json(
entries
.into_iter()
.map(QueryHistoryEntry::from)
.collect::<Vec<_>>(),
),
Ok(entries) => MetaHttpResponse::json(QueryHistoryListResponse::Entries(
entries.into_iter().map(QueryHistoryEntry::from).collect(),
)),
Err(e) => MetaHttpResponse::internal_error(e),
}
}
fn list_filter(params: &QueryHistoryListParams) -> Result<ListFilter, String> {
// A v1.1.x client sends no source and can only load Metrics entries.
let sources = parse_sources(params.source.as_deref())?;
let sources = if sources.is_empty() {
vec![DEFAULT_SOURCE.to_string()]
} else {
sources
};
Ok(ListFilter {
sources,
status: parse_status(params.status.as_deref())?,
stream: params.stream.clone(),
q: params.q.clone(),
starred: params.starred,
ran_from: params.ran_from,
ran_to: params.ran_to,
})
}
#[cfg(test)]
mod tests {
use infra::table::entity::query_history::Entity;
use sea_orm::{ConnectionTrait, Schema};
use serde_json::{Value, json};
use super::*;
const ALL: &str = "source=logs,traces,metrics,dbm,profiles";
async fn setup() {
infra::db::create_table().await.unwrap();
let conn = infra::db::get_orm_client_rw().await;
let backend = conn.get_database_backend();
let mut stmt = Schema::new(backend).create_table_from_entity(Entity);
conn.execute(backend.build(stmt.if_not_exists()))
.await
.unwrap();
static TABLE: tokio::sync::OnceCell<()> = tokio::sync::OnceCell::const_new();
TABLE
.get_or_init(|| async {
infra::db::create_table().await.unwrap();
let conn = infra::db::get_orm_client_rw().await;
query_history::create_table_for_tests(conn).await.unwrap();
})
.await;
}
fn user() -> String {
@@ -349,6 +574,17 @@ mod tests {
})
}
fn request(query: &str, context: Value) -> QueryHistoryRequest {
QueryHistoryRequest {
source: None,
query: query.to_string(),
query_key: None,
streams: Vec::new(),
status: None,
context,
}
}
async fn parts(resp: Response) -> (StatusCode, Value) {
let status = resp.status();
let bytes = axum::body::to_bytes(resp.into_body(), usize::MAX)
@@ -360,23 +596,26 @@ mod tests {
)
}
async fn post(org: &str, user: &str, query: &str, context: Value) -> (StatusCode, Value) {
let req = QueryHistoryRequest {
query: query.to_string(),
context,
};
async fn post_req(org: &str, user: &str, req: QueryHistoryRequest) -> (StatusCode, Value) {
parts(record(Path(org.to_string()), headers(user), Json(req)).await).await
}
async fn post(org: &str, user: &str, query: &str, context: Value) -> (StatusCode, Value) {
post_req(org, user, request(query, context)).await
}
async fn get(org: &str, user: &str, params: &str) -> Vec<Value> {
let params: QueryHistoryListParams = parse_params(params);
let (status, body) =
parts(list(Path(org.to_string()), headers(user), Query(params)).await).await;
assert_eq!(status, StatusCode::OK);
let (status, body) = get_raw(org, user, params).await;
assert_eq!(status, StatusCode::OK, "{body}");
body.as_array().unwrap().clone()
}
fn parse_params(params: &str) -> QueryHistoryListParams {
async fn get_raw(org: &str, user: &str, params: &str) -> (StatusCode, Value) {
let params: QueryHistoryListParams = parse_params(params);
parts(list(Path(org.to_string()), headers(user), Query(params)).await).await
}
fn parse_params<T: serde::de::DeserializeOwned>(params: &str) -> T {
let uri: axum::http::Uri = format!("/?{params}").parse().unwrap();
Query::try_from_uri(&uri).unwrap().0
}
@@ -392,6 +631,17 @@ mod tests {
parts(delete(path, headers(user)).await).await.0
}
async fn clear_all(org: &str, user: &str, params: &str) -> (StatusCode, Value) {
let params: QueryHistoryClearParams = parse_params(params);
parts(clear(Path(org.to_string()), headers(user), Query(params)).await).await
}
fn queries(rows: Vec<Value>) -> Vec<String> {
rows.iter()
.map(|e| e["query"].as_str().unwrap().to_string())
.collect()
}
#[tokio::test]
async fn test_another_users_entry_is_not_found() {
setup().await;
@@ -438,73 +688,317 @@ mod tests {
}
#[tokio::test]
async fn test_rerun_updates_the_latest_entry_context() {
async fn test_an_old_client_without_source_or_key_merges_by_query_into_metrics() {
setup().await;
let a = user();
post("default", &a, "up", json!({"chart": "line"})).await;
post("default", &a, "up", json!({"chart": "bar"})).await;
let (_, entry) = post("default", &a, "up", json!({"chart": "bar"})).await;
assert_eq!(entry["source"], "metrics");
assert_eq!(entry["status"], "ok");
assert_eq!(entry["run_count"], 2);
assert_eq!(entry["streams"], json!([]));
assert_eq!(entry["query_key"].as_str().unwrap().len(), 64);
let listed = get("default", &a, "").await;
assert_eq!(listed.len(), 1);
assert_eq!(listed[0]["context"]["chart"], "bar");
assert!(listed[0]["updated_at"].as_i64() >= listed[0]["created_at"].as_i64());
}
#[tokio::test]
async fn test_oversized_query_or_context_is_rejected() {
async fn test_a_v1_1_row_without_updated_at_answers_its_created_at() {
use sea_orm::ConnectionTrait;
setup().await;
let a = user();
let long_query = "x".repeat(MAX_QUERY_BYTES + 1);
let id = config::ider::uuid();
infra::db::get_orm_client_rw()
.await
.execute_unprepared(&format!(
"INSERT INTO query_history (id, org_id, user_email, query, context, created_at, \
updated_at, source) VALUES ('{id}', 'default', '{a}', 'up', '{{}}', 500, 0, \
'metrics')"
))
.await
.unwrap();
let listed = get("default", &a, "").await;
assert_eq!(
post("default", &a, &long_query, json!({})).await.0,
StatusCode::BAD_REQUEST
(
listed[0]["created_at"].as_i64(),
listed[0]["updated_at"].as_i64()
),
(Some(500), Some(500))
);
let big = json!({"d": "x".repeat(MAX_CONTEXT_BYTES)});
assert_eq!(
post("default", &a, "up", big).await.0,
StatusCode::BAD_REQUEST
);
let max_query = "x".repeat(MAX_QUERY_BYTES);
assert_eq!(
post("default", &a, &max_query, json!({})).await.0,
StatusCode::OK
);
assert_eq!(get("default", &a, "").await.len(), 1);
let (status, starred) = patch("default", &a, &id, true).await;
assert_eq!(status, StatusCode::OK);
assert_eq!(starred["updated_at"], 500);
assert_eq!(starred["created_at"], 500);
}
#[tokio::test]
async fn test_blank_query_is_rejected() {
async fn test_repeats_of_one_search_stay_one_entry() {
setup().await;
let a = user();
for query in ["", " ", "\n\t "] {
for _ in 0..3 {
let req = QueryHistoryRequest {
source: Some("logs".to_string()),
query_key: Some("k8s|sql|up|".to_string()),
streams: vec!["k8s".to_string()],
..request("up", json!({}))
};
assert_eq!(post_req("default", &a, req).await.0, StatusCode::OK);
}
let rows = get("default", &a, "source=logs").await;
assert_eq!(rows.len(), 1);
assert_eq!(rows[0]["run_count"], 3);
assert_eq!(rows[0]["streams"], json!(["k8s"]));
}
#[tokio::test]
async fn test_query_key_is_a_hash_per_user_and_falls_back_to_the_query() {
setup().await;
let (a, b) = (user(), user());
let keyed = |key: Option<&str>| QueryHistoryRequest {
source: Some("logs".to_string()),
query_key: key.map(str::to_string),
..request("up", json!({}))
};
let (_, first) = post_req("default", &a, keyed(Some("k8s|sql|up|"))).await;
let key = first["query_key"].as_str().unwrap();
assert_eq!(key.len(), 64);
assert!(key.chars().all(|c| c.is_ascii_hexdigit()), "{key}");
assert_ne!(key, "k8s|sql|up|");
let (_, other_user) = post_req("default", &b, keyed(Some("k8s|sql|up|"))).await;
assert_eq!(other_user["query_key"], first["query_key"]);
assert_ne!(other_user["id"], first["id"], "user_email is in the key");
assert_eq!(other_user["run_count"], 1);
let (_, no_key) = post_req("default", &a, keyed(None)).await;
let (_, query_as_key) = post_req("default", &a, keyed(Some("up"))).await;
assert_eq!(query_as_key["id"], no_key["id"], "no key hashes the query");
assert_eq!(query_as_key["run_count"], 2);
assert_eq!(get("default", &a, "source=logs").await.len(), 2);
}
#[test]
fn test_invalid_stream_names_are_listed_in_the_error() {
let req = QueryHistoryRequest {
streams: vec!["ok".to_string(), "a,b".to_string(), String::new()],
..request("up", json!({}))
};
assert_eq!(
validate_record(&req).unwrap_err(),
r#"stream names must be non-empty and contain no comma: "a,b", """#
);
}
#[tokio::test]
async fn test_record_validation() {
setup().await;
let a = user();
let refused = [
QueryHistoryRequest {
source: Some("dashboards".to_string()),
..request("up", json!({}))
},
QueryHistoryRequest {
status: Some("timeout".to_string()),
..request("up", json!({}))
},
QueryHistoryRequest {
streams: (0..=MAX_STREAMS).map(|i| format!("s{i}")).collect(),
..request("up", json!({}))
},
QueryHistoryRequest {
streams: vec!["a,b".to_string()],
..request("up", json!({}))
},
request(&"x".repeat(MAX_QUERY_BYTES + 1), json!({})),
request("up", json!({"d": "x".repeat(MAX_CONTEXT_BYTES)})),
request("", json!({})),
request(" \n\t ", json!({})),
];
for req in refused {
assert_eq!(
post("default", &a, query, json!({})).await.0,
StatusCode::BAD_REQUEST,
"{query:?}"
post_req("default", &a, req).await.0,
StatusCode::BAD_REQUEST
);
}
assert!(get("default", &a, "").await.is_empty());
let accepted = QueryHistoryRequest {
source: Some("traces".to_string()),
status: Some("error".to_string()),
streams: (0..MAX_STREAMS).map(|i| format!("s{i}")).collect(),
..request(&"x".repeat(MAX_QUERY_BYTES), json!({}))
};
let (status, entry) = post_req("default", &a, accepted).await;
assert_eq!(status, StatusCode::OK, "{entry}");
assert_eq!(entry["status"], "error");
assert_eq!(entry["source"], "traces");
}
#[tokio::test]
async fn test_list_applies_filters() {
setup().await;
let a = user();
for query in ["up", "rate(http_total[5m])", "sum(http_total)"] {
post("default", &a, query, json!({})).await;
for (source, query, stream, status) in [
("metrics", "up", None, "ok"),
("logs", "level = 'error'", Some("k8s_logs"), "error"),
("traces", "service = 'cart'", Some("default"), "ok"),
("metrics", "sum(http_total)", None, "ok"),
] {
let req = QueryHistoryRequest {
source: Some(source.to_string()),
streams: stream.map(|s| vec![s.to_string()]).unwrap_or_default(),
status: Some(status.to_string()),
..request(query, json!({}))
};
post_req("default", &a, req).await;
}
let queries = |rows: Vec<Value>| {
rows.iter()
.map(|e| e["query"].as_str().unwrap().to_string())
.collect::<Vec<_>>()
};
assert_eq!(
queries(get("default", &a, "q=http_total&limit=1").await),
queries(get("default", &a, "source=logs,traces").await),
vec!["service = 'cart'", "level = 'error'"]
);
assert_eq!(
queries(get("default", &a, &format!("{ALL}&status=error")).await),
vec!["level = 'error'"]
);
assert_eq!(
queries(get("default", &a, &format!("{ALL}&stream=k8s_logs")).await),
vec!["level = 'error'"]
);
assert_eq!(
queries(get("default", &a, &format!("{ALL}&q=K8S")).await),
vec!["level = 'error'"]
);
assert_eq!(
queries(get("default", &a, "").await),
vec!["sum(http_total)", "up"],
"a client that sends no source is a v1.1.x Metrics page"
);
assert_eq!(
queries(get("default", &a, "source=metrics&limit=1").await),
vec!["sum(http_total)"]
);
assert_eq!(
queries(get("default", &a, "q=http_total&offset=1").await),
vec!["rate(http_total[5m])"]
queries(get("default", &a, "source=metrics&offset=1").await),
vec!["up"]
);
assert!(
get("default", &a, &format!("{ALL}&ran_from=1&ran_to=2"))
.await
.is_empty()
);
assert_eq!(
get("default", &a, &format!("{ALL}&ran_from=1")).await.len(),
4
);
assert!(
get("default", &a, &format!("{ALL}&starred=true"))
.await
.is_empty()
);
assert_eq!(
get_raw("default", &a, "source=dashboards").await.0,
StatusCode::BAD_REQUEST
);
assert_eq!(
get_raw("default", &a, "status=partial").await.0,
StatusCode::BAD_REQUEST
);
}
#[tokio::test]
async fn test_count_only_answers_the_count_for_the_filters() {
setup().await;
let a = user();
for (source, query) in [
("logs", "a"),
("logs", "b"),
("traces", "c"),
("metrics", "d"),
] {
let req = QueryHistoryRequest {
source: Some(source.to_string()),
..request(query, json!({}))
};
post_req("default", &a, req).await;
}
let starred = get("default", &a, "source=logs").await[0]["id"]
.as_str()
.unwrap()
.to_string();
patch("default", &a, &starred, true).await;
let count = |params: &str| {
let a = a.clone();
let params = format!("count_only=true&{params}");
async move { get_raw("default", &a, &params).await }
};
assert_eq!(
count("source=logs").await,
(StatusCode::OK, json!({"count": 2}))
);
assert_eq!(
count("source=logs&starred=false").await.1,
json!({"count": 1})
);
assert_eq!(
count(&format!("{ALL}&starred=true")).await.1,
json!({"count": 1})
);
assert_eq!(count(ALL).await.1, json!({"count": 4}));
assert_eq!(count("source=logs&limit=1").await.1, json!({"count": 2}));
assert_eq!(count("source=nope").await.0, StatusCode::BAD_REQUEST);
let disabled = list_entries(
false,
"default",
&a,
parse_params("count_only=true&source=logs"),
)
.await;
assert_eq!(parts(disabled).await, (StatusCode::OK, json!({"count": 0})));
}
#[tokio::test]
async fn test_clear_my_history() {
setup().await;
let (a, b) = (user(), user());
for (source, query) in [
("logs", "a"),
("logs", "b"),
("traces", "c"),
("metrics", "d"),
] {
let req = QueryHistoryRequest {
source: Some(source.to_string()),
..request(query, json!({}))
};
post_req("default", &a, req).await;
}
post("default", &b, "up", json!({})).await;
let starred = get("default", &a, "source=logs").await[0]["id"]
.as_str()
.unwrap()
.to_string();
patch("default", &a, &starred, true).await;
let (status, body) = clear_all("default", &a, "source=logs").await;
assert_eq!(status, StatusCode::OK);
assert_eq!(body, json!({"deleted": 1}));
assert_eq!(clear_all("default", &a, "").await.1, json!({"deleted": 2}));
assert_eq!(queries(get("default", &a, ALL).await), vec!["b"]);
assert_eq!(
clear_all("default", &a, "include_starred=true").await.1,
json!({"deleted": 1})
);
assert!(get("default", &a, ALL).await.is_empty());
assert_eq!(get("default", &b, ALL).await.len(), 1);
assert_eq!(
clear_all("default", &a, "source=nope").await.0,
StatusCode::BAD_REQUEST
);
assert!(get("default", &a, "starred=true").await.is_empty());
}
#[tokio::test]
@@ -512,10 +1006,7 @@ mod tests {
setup().await;
let a = user();
post("default", &a, "up", json!({})).await;
let req = QueryHistoryRequest {
query: "rate(x[5m])".to_string(),
context: json!({}),
};
let req = request("rate(x[5m])", json!({}));
let (status, body) = parts(record_entry(false, "default", &a, req).await).await;
assert_eq!(status, StatusCode::NO_CONTENT);
assert_eq!(body, Value::Null, "204 carries no body");
@@ -535,4 +1026,37 @@ mod tests {
assert_eq!(list_limit(Some(500)), 200);
assert_eq!(list_limit(Some(0)), 1);
}
#[tokio::test]
async fn test_an_offset_past_i64_max_is_a_bad_request() {
setup().await;
let a = user();
post("default", &a, "up", json!({})).await;
let max = i64::MAX as u64;
let (status, _) = get_raw("default", &a, &format!("offset={}", max + 1)).await;
assert_eq!(status, StatusCode::BAD_REQUEST);
let (status, _) = get_raw(
"default",
&a,
&format!("offset={}&count_only=true", u64::MAX),
)
.await;
assert_eq!(status, StatusCode::BAD_REQUEST);
assert!(
get("default", &a, &format!("offset={max}"))
.await
.is_empty()
);
assert_eq!(list_offset(None), Ok(0));
}
#[test]
fn test_sources_parse_as_a_comma_list_of_known_pages() {
assert_eq!(parse_sources(None), Ok(vec![]));
assert_eq!(
parse_sources(Some("logs, traces,")),
Ok(vec!["logs".to_string(), "traces".to_string()])
);
assert!(parse_sources(Some("logs,alerts")).is_err());
}
}
src/api/management/src/request/status/mod.rs
+22 -2
View File
@@ -249,6 +249,10 @@ struct ConfigResponse<'a> {
synthetics_recorder_extension_url: String,
/// The only DBM flag (`ZO_DB_MONITORING_ENABLED`): one OSS switch for every signal.
database_monitoring_enabled: bool,
/// Off hides recent searches in the UI; the server then records nothing.
query_history_enabled: bool,
/// Days unstarred recent searches are kept; 0 or less keeps them until the per-page cap.
query_history_retention_days: i64,
enable_cross_linking: bool,
show_fts_field_values: bool,
search_inspector_enabled: bool,
@@ -668,6 +672,8 @@ pub async fn zo_config(
synthetics_browser_max_steps,
synthetics_recorder_extension_url: synthetics_recorder_extension_url.to_string(),
database_monitoring_enabled: cfg.db_monitoring.enabled,
query_history_enabled: cfg.limit.query_history_enabled,
query_history_retention_days: cfg.limit.query_history_retention_days,
enable_cross_linking: cfg.common.enable_cross_linking,
show_fts_field_values: cfg.common.show_fts_field_values,
search_inspector_enabled,
@@ -1286,11 +1292,11 @@ pub async fn redirect(
Ok(v) => v.map(|(new_user, pending_invites)| {
format!("&new_user_login={new_user}&pending_invites={pending_invites}")
}),
Err(_) => {
Err(e) => {
return Response::builder()
.status(StatusCode::UNAUTHORIZED)
.header(header::CONTENT_TYPE, "application/json")
.body(Body::from("\"Email Domain not allowed\""))
.body(Body::from(e.to_string()))
.unwrap();
}
};
@@ -2563,6 +2569,20 @@ mod tests {
assert_eq!(json["public_dashboards_enabled"], true);
}
#[tokio::test]
async fn test_zo_config_serves_query_history_settings() {
let cfg = get_config();
let json = zo_config_json().await;
assert_eq!(
json["query_history_enabled"],
cfg.limit.query_history_enabled
);
assert_eq!(
json["query_history_retention_days"],
cfg.limit.query_history_retention_days
);
}
#[tokio::test]
async fn test_zo_config_reports_slack_per_org() {
let reason = if cfg!(feature = "enterprise") {
src/common/src/meta/organization.rs
+34
View File
@@ -14,6 +14,8 @@
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use config::{meta::user::UserRole, stats::MemorySize};
#[cfg(feature = "cloud")]
use o2_enterprise::enterprise::domain_management::meta::DomainManagementConfig;
use serde::{Deserialize, Serialize};
use utoipa::ToSchema;
@@ -539,8 +541,16 @@ pub struct OrganizationSettingPayload {
pub claim_parser_function: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub cross_links: Option<Vec<config::meta::stream::CrossLink>>,
#[cfg(feature = "cloud")]
#[serde(skip_serializing_if = "Option::is_none")]
pub domain_org_mappings: Option<Vec<DomainOrgMapping>>,
#[cfg(feature = "cloud")]
#[serde(default)]
pub domain_management_config: Option<DomainManagementConfig>,
#[cfg(feature = "cloud")]
#[serde(default)]
pub enforce_allowed_orgs_only: Option<bool>,
#[cfg(feature = "cloud")]
#[serde(skip_serializing_if = "Option::is_none")]
pub budget_config: Option<Vec<BudgetNotificationConfiguration>>,
}
@@ -592,6 +602,12 @@ pub struct OrganizationSetting {
pub domain_org_mappings: Vec<DomainOrgMapping>,
#[cfg(feature = "cloud")]
#[serde(default)]
pub enforce_allowed_orgs_only: bool,
#[cfg(feature = "cloud")]
#[serde(default)]
pub domain_management_config: DomainManagementConfig,
#[cfg(feature = "cloud")]
#[serde(default)]
pub budget_config: Vec<BudgetNotificationConfiguration>,
}
@@ -631,6 +647,10 @@ impl Default for OrganizationSetting {
#[cfg(feature = "cloud")]
domain_org_mappings: Vec::new(),
#[cfg(feature = "cloud")]
domain_management_config: Default::default(),
#[cfg(feature = "cloud")]
enforce_allowed_orgs_only: false,
#[cfg(feature = "cloud")]
budget_config: Vec::new(),
}
}
@@ -1474,7 +1494,13 @@ mod tests {
#[cfg(feature = "enterprise")]
claim_parser_function: None,
cross_links: None,
#[cfg(feature = "cloud")]
domain_org_mappings: None,
#[cfg(feature = "cloud")]
domain_management_config: None,
#[cfg(feature = "cloud")]
enforce_allowed_orgs_only: None,
#[cfg(feature = "cloud")]
budget_config: None,
};
let json = serde_json::to_value(&payload).unwrap();
@@ -1558,6 +1584,10 @@ mod tests {
#[cfg(feature = "cloud")]
domain_org_mappings: vec![],
#[cfg(feature = "cloud")]
domain_management_config: Default::default(),
#[cfg(feature = "cloud")]
enforce_allowed_orgs_only: false,
#[cfg(feature = "cloud")]
budget_config: vec![],
};
let json = serde_json::to_value(&setting).unwrap();
@@ -1592,6 +1622,10 @@ mod tests {
#[cfg(feature = "cloud")]
domain_org_mappings: vec![],
#[cfg(feature = "cloud")]
domain_management_config: Default::default(),
#[cfg(feature = "cloud")]
enforce_allowed_orgs_only: false,
#[cfg(feature = "cloud")]
budget_config: vec![],
};
let json = serde_json::to_value(&setting).unwrap();
src/config/src/config.rs
+55 -7
View File
@@ -126,7 +126,10 @@ pub type RwBTreeMap<K, V> = tokio::sync::RwLock<BTreeMap<K, V>>;
// 126: add held_action to slack_link_requests.
// 127: add group_label_set to alert_states.
// 128: widen alert_dedup_state.fingerprint to varchar(256).
pub const DB_SCHEMA_VERSION: u64 = 128;
// 129: add source, query_key, streams, run_count, updated_at and status to query_history.
// 130: add suppress_repeats to alerts.
// 131: org domain mapping migrations
pub const DB_SCHEMA_VERSION: u64 = 131;
pub const DB_SCHEMA_KEY: &str = "/db_schema_version/";
// global version variables
@@ -526,6 +529,11 @@ pub const SYNTHETICS_RELOAD_CLASSES: &[(&str, SyntheticsReloadClass)] = &[
"ZO_SYNTHETICS_ORPHAN_DETECTION_ENABLED",
SyntheticsReloadClass::Hot,
),
("ZO_SYNTHETICS_LEASE_BATCH_SIZE", SyntheticsReloadClass::Hot),
(
"ZO_SYNTHETICS_MAX_INFLIGHT_PER_TICK",
SyntheticsReloadClass::Hot,
),
// Were `HotViaLimits` while these lived in `o2_enterprise` and had to be
// pushed across a seam into this crate. They are declared here now, so the
// seam and its class are both gone and they are plainly `Hot`.
@@ -580,6 +588,8 @@ pub(crate) fn synthetics_restart_required_changes(
devices: _,
scheduler_jitter_enabled: _,
orphan_detection_enabled: _,
lease_batch_size: _,
max_inflight_per_tick: _,
} = new;
let mut changed = Vec::new();
@@ -1126,14 +1136,14 @@ impl PublicAiChat {
}
}
/// Background self CPU/memory profile ingest into `_meta.self_profiles`.
/// Gated at runtime by `enabled`; sampling code is compile-gated on `profiling`.
/// Background self CPU/memory profile ingest into the `default` profiles stream.
/// Gated at runtime by `enabled`; CPU needs `cpu-profiling`, memory needs `profiling`.
#[derive(Debug, Serialize, EnvConfig, Default)]
pub struct SelfProfiles {
#[env_config(
name = "ZO_SELF_PROFILES_ENABLED",
default = false,
help = "Enable background self CPU/memory profile sampling into _meta.self_profiles"
help = "Enable background self CPU/memory profile sampling into the default profiles stream"
)]
pub enabled: bool,
#[env_config(
@@ -1493,6 +1503,23 @@ pub struct Synthetics {
help = "Kill switch for orphan detection, which reports enabled checks no scheduler is claiming."
)]
pub orphan_detection_enabled: bool,
/// Jobs leased per pool in one dispatcher lease call. Must be > 0; a
/// non-positive value falls back to the default. Read per tick.
#[env_config(
name = "ZO_SYNTHETICS_LEASE_BATCH_SIZE",
default = 10,
help = "Jobs the dispatcher leases per pool per lease call. Must be greater than 0."
)]
pub lease_batch_size: i64,
/// Upper bound on jobs the dispatcher leases across all pools in one tick.
/// While a pool's batch comes back full the dispatcher keeps leasing, up to
/// this cap. Must be > 0; a non-positive value falls back to the default.
#[env_config(
name = "ZO_SYNTHETICS_MAX_INFLIGHT_PER_TICK",
default = 100,
help = "Cap on jobs the dispatcher leases across all pools in one 2s tick. Must be greater than 0."
)]
pub max_inflight_per_tick: i64,
}
/// Feature 5 — SLO measurement (`alerts_2.md` §6b).
@@ -2911,7 +2938,7 @@ pub struct Limit {
#[env_config(
name = "ZO_QUERY_HISTORY_RETENTION_DAYS",
default = 14,
help = "How long unstarred query history entries are kept, in days. Starred entries are kept until deleted. 0 or less keeps every entry forever; it does not stop collection, ZO_QUERY_HISTORY_ENABLED=false does."
help = "How long unstarred query history entries are kept, in days. Starred entries are kept until deleted. 0 or less removes the age limit; each user still keeps at most 500 unstarred entries per source (logs, traces, metrics, dbm, profiles), the oldest of that source pruned first. It does not stop collection, ZO_QUERY_HISTORY_ENABLED=false does."
)]
pub query_history_retention_days: i64,
#[env_config(name = "ZO_ALERT_SCHEDULE_TIMEOUT", default = 90)] // seconds
@@ -5270,6 +5297,21 @@ fn check_synthetics_config(cfg: &mut Config) {
cfg.synthetics.max_net_timeout_ms = crate::meta::synthetics::DEFAULT_MAX_NET_TIMEOUT_MS;
}
if cfg.synthetics.lease_batch_size <= 0 {
log::error!(
"ZO_SYNTHETICS_LEASE_BATCH_SIZE ({}) must be greater than 0, using 10",
cfg.synthetics.lease_batch_size
);
cfg.synthetics.lease_batch_size = 10;
}
if cfg.synthetics.max_inflight_per_tick <= 0 {
log::error!(
"ZO_SYNTHETICS_MAX_INFLIGHT_PER_TICK ({}) must be greater than 0, using 100",
cfg.synthetics.max_inflight_per_tick
);
cfg.synthetics.max_inflight_per_tick = 100;
}
// Every synthetics var now has exactly one name, in every build — which
// means an OSS user can set a private-agent var and get silence back. One
// line per var is the difference between "configured and ignored" and
@@ -5450,6 +5492,8 @@ mod tests {
"ZO_SYNTHETICS_DEVICES",
"ZO_SYNTHETICS_SCHEDULER_JITTER_ENABLED",
"ZO_SYNTHETICS_ORPHAN_DETECTION_ENABLED",
"ZO_SYNTHETICS_LEASE_BATCH_SIZE",
"ZO_SYNTHETICS_MAX_INFLIGHT_PER_TICK",
];
fn synthetics_names_in_class(class: SyntheticsReloadClass) -> Vec<&'static str> {
@@ -5469,8 +5513,8 @@ mod tests {
fn synthetics_reload_classification_is_pinned() {
assert_eq!(
SYNTHETICS_RELOAD_CLASSES.len(),
16,
"Synthetics has 16 keys; every one needs a reload class"
18,
"Synthetics has 18 keys; every one needs a reload class"
);
let mut classified: Vec<&str> = SYNTHETICS_RELOAD_CLASSES
@@ -5502,7 +5546,9 @@ mod tests {
"ZO_SYNTHETICS_JOB_LEASE_SECS",
"ZO_SYNTHETICS_LAMBDA_BROWSER",
"ZO_SYNTHETICS_LAMBDA_NET",
"ZO_SYNTHETICS_LEASE_BATCH_SIZE",
"ZO_SYNTHETICS_MAX_CHECK_BUDGET_SECS",
"ZO_SYNTHETICS_MAX_INFLIGHT_PER_TICK",
"ZO_SYNTHETICS_MAX_NET_TIMEOUT_MS",
"ZO_SYNTHETICS_ORPHAN_DETECTION_ENABLED",
"ZO_SYNTHETICS_RECORDER_EXTENSION_URL",
@@ -5565,6 +5611,8 @@ mod tests {
cfg.devices = "desktop:800:600".to_string();
cfg.scheduler_jitter_enabled = !cfg.scheduler_jitter_enabled;
cfg.orphan_detection_enabled = !cfg.orphan_detection_enabled;
cfg.lease_batch_size += 1;
cfg.max_inflight_per_tick += 1;
}
#[test]
src/config/src/meta/alerts/alert.rs
+28
View File
@@ -165,6 +165,10 @@ pub struct Alert {
/// today. Named after Prometheus' `keep_firing_for`, with the same meaning.
#[serde(default)]
pub keep_firing_for: i64,
/// Notify once per state change; enterprise only, other builds ignore it.
#[serde(default)]
pub suppress_repeats: bool,
}
/// Accept a runbook link, or say why not.
@@ -266,11 +270,17 @@ impl Default for Alert {
notify_on_recovery: false,
recovery_destinations: vec![],
keep_firing_for: 0,
suppress_repeats: false,
}
}
}
impl Alert {
/// Whether repeat notifications are withheld; other builds than enterprise ignore the switch.
pub fn suppresses_repeats(&self) -> bool {
cfg!(feature = "enterprise") && self.suppress_repeats
}
/// Pending rule mode needs per-series state; the default count gate means any series.
pub fn normalize_promql_rule_mode(&mut self) {
if self.query_condition.query_type != super::QueryType::PromQL
@@ -1233,4 +1243,22 @@ mod tests {
let json = serde_json::to_value(&alert).unwrap();
assert!(json.get("runbook_url").is_none());
}
#[cfg(not(feature = "enterprise"))]
#[test]
fn repeat_suppression_is_ignored_outside_enterprise() {
let alert = Alert {
suppress_repeats: true,
..Default::default()
};
assert!(!alert.suppresses_repeats());
}
#[test]
fn an_alert_saved_before_repeat_suppression_reads_with_it_off() {
let mut stored = serde_json::to_value(Alert::default()).unwrap();
stored.as_object_mut().unwrap().remove("suppress_repeats");
let alert: Alert = serde_json::from_value(stored).unwrap();
assert!(!alert.suppress_repeats);
}
}
src/config/src/meta/alerts/dispatch.rs
+309 -52
View File
@@ -112,6 +112,8 @@ pub struct DispatchPlan {
pub inconsistent: Vec<String>,
/// Groups that would deliver but a downtime mutes; also counted in `suppressed`.
pub muted: Vec<String>,
/// Groups withheld because they repeat their last delivery; also counted in `suppressed`.
pub repeats: usize,
}
/// Decide which groups this evaluation notifies (MN-1, MN-8).
@@ -135,6 +137,7 @@ pub struct DispatchPlan {
/// * `max_sends` is the MN-8 knob: `0` = unlimited; otherwise it counts **qualifying** sends
/// (applied after `delivery_decision`), worst-first.
/// * A deliverable group `muted(key, decision, last_notified_level)` holds goes to `muted`, first.
#[allow(clippy::too_many_arguments)]
pub fn plan_dispatch(
classification: &GroupClassification,
states: &HashMap<String, AlertState>,
@@ -142,6 +145,7 @@ pub fn plan_dispatch(
tc: &TriggerCondition,
now: i64,
max_sends: usize,
suppress_repeats: bool,
muted: impl Fn(&str, &DeliveryDecision, Option<AlertLevel>) -> bool,
) -> DispatchPlan {
let mut items = Vec::new();
@@ -150,6 +154,7 @@ pub fn plan_dispatch(
let mut dropped_by_knob = Vec::new();
let mut inconsistent = Vec::new();
let mut muted_keys = Vec::new();
let mut repeats = 0usize;
// A key appearing twice in one classification is an invariant violation
// (validation rejects the known source, MN-12). Counting first means both
@@ -218,6 +223,17 @@ pub fn plan_dispatch(
muted_keys.push(key.clone());
continue;
}
if suppress_repeats
&& repeats_last_notification(
level,
state.episode_id.is_some(),
state.last_notified_level,
)
{
suppressed += 1;
repeats += 1;
continue;
}
// The knob counts QUALIFYING sends, so a silenced group never
// spends budget a deliverable one could have used.
if max_sends > 0 && items.len() >= max_sends {
@@ -237,7 +253,8 @@ pub fn plan_dispatch(
DeliveryDecision::SuppressedBySilence
| DeliveryDecision::SuppressedByWarningPolicy
| DeliveryDecision::SuppressedByDowntime(_)
| DeliveryDecision::SuppressedByAck(_) => suppressed += 1,
| DeliveryDecision::SuppressedByAck(_)
| DeliveryDecision::SuppressedAsRepeat => suppressed += 1,
DeliveryDecision::SuppressedByPending => pending += 1,
DeliveryDecision::NotFiring => {}
}
@@ -250,9 +267,19 @@ pub fn plan_dispatch(
dropped_by_knob,
inconsistent,
muted: muted_keys,
repeats,
}
}
/// The open recovery episode was already notified at `level`, so this firing tells nothing new.
pub fn repeats_last_notification(
level: AlertLevel,
episode_open: bool,
last_notified: Option<AlertLevel>,
) -> bool {
episode_open && last_notified == Some(level)
}
/// The write that records one group's **successful** delivery (MN-6).
///
/// Returns the full replacement row, or `None` when the carried **level-episode
@@ -744,6 +771,7 @@ mod tests {
&tc(None),
1_000 * SEC,
0,
false,
|_, _, _| false,
);
@@ -769,9 +797,16 @@ mod tests {
let c = classified(&[("crit", 150.0), ("ok1", 1.0), ("ok2", 2.0)], 500);
let states = states_for(&c, now, None, None);
let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| {
false
});
let plan = plan_dispatch(
&c,
&states,
&rows_for(&c),
&tc(None),
now,
0,
false,
|_, _, _| false,
);
assert_eq!(plan.items.len(), 1);
assert_eq!(
@@ -791,9 +826,16 @@ mod tests {
let c = classified(&[("a", 150.0)], 500);
let states = states_for(&c, now, None, None);
let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| {
false
});
let plan = plan_dispatch(
&c,
&states,
&rows_for(&c),
&tc(None),
now,
0,
false,
|_, _, _| false,
);
assert!(
plan.items[0].escalation,
@@ -816,6 +858,7 @@ mod tests {
&tc(None),
1_000 * SEC,
0,
false,
|_, _, _| false,
);
let b = plan
@@ -841,6 +884,7 @@ mod tests {
&tc(None),
1_000 * SEC,
0,
false,
|_, _, _| false,
);
let keys: Vec<_> = plan.items.iter().map(|i| i.group_key.clone()).collect();
@@ -863,9 +907,16 @@ mod tests {
let c = classified(&[("a", 150.0)], 500);
let states = states_for(&c, now, Some(AlertLevel::Critical), Some(now + 10 * MIN));
let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| {
false
});
let plan = plan_dispatch(
&c,
&states,
&rows_for(&c),
&tc(None),
now,
0,
false,
|_, _, _| false,
);
assert!(plan.items.is_empty());
assert_eq!(plan.suppressed, 1);
@@ -879,9 +930,16 @@ mod tests {
let c = classified(&[("a", 150.0)], 500);
let states = states_for(&c, now, Some(AlertLevel::Critical), Some(now - 1));
let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| {
false
});
let plan = plan_dispatch(
&c,
&states,
&rows_for(&c),
&tc(None),
now,
0,
false,
|_, _, _| false,
);
assert_eq!(plan.items.len(), 1);
assert!(
@@ -898,9 +956,16 @@ mod tests {
let c = classified(&[("a", 150.0)], 500);
let states = states_for(&c, now, Some(AlertLevel::Warning), Some(now + 10 * MIN));
let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| {
false
});
let plan = plan_dispatch(
&c,
&states,
&rows_for(&c),
&tc(None),
now,
0,
false,
|_, _, _| false,
);
assert_eq!(plan.items.len(), 1);
assert!(plan.items[0].escalation);
@@ -933,9 +998,16 @@ mod tests {
assert!(row.is_firing(), "NotifyFailed is still a firing state");
let states = HashMap::from([(key.clone(), row)]);
let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| {
false
});
let plan = plan_dispatch(
&c,
&states,
&rows_for(&c),
&tc(None),
now,
0,
false,
|_, _, _| false,
);
assert_eq!(
plan.items.len(),
@@ -958,6 +1030,7 @@ mod tests {
&tc(Some(false)),
1_000 * SEC,
0,
false,
|_, _, _| false,
);
@@ -999,7 +1072,9 @@ mod tests {
]),
);
let plan = plan_dispatch(&c, &states, &rows, &tc(None), now, 0, |_, _, _| false);
let plan = plan_dispatch(&c, &states, &rows, &tc(None), now, 0, false, |_, _, _| {
false
});
assert_eq!(plan.items.len(), 2, "only retained groups dispatch");
assert!(
@@ -1030,9 +1105,16 @@ mod tests {
state(ROLLUP_GROUP_KEY, AlertLevel::Critical, now, None, None),
);
let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| {
false
});
let plan = plan_dispatch(
&c,
&states,
&rows_for(&c),
&tc(None),
now,
0,
false,
|_, _, _| false,
);
assert_eq!(plan.items.len(), 1, "one group fired, so one page");
assert_eq!(plan.items[0].group_key, key_of("a"));
@@ -1060,9 +1142,16 @@ mod tests {
state(&gone, AlertLevel::Critical, now - 5 * MIN, None, None),
);
let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| {
false
});
let plan = plan_dispatch(
&c,
&states,
&rows_for(&c),
&tc(None),
now,
0,
false,
|_, _, _| false,
);
assert!(!plan.items.iter().any(|i| i.group_key == gone));
}
@@ -1084,6 +1173,7 @@ mod tests {
&tc(None),
now,
0,
false,
|_, _, _| false,
);
@@ -1111,6 +1201,7 @@ mod tests {
&tc(None),
now,
0,
false,
|_, _, _| false,
);
@@ -1139,9 +1230,16 @@ mod tests {
assert_eq!(c.groups.len(), 3, "fixture: `a` is observed twice");
let states = states_for(&c, now, None, None);
let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| {
false
});
let plan = plan_dispatch(
&c,
&states,
&rows_for(&c),
&tc(None),
now,
0,
false,
|_, _, _| false,
);
assert!(
!plan.items.iter().any(|i| i.group_key == key_of("a")),
@@ -1170,9 +1268,16 @@ mod tests {
assert_eq!(c.groups.len(), 3, "fixture: `a` observed at two levels");
let states = states_for(&c, now, None, None);
let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| {
false
});
let plan = plan_dispatch(
&c,
&states,
&rows_for(&c),
&tc(None),
now,
0,
false,
|_, _, _| false,
);
for item in &plan.items {
assert_ne!(
@@ -1202,9 +1307,16 @@ mod tests {
);
let states = states_for(&c, now, None, None);
let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 2, |_, _, _| {
false
});
let plan = plan_dispatch(
&c,
&states,
&rows_for(&c),
&tc(None),
now,
2,
false,
|_, _, _| false,
);
assert_eq!(plan.items.len(), 2, "`b` and `c` both page");
assert!(plan.dropped_by_knob.is_empty());
@@ -1219,7 +1331,9 @@ mod tests {
let states = states_for(&c, now, None, None);
let rows = rows_for(&c);
let plan = plan_dispatch(&c, &states, &rows, &tc(None), now, 0, |_, _, _| false);
let plan = plan_dispatch(&c, &states, &rows, &tc(None), now, 0, false, |_, _, _| {
false
});
assert_eq!(
&plan.items[0].row,
@@ -1243,9 +1357,16 @@ mod tests {
row.level_at = Some(now); // freshness advanced by a later same-level run
let states = HashMap::from([(key.clone(), row)]);
let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |_, _, _| {
false
});
let plan = plan_dispatch(
&c,
&states,
&rows_for(&c),
&tc(None),
now,
0,
false,
|_, _, _| false,
);
assert_eq!(
plan.items[0].episode,
@@ -1270,6 +1391,7 @@ mod tests {
&tc(None),
1_000 * SEC,
0,
false,
|_, _, _| false,
);
assert_eq!(plan.items.len(), 3);
@@ -1290,6 +1412,7 @@ mod tests {
&tc(None),
1_000 * SEC,
2,
false,
|_, _, _| false,
);
@@ -1324,9 +1447,16 @@ mod tests {
let c = build();
let states = states_for(&c, now, None, None);
let first = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 2, |_, _, _| {
false
});
let first = plan_dispatch(
&c,
&states,
&rows_for(&c),
&tc(None),
now,
2,
false,
|_, _, _| false,
);
let second = plan_dispatch(
&build(),
&states,
@@ -1334,6 +1464,7 @@ mod tests {
&tc(None),
now,
2,
false,
|_, _, _| false,
);
@@ -1367,9 +1498,16 @@ mod tests {
),
]);
let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 1, |_, _, _| {
false
});
let plan = plan_dispatch(
&c,
&states,
&rows_for(&c),
&tc(None),
now,
1,
false,
|_, _, _| false,
);
assert_eq!(plan.items.len(), 1);
assert_eq!(plan.items[0].group_key, key_b);
@@ -1387,9 +1525,16 @@ mod tests {
let (key_a, key_b) = (key_of("a"), key_of("b"));
let states = states_for(&c, now, None, None);
for _ in 0..2 {
let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 1, |k, _, _| {
k == key_a
});
let plan = plan_dispatch(
&c,
&states,
&rows_for(&c),
&tc(None),
now,
1,
false,
|k, _, _| k == key_a,
);
assert_eq!(plan.items.len(), 1);
assert_eq!(plan.items[0].group_key, key_b);
assert_eq!(plan.suppressed, 1);
@@ -1408,7 +1553,7 @@ mod tests {
let acked = |k: &str, d: &DeliveryDecision, last: Option<AlertLevel>| {
(k == key_a || k == key_b) && d.ack_withholds(last)
};
let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, acked);
let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, false, acked);
assert_eq!(
plan.muted,
vec![key_a],
@@ -1428,9 +1573,16 @@ mod tests {
let c = classified(&[("a", 150.0), ("b", 75.0)], 500);
let (key_a, key_b) = (key_of("a"), key_of("b"));
let states = states_for(&c, now, None, None);
let plan = plan_dispatch(&c, &states, &rows_for(&c), &tc(None), now, 0, |k, _, _| {
k == key_a
});
let plan = plan_dispatch(
&c,
&states,
&rows_for(&c),
&tc(None),
now,
0,
false,
|k, _, _| k == key_a,
);
assert_eq!(plan.items.len(), 1);
assert_eq!(plan.items[0].group_key, key_b);
assert_eq!(plan.muted, vec![key_a]);
@@ -2185,4 +2337,109 @@ mod tests {
assert_eq!(back, episode);
assert_eq!(back.notified_at_enqueue, (None, None));
}
// ═════════════════════════════════════════════════════════════════════
// suppress_repeats — notify once per state change
// ═════════════════════════════════════════════════════════════════════
#[test]
fn only_the_notified_level_inside_an_open_episode_is_a_repeat() {
use AlertLevel::{Critical, Warning};
// Still firing at the delivered level, or back at it inside a keep_firing_for hold.
assert!(repeats_last_notification(Critical, true, Some(Critical)));
// Escalated or de-escalated since the last delivery.
assert!(!repeats_last_notification(Critical, true, Some(Warning)));
assert!(!repeats_last_notification(Warning, true, Some(Critical)));
// The episode closed on a recovery: a new incident.
assert!(!repeats_last_notification(Critical, false, Some(Critical)));
// Nothing was ever delivered.
assert!(!repeats_last_notification(Critical, false, None));
}
#[test]
fn a_repeat_is_withheld_like_a_downtime_or_an_ack() {
assert!(DeliveryDecision::SuppressedAsRepeat.withheld());
assert!(DeliveryDecision::SuppressedByAck("u".into()).withheld());
assert!(!DeliveryDecision::SuppressedBySilence.withheld());
assert!(!DeliveryDecision::Deliver.withheld());
}
/// Every group of `c` delivered at Critical, inside an open recovery episode.
fn notified_states(c: &GroupClassification) -> HashMap<String, AlertState> {
let mut states = states_for(c, MIN, Some(AlertLevel::Critical), None);
for row in states.values_mut() {
row.episode_id = Some("ep-1".to_string());
}
states
}
#[test]
fn plan_dispatch_withholds_a_repeat_only_when_the_switch_is_on() {
let c = classified(&[("a", 150.0)], 500);
let states = notified_states(&c);
let plan = |on| {
plan_dispatch(
&c,
&states,
&rows_for(&c),
&tc(None),
10 * MIN,
0,
on,
|_, _, _| false,
)
};
assert_eq!(
plan(false).items.len(),
1,
"silence 0 and the switch off page again"
);
let on = plan(true);
assert!(on.items.is_empty());
assert_eq!((on.suppressed, on.repeats), (1, 1));
}
#[test]
fn a_repeat_does_not_spend_the_send_budget() {
// `a` repeats; `b` fires for the first time and must get the single slot.
let c = classified(&[("a", 150.0), ("b", 150.0)], 500);
let mut states = states_for(&c, MIN, None, None);
let a = states.get_mut(&key_of("a")).unwrap();
a.last_notified_level = Some(AlertLevel::Critical);
a.episode_id = Some("ep-1".to_string());
let plan = plan_dispatch(
&c,
&states,
&rows_for(&c),
&tc(None),
10 * MIN,
1,
true,
|_, _, _| false,
);
assert_eq!(plan.items.len(), 1);
assert_eq!(plan.items[0].group_key, key_of("b"));
assert!(plan.dropped_by_knob.is_empty());
assert_eq!(plan.repeats, 1);
}
#[test]
fn a_muted_group_is_reported_muted_not_repeated() {
let c = classified(&[("a", 150.0)], 500);
let states = notified_states(&c);
let plan = plan_dispatch(
&c,
&states,
&rows_for(&c),
&tc(None),
10 * MIN,
0,
true,
|_, _, _| true,
);
assert_eq!(plan.muted, vec![key_of("a")]);
assert_eq!(plan.repeats, 0);
}
}
src/config/src/meta/alerts/level.rs
+10
View File
@@ -366,6 +366,8 @@ pub enum DeliveryDecision {
SuppressedByDowntime(String),
/// The current firing is acknowledged by this user; repeats are withheld.
SuppressedByAck(String),
/// `suppress_repeats` is on and nothing changed since the last delivered notification.
SuppressedAsRepeat,
}
impl DeliveryDecision {
@@ -385,6 +387,14 @@ impl DeliveryDecision {
}
}
/// Withheld by a downtime, an ack or as a repeat, rather than by the level decision.
pub fn withheld(&self) -> bool {
matches!(
self,
Self::SuppressedByDowntime(_) | Self::SuppressedByAck(_) | Self::SuppressedAsRepeat
)
}
pub fn acked_by(&self) -> Option<&str> {
match self {
Self::SuppressedByAck(by) => Some(by),
src/config/src/meta/dashboards/v8/mod.rs
+62 -1
View File
@@ -480,6 +480,10 @@ pub struct FilterCondition {
pub value: Option<String>,
pub logical_operator: String,
pub filter_type: String,
/// Set when `value` is an explicit `''` from SQL, so the builder keeps `x <> ''`
/// instead of treating the empty value as "not entered yet".
#[serde(default, skip_serializing_if = "Option::is_none")]
pub empty_literal: Option<bool>,
}
#[derive(Debug, Clone, PartialEq, Hash, Serialize, Deserialize, ToSchema)]
@@ -810,6 +814,22 @@ pub struct PanelConfig {
#[serde(skip_serializing_if = "Option::is_none")]
flame_graph_value_field: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
boxplot_whisker: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
logs_message_field: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
logs_level_field: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
logs_time_field: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
logs_wrap: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
logs_prettify_json: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
logs_show_time: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
logs_order: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
hostmap_entity_label: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
hostmap_group_by: Option<Vec<String>>,
@@ -1556,6 +1576,39 @@ mod tests {
assert_eq!(again, d);
}
#[test]
fn test_filter_condition_empty_literal_roundtrip() {
let kept = serde_json::json!({
"type": "condition",
"values": [],
"column": { "field": "service_name", "streamAlias": null },
"operator": "<>",
"value": "",
"logicalOperator": "AND",
"filterType": "condition",
"emptyLiteral": true
});
let filter: PanelFilter = serde_json::from_value(kept).unwrap();
match &filter {
PanelFilter::Condition(c) => assert_eq!(c.empty_literal, Some(true)),
PanelFilter::Group(_) => panic!("expected a condition"),
}
assert_eq!(serde_json::to_value(&filter).unwrap()["emptyLiteral"], true);
let plain = serde_json::json!({
"type": "condition",
"values": [],
"column": { "field": "service_name", "streamAlias": null },
"operator": "=",
"value": "",
"logicalOperator": "AND",
"filterType": "condition"
});
let filter: PanelFilter = serde_json::from_value(plain).unwrap();
let back = serde_json::to_value(&filter).unwrap();
assert!(!back.as_object().unwrap().contains_key("emptyLiteral"));
}
#[test]
fn test_annotation_sources_change_the_hash() {
let plain: super::super::Dashboard = make_dashboard().into();
@@ -3022,7 +3075,15 @@ mod tests {
});
let rest4 = serde_json::json!({
"flame_graph_stack_field": "stack",
"flame_graph_value_field": "samples"
"flame_graph_value_field": "samples",
"boxplot_whisker": "iqr",
"logs_message_field": "message",
"logs_level_field": "level",
"logs_time_field": "_timestamp",
"logs_wrap": true,
"logs_prettify_json": false,
"logs_show_time": true,
"logs_order": "oldest"
});
for part in [rest, rest2, rest3, rest4] {
fields
src/config/src/meta/fleet.rs
+1 -1
View File
@@ -69,7 +69,7 @@ pub struct Collector {
pub version: Option<String>,
pub outdated: bool,
pub status: CollectorStatus,
/// The connection's peer address, honouring `X-Forwarded-For`.
/// The connection's client address, as `ZO_HTTP_REAL_IP_SOURCE` resolves it.
pub last_connected_from: Option<String>,
pub fleet: CollectorFleet,
pub token_id: Option<String>,
src/config/src/meta/public_dashboards.rs
+33 -1
View File
@@ -413,9 +413,12 @@ pub struct KeyedRange {
pub refresh_secs: i64,
}
/// A dashboard variable as the public viewer sees it: display label and frozen value only.
/// A dashboard variable as the public viewer sees it: name, display label and frozen value only.
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
pub struct PublicVariable {
/// The name panel titles reference as `$name`; empty in a result stored before it was added.
#[serde(default)]
pub name: String,
pub label: String,
pub value: serde_json::Value,
/// Set for a tab-scoped variable: the tab this value belongs to.
@@ -472,6 +475,7 @@ impl PublicVariable {
.into_iter()
.map(|(key, tab_id, panel_id)| {
let entry = Self {
name: v.name.clone(),
label: label.clone(),
value: values.get(&key).cloned().unwrap_or(serde_json::Value::Null),
tab_id,
@@ -750,6 +754,34 @@ mod tests {
assert_eq!(cfg.refresh_secs, 300);
}
#[test]
fn public_variables_carry_the_name_panel_titles_reference() {
let dash = Dashboard {
v8: Some(
serde_json::from_value(serde_json::json!({
"variables": { "list": [
{ "type": "custom", "name": "level", "label": "Level" },
{ "type": "custom", "name": "svc", "scope": "tabs", "tabs": ["t1"] }
] }
}))
.unwrap(),
),
..Default::default()
};
let values = BTreeMap::from([("level".to_string(), serde_json::json!("error"))]);
let list = PublicVariable::list(&dash, &values);
let names: Vec<(&str, &str)> = list
.iter()
.map(|v| (v.name.as_str(), v.label.as_str()))
.collect();
assert_eq!(names, [("level", "Level"), ("svc", "svc")]);
assert_eq!(serde_json::to_value(&list[0]).unwrap()["name"], "level");
let stored: PublicVariable =
serde_json::from_value(serde_json::json!({ "label": "Level", "value": "error" }))
.unwrap();
assert_eq!(stored.name, "");
}
#[test]
fn identity_names_the_org() {
assert_eq!(
src/config/src/meta/self_reporting/mod.rs
+4
View File
@@ -405,6 +405,7 @@ mod tests {
ack_source: None,
taken_over_from: None,
delivery_attempted: None,
repeat_suppressed: None,
trigger_source: None,
triggered_by: None,
recipient_count: None,
@@ -965,6 +966,7 @@ mod tests {
ack_source: None,
taken_over_from: None,
delivery_attempted: None,
repeat_suppressed: None,
trigger_source: None,
triggered_by: None,
recipient_count: None,
@@ -1057,6 +1059,7 @@ mod tests {
ack_source: None,
taken_over_from: None,
delivery_attempted: None,
repeat_suppressed: None,
trigger_source: None,
triggered_by: None,
recipient_count: None,
@@ -1133,6 +1136,7 @@ mod tests {
ack_source: None,
taken_over_from: None,
delivery_attempted: None,
repeat_suppressed: None,
trigger_source: None,
triggered_by: None,
recipient_count: None,
src/config/src/meta/self_reporting/usage.rs
+6
View File
@@ -354,6 +354,9 @@ pub struct TriggerData {
/// Whether a firing tried a destination, workflow or incident notification; None on old rows.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub delivery_attempted: Option<bool>,
/// Set when `suppress_repeats` withheld a notification that repeated the last delivered state.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub repeat_suppressed: Option<bool>,
/// "scheduled" or "manual"; None on older rows, which read as scheduled.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub trigger_source: Option<String>,
@@ -413,6 +416,7 @@ impl Default for TriggerData {
ack_source: None,
taken_over_from: None,
delivery_attempted: None,
repeat_suppressed: None,
trigger_source: None,
triggered_by: None,
recipient_count: None,
@@ -473,6 +477,7 @@ impl TriggerData {
ack_source: Some(AckSource::Api),
taken_over_from: Some(String::new()),
delivery_attempted: Some(true),
repeat_suppressed: None,
trigger_source: Some(String::new()),
triggered_by: Some(String::new()),
recipient_count: Some(0),
@@ -1915,6 +1920,7 @@ mod tests {
ack_source: None,
taken_over_from: None,
delivery_attempted: None,
repeat_suppressed: None,
trigger_source: None,
triggered_by: None,
recipient_count: None,
src/config/src/meta/status_pages.rs
+159 -5
View File
@@ -46,6 +46,10 @@ pub const CHECK_STATUS_UNKNOWN: i32 = 0;
pub const CHECK_STATUS_UP: i32 = 1;
pub const CHECK_STATUS_WARNING: i32 = 2;
pub const CHECK_STATUS_DOWN: i32 = 3;
/// The probe could not produce a verdict (dispatch dead-letter, expired lease,
/// infra fault). Says nothing about the target: it never opens an incident and
/// never counts toward recovery. Written by the dispatcher and the reaper.
pub const CHECK_STATUS_ERROR: i32 = 4;
const MICROS_PER_MIN: i64 = 60 * 1_000_000;
const MICROS_PER_DAY: i64 = 86_400 * 1_000_000;
@@ -120,6 +124,10 @@ pub struct EngineCheck {
pub first_fail_at: Option<i64>,
/// When the streak started failing, for time-based confirmation.
pub failing_since: Option<i64>,
/// True only when this tick sees a run result it has not seen before.
/// Recovery advances on distinct results, never on rebuild ticks, so the
/// same UP observed over ten ticks counts once.
pub fresh_result: bool,
}
/// An auto-incident notice currently open (or just resolved, for the merge
@@ -535,12 +543,33 @@ pub fn incident_actions(
snoozed: &HashSet<String>,
th: &Thresholds,
now: i64,
) -> Vec<EngineAction> {
incident_actions_with(
checks,
open_by_check,
recently_resolved,
snoozed,
|_| *th,
now,
)
}
/// [`incident_actions`] with per-check thresholds (each check is judged by the
/// confirmation settings of the page(s) it is mapped to).
pub fn incident_actions_with(
checks: &[EngineCheck],
open_by_check: &HashMap<String, OpenAutoIncident>,
recently_resolved: &HashMap<String, (String, i64)>,
snoozed: &HashSet<String>,
th_for: impl Fn(&str) -> Thresholds,
now: i64,
) -> Vec<EngineAction> {
let mut actions = Vec::new();
for check in checks {
let th = th_for(&check.id);
match open_by_check.get(&check.id) {
Some(open) => handle_open(check, open, th, now, &mut actions),
None => handle_closed(check, recently_resolved, snoozed, th, now, &mut actions),
Some(open) => handle_open(check, open, &th, now, &mut actions),
None => handle_closed(check, recently_resolved, snoozed, &th, now, &mut actions),
}
}
actions
@@ -568,7 +597,16 @@ fn handle_open(
now: i64,
actions: &mut Vec<EngineAction>,
) {
if check.last_check_status == CHECK_STATUS_UP {
// UP and WARNING both mean the target answered: they advance recovery, but
// only on a result this loop has not counted yet.
let healthy = matches!(
check.last_check_status,
CHECK_STATUS_UP | CHECK_STATUS_WARNING
);
if healthy {
if !check.fresh_result {
return;
}
let streak = open.recovery_streak + 1;
if streak >= th.confirm_recovery {
actions.push(EngineAction::Resolve {
@@ -581,8 +619,14 @@ fn handle_open(
streak,
});
}
} else if check.last_check_status == CHECK_STATUS_DOWN && open.recovery_streak > 0 {
// Failed again mid-recovery: the streak restarts from zero.
} else if matches!(
check.last_check_status,
CHECK_STATUS_DOWN | CHECK_STATUS_ERROR
) && open.recovery_streak > 0
{
// Failed again mid-recovery: the streak restarts from zero. ERROR does
// the same: an unknown verdict cannot prove recovery, so UP, ERROR, UP
// is not two consecutive good results. The incident stays open.
actions.push(EngineAction::RecoveryProgress {
notice_id: open.notice_id.clone(),
streak: 0,
@@ -761,9 +805,119 @@ mod tests {
consecutive_failures: streak,
first_fail_at: None,
failing_since: None,
fresh_result: true,
}
}
fn open_inc(streak: i32) -> HashMap<String, OpenAutoIncident> {
[(
"c1".to_string(),
OpenAutoIncident {
notice_id: "n1".into(),
check_id: "c1".into(),
recovery_streak: streak,
},
)]
.into()
}
/// Replays results against an open incident, applying RecoveryProgress
/// back into the streak; returns whether it resolved.
fn replay(statuses: &[(i32, bool)], th: Thresholds) -> bool {
let mut streak = 0;
for &(status, fresh) in statuses {
let mut c = check("c1", status, 0);
c.fresh_result = fresh;
let acts = incident_actions(
&[c],
&open_inc(streak),
&HashMap::new(),
&HashSet::new(),
&th,
0,
);
for a in acts {
match a {
EngineAction::Resolve { .. } => return true,
EngineAction::RecoveryProgress { streak: s, .. } => streak = s,
_ => panic!("unexpected {a:?}"),
}
}
}
false
}
fn th(failures: i32, recovery: i32) -> Thresholds {
Thresholds {
confirm_failures: failures,
confirm_recovery: recovery,
confirm_after_micros: None,
}
}
#[test]
fn up_error_up_does_not_resolve_with_recovery_two() {
let s = CHECK_STATUS_UP;
let e = CHECK_STATUS_ERROR;
assert!(!replay(&[(s, true), (e, true), (s, true)], th(3, 2)));
assert!(replay(
&[(s, true), (e, true), (s, true), (s, true)],
th(3, 2)
));
}
#[test]
fn error_never_opens_an_incident() {
let c = check("c1", CHECK_STATUS_ERROR, 50);
let acts = incident_actions(
&[c],
&HashMap::new(),
&HashMap::new(),
&HashSet::new(),
&th(1, 1),
0,
);
assert!(acts.is_empty());
}
#[test]
fn error_keeps_an_open_incident_open() {
let e = CHECK_STATUS_ERROR;
assert!(!replay(&[(e, true); 10], th(3, 1)));
}
#[test]
fn warning_advances_recovery() {
let w = CHECK_STATUS_WARNING;
assert!(replay(&[(w, true), (w, true)], th(3, 2)));
}
#[test]
fn the_same_result_over_many_ticks_counts_once() {
let mut ticks = vec![(CHECK_STATUS_UP, true)];
ticks.extend(std::iter::repeat_n((CHECK_STATUS_UP, false), 20));
assert!(!replay(&ticks, th(3, 2)));
ticks.push((CHECK_STATUS_UP, true));
assert!(replay(&ticks, th(3, 2)));
}
#[test]
fn seven_failures_four_recoveries() {
let t = th(7, 4);
let none = HashMap::new();
let quiet = HashSet::new();
let six = [check("c1", CHECK_STATUS_DOWN, 6)];
assert!(incident_actions(&six, &none, &HashMap::new(), &quiet, &t, 0).is_empty());
let seven = [check("c1", CHECK_STATUS_DOWN, 7)];
assert_eq!(
incident_actions(&seven, &none, &HashMap::new(), &quiet, &t, 0).len(),
1
);
let up = CHECK_STATUS_UP;
assert!(!replay(&[(up, true); 3], t));
assert!(replay(&[(up, true); 4], t));
}
#[test]
fn union_counts_overlaps_once() {
// Two checks confirm the same outage: 10:00-10:30 and 10:10-10:40.
src/config/src/meta/user.rs
+12
View File
@@ -153,7 +153,10 @@ impl FromStr for UserRole {
"user" | "allowed_user" => Ok(UserRole::User),
"service_account" => Ok(UserRole::ServiceAccount),
"sre_agent" => Ok(UserRole::SreAgent),
#[cfg(not(feature = "enterprise"))]
_ => Ok(UserRole::Admin),
#[cfg(feature = "enterprise")]
_ => Ok(UserRole::User),
}
}
}
@@ -413,9 +416,18 @@ mod tests {
#[test]
fn test_invalid_string_defaults_to_admin() {
#[cfg(feature = "enterprise")]
assert_eq!(UserRole::from_str("invalid").unwrap(), UserRole::User);
#[cfg(not(feature = "enterprise"))]
assert_eq!(UserRole::from_str("invalid").unwrap(), UserRole::Admin);
#[cfg(feature = "enterprise")]
assert_eq!(UserRole::from_str("").unwrap(), UserRole::User);
#[cfg(not(feature = "enterprise"))]
assert_eq!(UserRole::from_str("").unwrap(), UserRole::Admin);
#[cfg(not(feature = "enterprise"))]
assert_eq!(UserRole::from_str("Admin").unwrap(), UserRole::Admin); // case sensitive
#[cfg(feature = "enterprise")]
assert_eq!(UserRole::from_str("Admin").unwrap(), UserRole::User); // case sensitive
}
}
src/config/src/utils/str.rs
+8
View File
@@ -34,11 +34,19 @@ pub static EMAIL_REGEX: LazyLock<Regex> = LazyLock::new(|| {
.unwrap()
});
pub static DOMAIN_REGEX: LazyLock<Regex> =
LazyLock::new(|| Regex::new(r"^([a-zA-Z0-9]+([\-\.][a-zA-Z0-9]+)*\.[a-zA-Z]{2,63})$").unwrap());
/// Returns true if `email` is syntactically valid per [`EMAIL_REGEX`].
pub fn is_valid_email(email: &str) -> bool {
EMAIL_REGEX.is_match(email)
}
/// Returns true if `domain` is syntactically valid per [`DOMAIN_REGEX`].
pub fn is_valid_domain(domain: &str) -> bool {
DOMAIN_REGEX.is_match(domain)
}
/// Characters an OpenFGA object id cannot represent. Detection
/// ([`is_ofga_unsupported`]) and sanitization ([`into_ofga_supported_format`])
/// share this pattern so they always agree on what is unsafe.
src/core/Cargo.toml
+2 -1
View File
@@ -48,9 +48,10 @@ vectorscan = [
]
# Lets dependants' tests install a fake stream access checker; never enabled outside dev-dependencies.
test-utils = []
cpu-profiling = ["dep:pprof"]
profiling = [
"cpu-profiling",
"dep:jemalloc_pprof",
"dep:pprof",
"dep:tikv-jemalloc-ctl",
]
src/core/src/alerts/alert.rs
+33
View File
@@ -327,6 +327,8 @@ pub enum AlertError {
CronHasNoFutureOccurrence { cron: String },
#[error("Realtime alerts cannot notify on recovery or keep firing")]
RecoveryOnRealtimeAlert,
#[error("Realtime alerts cannot suppress repeat notifications")]
RepeatSuppressionOnRealtimeAlert,
#[error("recovery destinations were set without notify_on_recovery, so nothing would use them")]
RecoveryDestinationsWithoutRecovery,
#[error(
@@ -895,6 +897,10 @@ async fn prepare_alert(
if alert.is_real_time && (alert.notify_on_recovery || alert.keep_firing_for != 0) {
return Err(AlertError::RecoveryOnRealtimeAlert);
}
// A realtime alert keeps no state row, so it has no last delivered state to compare against.
if alert.is_real_time && alert.suppress_repeats {
return Err(AlertError::RepeatSuppressionOnRealtimeAlert);
}
// Multi-level thresholds (alerts_2.md Feature 1). Rejected at write time so
// an unreachable warning level can never reach the evaluator.
@@ -8853,6 +8859,33 @@ mod tests {
}
}
#[tokio::test]
async fn prepare_alert_refuses_repeat_suppression_on_a_realtime_alert() {
use infra::schema::{STREAM_SCHEMAS_LATEST, SchemaCache};
let org_id = "test_org_alert_save_repeat_realtime";
let key = format!("{org_id}/{}/app", StreamType::Logs);
STREAM_SCHEMAS_LATEST.write().await.insert(
key.clone(),
SchemaCache::new(arrow_schema::Schema::new(vec![arrow_schema::Field::new(
"id",
DataType::Utf8,
true,
)])),
);
let mut alert = Alert::default();
alert.trigger_condition.frequency = 60;
alert.is_real_time = true;
alert.suppress_repeats = true;
let res = prepare_alert(org_id, "app", "realtime", &mut alert, true, false).await;
STREAM_SCHEMAS_LATEST.write().await.remove(&key);
assert!(
matches!(res, Err(AlertError::RepeatSuppressionOnRealtimeAlert)),
"{res:?}"
);
}
#[tokio::test]
async fn prepare_alert_rejects_a_cron_with_no_future_occurrence() {
let err = prepare_schedule(|a| {
src/core/src/alerts/prom_rule.rs
+1
View File
@@ -446,6 +446,7 @@ mod tests {
&TriggerCondition::default(),
600,
0,
false,
|_, _, _| false,
);
assert!(dispatch.inconsistent.is_empty());
src/core/src/alerts/scheduler/handlers.rs
+130 -13
View File
@@ -515,6 +515,8 @@ struct GroupDispatchOutcome {
downtime_id: Option<String>,
/// Who acked the first group an ack kept silent.
acked_by: Option<String>,
/// Some group was withheld because it repeated its last delivery.
repeated: bool,
}
impl GroupDispatchOutcome {
@@ -629,6 +631,7 @@ async fn dispatch_per_group(
state_failed: true,
downtime_id: None,
acked_by: None,
repeated: false,
});
};
@@ -647,6 +650,7 @@ async fn dispatch_per_group(
state_failed: true,
downtime_id: None,
acked_by: None,
repeated: false,
});
}
};
@@ -684,6 +688,7 @@ async fn dispatch_per_group(
&alert.trigger_condition,
now_micros(),
cfg.limit.alert_max_group_notifications_per_eval,
alert.suppresses_repeats(),
|key, decision, last_notified| {
muted_groups.contains_key(key)
|| (acked_groups.contains_key(key) && decision.ack_withholds(last_notified))
@@ -749,7 +754,7 @@ async fn dispatch_per_group(
continue;
}
// Chosen before the send: a resolve only closes a ticket opened under the key it quotes.
let episode_key = alert.notify_on_recovery.then(|| {
let episode_key = (alert.notify_on_recovery || alert.suppresses_repeats()).then(|| {
states
.get(&item.group_key)
.and_then(|state| state.episode_id.clone())
@@ -771,7 +776,7 @@ async fn dispatch_per_group(
// last so a label value containing `{...}` cannot expand.
Some(&item.labels),
&[],
episode_key.clone(),
episode_key.clone().filter(|_| alert.notify_on_recovery),
)
.await;
@@ -835,9 +840,11 @@ async fn dispatch_per_group(
log::info!(
"[SCHEDULER trace_id {trace_id}] alert {alert_id}: per-group dispatch delivered={delivered} \
pending={} failed={failed} suppressed={} downtime={downtime_suppressed} candidates={}",
pending={} failed={failed} suppressed={} repeats={} downtime={downtime_suppressed} \
candidates={}",
plan.pending,
plan.suppressed,
plan.repeats,
plan.items.len()
);
Some(GroupDispatchOutcome {
@@ -849,6 +856,7 @@ async fn dispatch_per_group(
state_failed: false,
downtime_id,
acked_by,
repeated: plan.repeats > 0,
})
}
@@ -3053,6 +3061,45 @@ pub(crate) async fn correlate_muted_incident(
}
}
/// A repeat still correlates, or a continuous firing lets its incident go stale and auto-resolve.
#[cfg(feature = "enterprise")]
async fn correlate_repeated_incident(
trace_id: &str,
alert: &config::meta::alerts::alert::Alert,
rows: &[config::utils::json::Map<String, config::utils::json::Value>],
triggered_at: i64,
eval_level: Option<config::meta::alerts::level::AlertLevel>,
actual_value: Option<f64>,
) {
let Some(first_row) = rows.first() else {
return;
};
let incidents_enabled = o2_enterprise::enterprise::common::config::get_config()
.incidents
.enabled;
if !correlates_repeated_firing(alert.creates_incident, incidents_enabled) {
return;
}
// Correlation itself stays silent on a repeat and only notifies a new or escalated incident.
if let Err(e) = crate::alerts::incidents::correlate_alert_to_incident(
alert,
first_row,
rows,
triggered_at,
eval_level,
actual_value,
None,
)
.await
{
log::error!(
"[SCHEDULER trace_id {trace_id}] repeat incident correlation failed for {}/{}: {e}",
alert.org_id,
alert.name
);
}
}
/// Whether a matched single-series run is still inside its pending period.
fn holds_pending(
alert: &config::meta::alerts::alert::Alert,
@@ -3134,6 +3181,12 @@ fn correlates_muted_firing(
creates_incident && incidents_enabled && downtime.incident_mode.is_muted()
}
/// Whether a firing withheld as a repeat still feeds its incident.
#[cfg(feature = "enterprise")]
fn correlates_repeated_firing(creates_incident: bool, incidents_enabled: bool) -> bool {
creates_incident && incidents_enabled
}
/// Who acked a single-row alert's current firing, when that ack withholds this run's send.
async fn single_row_ack(
alert: &config::meta::alerts::alert::Alert,
@@ -3191,6 +3244,32 @@ fn records_delivery_state(alert_level_delivery: bool, delivery: &DeliveryDecisio
alert_level_delivery && delivery.resets_silence()
}
/// Whether a single-row firing at `level` repeats what its open recovery episode already sent.
fn single_row_repeats(
alert: &config::meta::alerts::alert::Alert,
level: Option<config::meta::alerts::level::AlertLevel>,
rollup: Option<&config::meta::alerts::state::AlertState>,
recorded_level: Option<i32>,
alert_level_delivery: bool,
) -> bool {
let (Some(level), Some(rollup)) = (level.filter(|l| l.is_firing()), rollup) else {
return false;
};
// Only the multi-level path records the delivered level; a single-level alert fires at one.
let last_notified = if alert_level_delivery {
recorded_level.and_then(config::meta::alerts::level::AlertLevel::from_i32)
} else {
Some(level)
};
alert.suppresses_repeats()
&& !alert.query_condition.multi_alert_enabled()
&& config::meta::alerts::dispatch::repeats_last_notification(
level,
rollup.episode_id.is_some(),
last_notified,
)
}
/// A delivered firing ends the Muted chip; a pending run persists before the downtime decision.
#[cfg(feature = "enterprise")]
fn clears_recorded_mute(outcome: &RunOutcome) -> bool {
@@ -3594,7 +3673,7 @@ async fn handle_alert_triggers(
.unwrap();
let last_states = if !alert.query_condition.multi_alert_enabled()
&& alert.pending_period_sec > 0
&& (alert.pending_period_sec > 0 || alert.suppresses_repeats())
{
load_tracked_group_states(&alert.get_unique_key()).await.inspect_err(|e|{
log::error!("[SCHEDULER trace_id {scheduler_trace_id}] alert {}/{} error in getting alert state: {e}",trigger.org,trigger.module_key);
@@ -3927,10 +4006,27 @@ async fn handle_alert_triggers(
} else {
None
};
let delivery = match (downtime.as_ref(), acked_by) {
(Some(downtime), _) => DeliveryDecision::SuppressedByDowntime(downtime.id.clone()),
(None, Some(by)) => DeliveryDecision::SuppressedByAck(by),
(None, None)
if level_delivery.should_deliver()
&& single_row_repeats(
&alert,
eval_level,
last_states.get(config::meta::alerts::state::ROLLUP_GROUP_KEY),
trigger_data.last_notified_level,
alert_level_delivery,
) =>
{
DeliveryDecision::SuppressedAsRepeat
}
(None, None) => level_delivery,
};
// D6: a suppressed run starts no silence window, so the first run after the window delivers.
if starts_silence_window(
trigger_results.data.is_some(),
downtime.is_some() || acked_by.is_some(),
delivery.withheld(),
alert.trigger_condition.silence,
evaluates_through_silence,
) {
@@ -3952,14 +4048,7 @@ async fn handle_alert_triggers(
// send notification
// Nowhere to send means no delivery to record: no silence window or episode.
let has_target = has_delivery_target(&alert);
let delivery = match (downtime.as_ref(), acked_by) {
(Some(downtime), _) => DeliveryDecision::SuppressedByDowntime(downtime.id.clone()),
(None, Some(by)) => DeliveryDecision::SuppressedByAck(by),
(None, None) => level_delivery,
};
if !delivery.should_deliver()
&& (alert_level_delivery || downtime.is_some() || delivery.acked_by().is_some())
{
if !delivery.should_deliver() && (alert_level_delivery || delivery.withheld()) {
log::info!(
"[SCHEDULER trace_id {scheduler_trace_id}] delivery suppressed ({delivery:?}) for {}/{}",
new_trigger.org,
@@ -4016,6 +4105,20 @@ async fn handle_alert_triggers(
)
.await;
}
#[cfg(feature = "enterprise")]
if delivery == DeliveryDecision::SuppressedAsRepeat
&& let Some(rows) = trigger_results.data.as_deref()
{
correlate_repeated_incident(
&scheduler_trace_id,
&alert,
rows,
triggered_at,
eval_level,
trigger_results.actual_value,
)
.await;
}
// What this evaluation DELIVERED, not what it decided to: the episode opens on a landed send.
let mut episode_delivered = false;
@@ -4213,6 +4316,9 @@ async fn handle_alert_triggers(
}
dispatch.record_downtime(&mut trigger_data_stream);
if dispatch.repeated {
trigger_data_stream.repeat_suppressed = Some(true);
}
// The rollup row was committed before anything was sent.
if let Some(outcome) = dispatch.rollup_rewrite()
&& let Some(alert_id) = alert.id.as_ref()
@@ -4486,6 +4592,8 @@ async fn handle_alert_triggers(
"alerts",
by.to_string(),
);
} else if delivery == DeliveryDecision::SuppressedAsRepeat {
trigger_data_stream.repeat_suppressed = Some(true);
}
} else if trigger_results.frozen {
// Frozen is not Normal: nothing was measured (§7.6). `Skipped` is
@@ -7384,6 +7492,7 @@ mod tests {
state_failed: false,
downtime_id: None,
acked_by: None,
repeated: false,
}
}
@@ -8944,6 +9053,14 @@ mod tests {
);
}
#[cfg(feature = "enterprise")]
#[test]
fn a_repeat_still_feeds_its_incident_only_when_incidents_apply() {
assert!(correlates_repeated_firing(true, true));
assert!(!correlates_repeated_firing(false, true));
assert!(!correlates_repeated_firing(true, false));
}
#[cfg(feature = "enterprise")]
#[test]
fn only_an_unsuppressed_matched_run_clears_the_recorded_mute() {
src/core/src/fleet.rs
+13 -3
View File
@@ -58,9 +58,19 @@ pub async fn get_pipelines(
}
/// The handler has already checked the caller's org-wide fleet access, so this runs as the server.
pub async fn list_audit(org_id: &str, query: AuditQuery) -> Result<AuditList, anyhow::Error> {
o2_enterprise::enterprise::fleet::audit::list_audit(org_id, query, now_micros(), search_audit)
.await
pub async fn list_audit(
org_id: &str,
query: AuditQuery,
fleets: Option<&HashSet<String>>,
) -> Result<AuditList, anyhow::Error> {
o2_enterprise::enterprise::fleet::audit::list_audit(
org_id,
query,
fleets,
now_micros(),
search_audit,
)
.await
}
/// The check `/prometheus/api/v1/query` makes per metric stream; a refused query is `None`.
src/core/src/http.rs
+1
View File
@@ -99,6 +99,7 @@ impl From<AlertError> for Response {
| AlertError::TzOffsetOutOfRange
| AlertError::CronHasNoFutureOccurrence { .. }
| AlertError::RecoveryOnRealtimeAlert
| AlertError::RepeatSuppressionOnRealtimeAlert
| AlertError::RecoveryDestinationsWithoutRecovery
| AlertError::RecoveryDestinationIsPlatform { .. }
| AlertError::RecoveryNeedsJsonPayload { .. }
src/core/src/lib.rs
+5 -1
View File
@@ -67,7 +67,7 @@ pub mod public_dashboards;
pub mod ratelimit;
pub mod rum_pa;
use search_service as search;
#[cfg(feature = "profiling")]
#[cfg(feature = "cpu-profiling")]
pub mod self_profiles;
pub mod self_reporting;
pub mod service;
@@ -88,6 +88,10 @@ pub mod users;
#[cfg(feature = "enterprise")]
pub mod workflows;
#[cfg(feature = "cloud")]
pub mod org_domain_ownership;
#[cfg(feature = "cloud")]
pub mod org_domain_requests;
/// One authorized AI operation, or `()` in builds without the quota module, so
/// the RCA signature keeps one shape instead of being cfg-gated at every caller.
#[cfg(feature = "cloud")]
src/core/src/logs/bulk.rs
+7 -2
View File
@@ -288,6 +288,12 @@ pub async fn ingest(
written_any = true;
}
for status in v.status {
// the stream status keeps the items but not the write's errors flag
bulk_res.errors |= status
.items
.iter()
.flat_map(|item| item.values())
.any(|item| item.error.is_some());
bulk_res.items.extend(status.items);
}
}
@@ -621,8 +627,7 @@ mod tests {
} else {
assert!(rejected.is_empty());
}
// not set: Fluent Bit retries a whole chunk on it, duplicating the other streams' rows
assert!(!resp.errors);
assert_eq!(resp.errors, bounded);
assert!(
infra::schema::exists("test_org_id", StreamType::Logs, "test_bulk_key_len_ok").await
);
src/core/src/logs/mod.rs
+2 -1
View File
@@ -396,6 +396,7 @@ fn reject_stream_rows(
.map(|(_, record)| record.get("_id").and_then(Value::as_str).map(String::from))
.chain(columnar.into_iter().flat_map(|mut c| c.take_bulk_ids()));
for doc_id in doc_ids {
bulk_res.errors = true;
bulk::add_record_status(
stream_name.to_string(),
doc_id,
@@ -798,7 +799,7 @@ mod tests {
let IngestionStatus::Bulk(bulk) = bulk else {
unreachable!()
};
assert!(!bulk.errors);
assert!(bulk.errors);
let ids: Vec<&str> = bulk
.items
.iter()
src/core/src/metrics/ingest.rs
+2 -2
View File
@@ -105,10 +105,10 @@ impl StreamKeyChecks {
}
/// Removes the records bound for streams that cannot be stored, as (stream, reason, records).
pub(super) fn drop_unstorable<T>(
pub(super) fn drop_unstorable<R>(
&mut self,
org_id: &str,
records_by_stream: &mut RecordsByStream<T>,
records_by_stream: &mut HashMap<String, Vec<R>>,
) -> Vec<(String, String, usize)> {
let mut dropped = Vec::new();
records_by_stream.retain(|stream_name, records| {
src/core/src/metrics/prom.rs
+94
View File
@@ -143,10 +143,15 @@ pub async fn remote_write(
// check if stream is deleting from cache
let mut stream_delete_status: HashMap<String, bool> = HashMap::new();
let mut skipped_records: u32 = 0;
// remote write has no partial success: a 4xx drops the whole batch, a 5xx retries it forever
let mut stream_key_checks = ingest::StreamKeyChecks::default();
// parse metadata
for item in request.metadata {
let metric_name = format_stream_name(item.metric_family_name.to_string());
if stream_key_checks.violation(org_id, &metric_name).is_some() {
continue;
}
let schema = infra::schema::get(org_id, &metric_name, StreamType::Metrics)
.await
.unwrap_or_else(|_| Schema::empty());
@@ -218,6 +223,8 @@ pub async fn remote_write(
}
}
unique_metrics.retain(|name| stream_key_checks.violation(org_id, name).is_none());
let mut preload_pipeline_time = 0u128;
let mut preload_uds_time = 0u128;
let mut preload_schema_time = 0u128;
@@ -379,6 +386,10 @@ pub async fn remote_write(
continue;
}
if stream_key_checks.violation(org_id, &metric_name).is_some() {
continue;
}
// Note: All configurations (pipeline, UDS, schema, partition, alerts) are now pre-loaded
// before the loop to avoid repeated async queries
@@ -552,6 +563,8 @@ pub async fn remote_write(
.map(|(record, timestamp)| (record, timestamp, None)),
);
}
// histogram bucket, exemplar and pipeline output streams can be too long when the metric fits
stream_key_checks.drop_unstorable(org_id, &mut json_data_by_stream);
let step_start = std::time::Instant::now();
for (stream_name, mut json_data) in json_data_by_stream {
@@ -2309,6 +2322,87 @@ mod tests {
assert!(matchers.iter().all(|m| m.is_match("http_requests")));
}
// cloud builds reject the unknown test org at the trial check
#[cfg(not(feature = "cloud"))]
#[tokio::test]
async fn remote_write_drops_only_the_streams_whose_key_is_too_long() {
use infra::file_list::STREAM_KEY_MAX_LEN;
use prost::Message;
use proto::prometheus_rpc;
let org_id = "test_org_id";
let now = Utc::now().timestamp_millis();
let series = |name: &str| prometheus_rpc::TimeSeries {
labels: vec![prometheus_rpc::Label {
name: NAME_LABEL.to_string(),
value: name.to_string(),
}],
samples: vec![prometheus_rpc::Sample {
value: 1.0,
timestamp: now,
}],
..Default::default()
};
let long_name = "m".repeat(STREAM_KEY_MAX_LEN);
// `_sum` fills the key exactly; `_count`, `_bucket` and `_exemplars` overflow it
let histogram_name =
"h".repeat(STREAM_KEY_MAX_LEN - format!("{org_id}/metrics/_sum").len());
let histogram = prometheus_rpc::TimeSeries {
samples: vec![],
exemplars: vec![prometheus_rpc::Exemplar {
labels: vec![],
value: 1.0,
timestamp: now,
}],
histograms: vec![prometheus_rpc::Histogram {
sum: 1.0,
count: Some(prometheus_rpc::histogram::Count::CountInt(1)),
zero_count: Some(prometheus_rpc::histogram::ZeroCount::ZeroCountInt(1)),
timestamp: now,
..Default::default()
}],
..series(&histogram_name)
};
let request = prometheus_rpc::WriteRequest {
timeseries: vec![
series("test_prom_key_len_ok"),
series(&long_name),
histogram,
],
metadata: vec![prometheus_rpc::MetricMetadata {
r#type: prometheus_rpc::metric_metadata::MetricType::Gauge as i32,
metric_family_name: long_name.clone(),
..Default::default()
}],
};
let body = snap::raw::Encoder::new()
.compress_vec(&request.encode_to_vec())
.unwrap();
remote_write(
org_id,
Bytes::from(body),
IngestUser::User("root@example.com".to_string()),
)
.await
.unwrap();
let bounded =
infra::file_list::stream_key_violation(org_id, StreamType::Metrics, &long_name)
.is_some();
let expected = [
("test_prom_key_len_ok".to_string(), true),
(format!("{histogram_name}_sum"), true),
(long_name, !bounded),
(format!("{histogram_name}_count"), !bounded),
(format!("{histogram_name}_bucket"), !bounded),
(exemplars_stream_name(&histogram_name), !bounded),
];
for (name, created) in expected {
let exists = infra::schema::exists(org_id, StreamType::Metrics, &name).await;
assert_eq!(exists, created, "{name}");
}
}
#[tokio::test]
async fn remote_write_rejects_a_snappy_header_declaring_more_than_the_limit() {
let mut n = get_config().limit.req_payload_limit as u64 + 1;
src/core/src/org_cleanup/coverage.rs
+2
View File
@@ -230,6 +230,8 @@ pub const COVERAGE: &[Coverage] = &[
table("workflow_errors", DB),
table("workflow_run_data", DB),
table("workflows", DB),
table("org_domain_ownership", DB),
table("org_domain_requests", DB),
covered(Kind::KvPrefix, "/cipher_keys/{org}/", DB),
covered(Kind::KvPrefix, "/compact/downsampling/{org}/", DB),
covered(Kind::KvPrefix, "/compact/files/{org}/", DB),
src/core/src/org_cleanup/mod.rs
+5 -8
View File
@@ -32,6 +32,9 @@ use infra::{
};
use sea_orm::TransactionTrait;
#[cfg(feature = "cloud")]
use crate::{org_domain_ownership, org_domain_requests};
const LOCK_KEY: &str = "/org_cleanup/worker_lock";
const MAX_ATTEMPTS: i32 = 10;
@@ -1034,17 +1037,11 @@ async fn delete_org_offsets_and_caches(org_id: &str) -> Result<(), anyhow::Error
Ok(())
}
// The mappings live in the `_meta` org's settings, so the org's own settings delete misses them.
#[cfg(feature = "cloud")]
async fn delete_org_domain_mappings(org_id: &str) -> Result<(), anyhow::Error> {
// The shared writer serializes this with the settings API and other orgs' cleanups.
let org_id = org_id.to_string();
crate::db::organization::update_org_setting(config::META_ORG_ID, move |settings| {
let before = settings.domain_org_mappings.len();
settings.domain_org_mappings.retain(|m| m.org_id != org_id);
settings.domain_org_mappings.len() != before
})
.await?;
org_domain_requests::delete_by_org(&org_id).await?;
org_domain_ownership::delete_by_org(&org_id).await?;
Ok(())
}
src/core/src/org_domain_ownership.rs
+288
View File
@@ -0,0 +1,288 @@
// Copyright 2026 OpenObserve Inc.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use std::sync::{Arc, LazyLock};
use hashbrown::HashMap;
use infra::{
coordinator::get_coordinator,
db::Event,
table::{
entity::{org_domain_ownership::Model, status_page_custom_domains},
org_domain_ownership::{OwnershipRecord, OwnershipState},
},
};
static CACHE: LazyLock<config::RwAHashMap<String, String>> =
LazyLock::new(|| tokio::sync::RwLock::new(HashMap::new()));
pub const ODO_PREFIX: &str = "/org_domain_ownership/";
const DOMAIN_EXPIRY_BUFFER_SEC: i64 = 60 * 30;
pub async fn init() {
let items = match infra::table::org_domain_ownership::list_active_domain_org_map().await {
Ok(v) => v,
Err(e) => {
log::error!("error in listing active domain mapping for init caching : {e}");
return;
}
};
let mut lock = CACHE.write().await;
for (domain, org) in items {
lock.insert(domain, org);
}
}
pub async fn get_cached_org_for_domain(domain: &str) -> Option<String> {
let lock = CACHE.read().await;
let org = lock.get(domain);
org.map(ToOwned::to_owned)
}
pub async fn remove_domain_from_cache(domain: &str) {
let mut lock = CACHE.write().await;
lock.remove(domain);
}
pub async fn add_domain_to_cache(domain: &str, org: &str) {
let mut lock = CACHE.write().await;
lock.insert(domain.to_string(), org.to_string());
}
pub async fn get_verified_org_for_domain(domain: &str) -> Result<Option<String>, anyhow::Error> {
let lock = CACHE.read().await;
let org = lock.get(domain);
if let Some(org) = org {
return Ok(Some(org.to_owned()));
}
drop(lock);
infra::table::org_domain_ownership::get_verified_org_for_domain(domain).await
}
pub async fn save_org_domain_mapping(org_id: &str, domain: &str) -> Result<(), anyhow::Error> {
let record = OwnershipRecord {
org_id: org_id.to_owned(),
domain: domain.to_owned(),
state: OwnershipState::Pending,
};
infra::table::org_domain_ownership::save_org_domain_mapping(record, false).await
}
pub async fn delete_linked_domain(record: Model) -> Result<(), anyhow::Error> {
infra::table::org_domain_ownership::delete_linked_domain(&record.org_id, &record.domain)
.await?;
// only sync cache for domain which is verified
// non verified domain is not going to be in cache
if record.verification_state == OwnershipState::Verified as i32 {
let mut lock = CACHE.write().await;
let cached = lock.get(&record.domain);
if cached.is_some_and(|v| v == &record.org_id) {
lock.remove(&record.domain);
}
drop(lock);
emit_sync_event(&record.org_id, &record.domain).await;
}
Ok(())
}
pub async fn get_domain_org_record(
org_id: &str,
domain: &str,
) -> Result<Option<Model>, anyhow::Error> {
let res = infra::table::org_domain_ownership::get_domain_org_record(org_id, domain).await?;
Ok(res)
}
pub async fn verify(mut record: Model) -> Result<Model, anyhow::Error> {
if record.meta_mapped {
return Ok(record);
}
let res = o2_enterprise::enterprise::status_pages::domain_verifier::check_domain_ownership(
status_page_custom_domains::Model {
domain: record.domain.clone(),
verification_token: record.verification_token.clone(),
last_checked_at: record.last_checked_at,
updated_at: record.updated_at,
verification_state: record.verification_state,
verification_failure_reason: record.verification_failure_reason,
verified_at: record.verified_at,
..Default::default()
},
Some(DOMAIN_EXPIRY_BUFFER_SEC),
)
.await?;
record.last_checked_at = res.last_checked_at;
record.updated_at = res.updated_at;
record.verification_state = res.verification_state;
record.verification_failure_reason = res.verification_failure_reason;
record.verified_at = res.verified_at;
infra::table::org_domain_ownership::save_domain_org_record(record.clone()).await?;
Ok(record)
}
pub async fn verify_domain_now(org_id: &str, domain: &str) -> Result<Model, anyhow::Error> {
let record = infra::table::org_domain_ownership::get_domain_org_record(org_id, domain).await?;
let Some(record) = record else {
return Err(anyhow::anyhow!(
"no mapping for domain {domain} found for org {org_id}"
));
};
if record.meta_mapped {
return Ok(record);
}
let updated_record = verify(record).await?;
if updated_record.verification_state == OwnershipState::Verified as i32 {
let mut lock = CACHE.write().await;
lock.insert(domain.to_owned(), org_id.to_owned());
}
emit_sync_event(org_id, domain).await;
Ok(updated_record)
}
pub async fn update_state(org_id: &str, domain: &str, new_state: i32) -> Result<(), anyhow::Error> {
let mut lock = CACHE.write().await;
let existing = lock.get(domain);
if new_state == OwnershipState::Verified as i32 {
lock.insert(domain.to_owned(), org_id.to_owned());
} else if existing.is_some_and(|v| org_id == v) {
lock.remove(domain);
}
drop(lock);
emit_sync_event(org_id, domain).await;
Ok(())
}
pub async fn get_domains_for_org(org_id: &str) -> Result<Vec<Model>, anyhow::Error> {
let records = infra::table::org_domain_ownership::get_domains_for_org(org_id).await?;
Ok(records)
}
pub async fn link_from_meta(org_id: &str, domain: &str) -> Result<(), anyhow::Error> {
let domains = get_domains_for_org(org_id).await?;
let mapped = domains
.into_iter()
.filter(|v| v.domain == domain)
.next()
.is_some();
if mapped {
infra::table::org_domain_ownership::mark_meta_mapped(org_id, domain).await?;
} else {
let record = OwnershipRecord {
org_id: org_id.to_owned(),
domain: domain.to_owned(),
state: OwnershipState::Verified,
};
infra::table::org_domain_ownership::save_org_domain_mapping(record, true).await?;
}
emit_sync_event(org_id, domain).await;
Ok(())
}
pub async fn delete_by_org(org_id: &str) -> Result<(), anyhow::Error> {
let domains = infra::table::org_domain_ownership::get_domains_for_org(org_id).await?;
for domain in domains {
delete_linked_domain(domain).await?;
}
Ok(())
}
async fn emit_sync_event(org_id: &str, domain: &str) {
let cluster_coordinator = get_coordinator().await;
if let Err(e) = cluster_coordinator
.put(
&format!("{ODO_PREFIX}{org_id}/{domain}"),
"".into(),
true,
None,
)
.await
{
log::error!(
"error sending cluster sync message for org domain ownership sync for org {org_id} : {e}"
);
}
}
pub async fn watch() -> Result<(), anyhow::Error> {
let cluster_coordinator = ::infra::db::get_coordinator().await;
let mut events = cluster_coordinator.watch(ODO_PREFIX).await?;
let events = Arc::get_mut(&mut events).unwrap();
log::info!("Start watching org_domain_ownership");
loop {
let ev = match events.recv().await {
Some(ev) => ev,
None => {
log::error!("watch_org_domain_ownership: event channel closed");
return Ok(());
}
};
if let Event::Put(ev) = ev {
let Some(key) = ev.key.strip_prefix(ODO_PREFIX) else {
log::error!("unexpected key for org domain prefix watch : {}", ev.key);
continue;
};
let Some((org, domain)) = key.split_once("/") else {
log::error!(
"invalid key received for org domain ownership sync : {}",
ev.key
);
continue;
};
log::info!("received sync event for org domain ownership for org {org}");
let record = match infra::table::org_domain_ownership::get_domain_org_record(
org, domain,
)
.await
{
Ok(v) => v,
Err(e) => {
log::error!(
"error in retrieving org domain ownership record from db for {org} domain {domain} : {e}"
);
continue;
}
};
let cached = get_cached_org_for_domain(domain).await;
let mut lock = CACHE.write().await;
match record {
None => {
if cached.is_some_and(|v| v == org) {
log::info!(
"removing org {org} domain {domain} from memory cache as record not found in db"
);
lock.remove(domain);
}
}
Some(v) => {
if v.verification_state == OwnershipState::Verified as i32 {
log::info!(
"added org {org} domain {domain} from memory cache as verified in db"
);
lock.insert(domain.to_owned(), org.to_owned());
} else if cached.is_some_and(|v| v == org) {
log::info!(
"removed org {org} domain {domain} from memory cache as not verified in db"
);
lock.remove(domain);
}
}
}
}
}
}
src/core/src/org_domain_requests.rs
+72
View File
@@ -0,0 +1,72 @@
// Copyright 2026 OpenObserve Inc.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use config::utils::time::now_micros;
use infra::table::{
entity::org_domain_requests::Model,
org_domain_requests::{self, Status},
};
pub async fn save_org_domain_request(
org_id: &str,
domain: &str,
req_by: &str,
) -> Result<(), anyhow::Error> {
let now = now_micros();
let req = Model {
id: config::ider::generate(),
org_id: org_id.to_string(),
domain: domain.trim().to_lowercase(),
created_at: now,
updated_at: now,
requested_by: req_by.to_string(),
answered_by: None,
status: Status::Pending as i32,
reason: None,
};
org_domain_requests::save_org_domain_request(req).await?;
Ok(())
}
pub async fn list_all_req() -> Result<Vec<Model>, anyhow::Error> {
Ok(org_domain_requests::list_all().await?)
}
pub async fn list_by_org(org_id: &str) -> Result<Vec<Model>, anyhow::Error> {
Ok(org_domain_requests::list_by_org(org_id).await?)
}
pub async fn get_req_by_id(id: &str) -> Result<Option<Model>, anyhow::Error> {
Ok(org_domain_requests::get_by_id(id).await?)
}
pub async fn get_req_by_org_domain(
org_id: &str,
domain: &str,
) -> Result<Option<Model>, anyhow::Error> {
Ok(org_domain_requests::get_by_org_domain(org_id, domain).await?)
}
pub async fn update_status(
id: &str,
user: &str,
new_status: Status,
reason: String,
) -> Result<(), anyhow::Error> {
Ok(org_domain_requests::update_status(id, user, new_status, reason).await?)
}
pub async fn delete_by_org(org_id: &str) -> Result<(), anyhow::Error> {
org_domain_requests::delete_by_org(org_id).await
}
src/core/src/organization.rs
+19
View File
@@ -25,6 +25,8 @@ use config::{
},
utils::{json, rand::generate_random_string, time},
};
#[cfg(feature = "cloud")]
use db::organization::get_org_setting;
use db::{self, org_users, user::is_root_user};
use infra::table::{self, org_users::UserOrgExpandedRecord};
#[cfg(feature = "enterprise")]
@@ -46,6 +48,8 @@ use {
#[cfg(feature = "cloud")]
use super::self_reporting::cloud_events::{CloudEvent, EventType, enqueue_cloud_event};
#[cfg(feature = "cloud")]
use crate::org_domain_ownership::get_cached_org_for_domain;
use crate::{
auth::{delete_org_tuples, save_org_tuples},
common::{
@@ -1103,6 +1107,21 @@ pub async fn generate_invitation(
return Err(anyhow::anyhow!("Email Domain not allowed for {invitee}"));
}
let Some((_, domain)) = invitee.split_once('@') else {
return Err(anyhow::anyhow!("invalid email id {invitee}"));
};
if let Some(linked_org) = get_cached_org_for_domain(domain).await {
let settings = get_org_setting(&linked_org).await?;
for mapping in settings.domain_org_mappings {
if mapping.domain == domain && mapping.org_id == org_id {
return Err(anyhow::anyhow!(
"invitee '{invitee}' domain is already mapped to org {org_id} so cannot send invite"
));
}
}
}
if get_billing_by_org_id(org_id).await?.is_none() {
// If the org we are inviting to is paid, its fine to send invitations
// irrespective of what other orgs invitees are part of.
src/core/src/public_dashboards/defaults.rs
+6
View File
@@ -113,6 +113,12 @@ impl<'a> Defaults<'a> {
self.values
}
/// Whether any value waits for a values query.
#[cfg(feature = "enterprise")]
pub fn has_pending(&self) -> bool {
!self.pending.is_empty()
}
/// The values queries `from` reaches through their stream, field and filter references.
#[cfg_attr(not(feature = "enterprise"), allow(dead_code))]
pub fn reachable_queries(
src/core/src/public_dashboards/mask.rs
+1
View File
@@ -236,6 +236,7 @@ mod tests {
#[test]
fn gate02_a_variable_from_a_lost_values_stream_is_nulled() {
let var = |label: &str, value| PublicVariable {
name: label.into(),
label: label.into(),
value,
tab_id: None,
src/core/src/public_dashboards/materialize.rs
+1
View File
@@ -1457,6 +1457,7 @@ pub(super) mod tests {
Run {
panels,
variables: vec![PublicVariable {
name: "svc".into(),
label: "svc".into(),
value: json!("checkout"),
tab_id: None,
src/core/src/public_dashboards/mod.rs
+318 -53
View File
@@ -352,9 +352,7 @@ pub struct LinkValues {
impl LinkValues {
pub fn new(dash: &Dashboard, frozen: Option<&str>) -> Self {
let values = Defaults::new(dash, parse_frozen_vars(frozen)).into_values();
Self {
vars: QueryVars::new(dash, values, None),
}
Self::from_values(dash, values)
}
/// The panel's query text as its build substitutes it; the window only sizes `$__interval`.
@@ -376,6 +374,12 @@ impl LinkValues {
let vars = self.vars.for_panel(tab_id, &panel.id);
plan_promql(panel, &vars, now - 3_600_000_000, now)
}
fn from_values(dash: &Dashboard, values: BTreeMap<String, serde_json::Value>) -> Self {
Self {
vars: QueryVars::new(dash, values, None),
}
}
}
/// One authorizer health answer per build or deadline check, asked only when a check denies.
@@ -808,7 +812,14 @@ pub async fn update_link(
let now = now_micros();
validate_dashboard(&dash, &cfg.frozen_variables, now).map_err(Invalid)?;
#[cfg(feature = "enterprise")]
authorize_variable_edit(&link, &dash, &cfg.frozen_variables, user_id).await?;
authorize_variable_edit(
&link,
&dash,
&cfg.frozen_variables,
range_changed(&link, &cfg.time_range).then_some(cfg.time_range.ranges.as_slice()),
user_id,
)
.await?;
let ranges = serde_json::to_string(&cfg.time_range.ranges)?;
let frozen = serde_json::to_string(&cfg.frozen_variables)?;
edit(&link.org_id, &link.id, |l| {
@@ -1508,15 +1519,29 @@ where
paused
}
/// Enterprise: the editor must read every stream a changed frozen value makes a panel read.
/// Whether an edit changes the set of ranges a link serves or its default range.
#[cfg(feature = "enterprise")]
fn range_changed(link: &PublicDashboard, policy: &TimeRangePolicy) -> bool {
let keys = |ranges: &[TimeRange]| ranges.iter().map(TimeRange::key).collect::<BTreeSet<_>>();
let stored = time_ranges(link);
keys(&stored.ranges) != keys(&policy.ranges) || stored.default.key() != policy.default.key()
}
/// Enterprise: the editor must read what a changed frozen value or range makes the link read.
#[cfg(feature = "enterprise")]
async fn authorize_variable_edit(
link: &PublicDashboard,
dash: &Dashboard,
frozen: &BTreeMap<String, serde_json::Value>,
changed_ranges: Option<&[TimeRange]>,
user_id: &str,
) -> Result<(), anyhow::Error> {
let sources = variable_edit_sources(link, dash, frozen);
let sources = match changed_ranges {
Some(ranges) => {
range_edit_sources(&ENGINE, link, dash, frozen, ranges, user_id, now_micros()).await?
}
None => variable_edit_sources(link, dash, frozen),
};
if sources.is_empty() {
return Ok(());
}
@@ -1560,57 +1585,190 @@ fn variable_edit_sources(
let values = LinkValues::new(dash, new_json.as_deref());
for tab in &v8.tabs {
for panel in &tab.panels {
let texts = template_texts(panel);
let texts: Vec<&str> = texts.iter().map(String::as_str).collect();
let reads_changed = values
.vars
.referenced(&tab.tab_id, &panel.id, &texts)
.iter()
.any(|(_, key)| reached.contains(key));
if !reads_changed {
continue;
}
let promql = panel.query_type == "promql";
let planned = promql
.then(|| values.plan_promql(&tab.tab_id, panel).ok())
.flatten();
for (qi, q) in panel.queries.iter().enumerate() {
let lang = if promql { "promql" } else { "sql" };
// A sent PromQL query is checked as the build sends it, a formula as its expansion.
let text = match planned.as_ref().and_then(|p| p[qi].clone()) {
Some(expr) => expr,
None => {
let Some(text) = query_text(q) else {
continue;
};
values.substitute(&tab.tab_id, &panel.id, text, lang)
}
};
sources.push(if promql {
QuerySource::PromQl {
org_id: link.org_id.clone(),
query: text,
}
} else {
QuerySource::Sql {
org_id: link.org_id.clone(),
sql: text,
default_type: q.fields.stream_type,
}
});
let vrl = q.vrl_function_query.as_deref().map(str::trim);
if let Some(vrl) = vrl.filter(|f| !promql && !f.is_empty()) {
sources.push(QuerySource::Vrl {
org_id: link.org_id.clone(),
source: vrl.to_string(),
});
}
if panel_keys(&values, tab, panel).any(|key| reached.contains(&key)) {
sources.extend(panel_sources(&link.org_id, &values, tab, panel));
}
}
}
dedup(sources)
}
/// Every query the link runs in each range, with its values queries run as the editor.
#[cfg(feature = "enterprise")]
async fn range_edit_sources(
search: &dyn PanelSearch,
link: &PublicDashboard,
dash: &Dashboard,
frozen: &BTreeMap<String, serde_json::Value>,
ranges: &[TimeRange],
user_id: &str,
now: i64,
) -> Result<Vec<crate::authz::QuerySource>, anyhow::Error> {
use crate::authz::QuerySource;
let Some(v8) = dash.v8.as_ref() else {
return Ok(vec![]);
};
let editor = PublicDashboard {
published_by: user_id.to_string(),
..link.clone()
};
let probe = AuthorizerProbe::default();
let family = TraceFamily::new(&link.org_id);
let mut sources = Vec::new();
// Values depend on the window only when a values query picks them.
let windows: Vec<&TimeRange> = if Defaults::new(dash, frozen.clone()).has_pending() {
ranges.iter().collect()
} else {
ranges.iter().take(1).collect()
};
for range in windows {
let range_key = range.key();
let (start, end) = materialize::window(range, now);
let ctx = Ctx {
search,
probe: &probe,
link: &editor,
dash,
identity: public_dashboard_identity(&link.org_id),
range_key: &range_key,
start,
end,
now,
family: &family,
};
let filled = fill_defaults(&ctx, frozen.clone()).await;
sources.extend(filled.deps.values().flat_map(|dep| {
dep.sources
.iter()
.map(|(stream_type, name)| QuerySource::Stream {
org_id: link.org_id.clone(),
stream_type: *stream_type,
name: name.clone(),
})
}));
if let Some(key) = filled.denied.first() {
return Err(Forbidden(format!(
"Unauthorized Access: no read permission for the values of variable {key}"
))
.into());
}
// A later build may resolve the value, so an unchecked panel must not be saved.
if let Some(key) = filled.failed.first() {
return Err(Invalid(format!(
"The values of variable {key} could not be loaded to check stream access for this range. Try again."
))
.into());
}
let values = LinkValues::from_values(dash, filled.values);
for tab in &v8.tabs {
for panel in &tab.panels {
sources.extend(panel_sources(&link.org_id, &values, tab, panel));
}
}
}
Ok(dedup(sources))
}
/// Keys of the scoped values the panel's queries read.
#[cfg(feature = "enterprise")]
fn panel_keys<'a>(
values: &'a LinkValues,
tab: &'a Tab,
panel: &'a Panel,
) -> impl Iterator<Item = String> + 'a {
let texts = template_texts(panel);
let texts: Vec<&str> = texts.iter().map(String::as_str).collect();
values
.vars
.referenced(&tab.tab_id, &panel.id, &texts)
.into_iter()
.map(|(_, key)| key)
}
/// The queries and VRL functions a panel's build sends; a custom chart never runs.
#[cfg(feature = "enterprise")]
fn panel_sources(
org_id: &str,
values: &LinkValues,
tab: &Tab,
panel: &Panel,
) -> Vec<crate::authz::QuerySource> {
use crate::authz::QuerySource;
let mut sources = Vec::new();
if panel.typ == "custom_chart" || withholds_vrl(org_id, panel) {
return sources;
}
let promql = panel.query_type == "promql";
let planned = promql
.then(|| values.plan_promql(&tab.tab_id, panel).ok())
.flatten();
for (qi, q) in panel.queries.iter().enumerate() {
let lang = if promql { "promql" } else { "sql" };
// A sent PromQL query is checked as the build sends it, a formula as its expansion.
let text = match planned.as_ref().map(|p| p[qi].clone()) {
Some(Some(expr)) => expr,
// A hidden or empty query is never sent; formulas carry their inputs.
Some(None) => continue,
None => {
let Some(text) = query_text(q) else {
continue;
};
values.substitute(&tab.tab_id, &panel.id, text, lang)
}
};
sources.push(if promql {
QuerySource::PromQl {
org_id: org_id.to_string(),
query: text,
}
} else {
QuerySource::Sql {
org_id: org_id.to_string(),
sql: text,
default_type: q.fields.stream_type,
}
});
let vrl = q.vrl_function_query.as_deref().map(str::trim);
if let Some(vrl) = vrl.filter(|f| !promql && !f.is_empty()) {
sources.push(QuerySource::Vrl {
org_id: org_id.to_string(),
source: vrl.to_string(),
});
}
}
sources
}
/// Whether the build withholds the panel for a VRL whose reads it cannot prove, as `run_sql_panel`
/// does.
#[cfg(feature = "enterprise")]
fn withholds_vrl(org_id: &str, panel: &Panel) -> bool {
panel.query_type != "promql"
&& panel.queries.iter().any(|q| {
q.vrl_function_query
.as_deref()
.map(str::trim)
.filter(|f| !f.is_empty())
.is_some_and(|vrl| {
transform::enrichment_tables_read(vrl, org_id).is_err()
|| skips_vrl_in_search(vrl)
})
})
}
#[cfg(feature = "enterprise")]
fn dedup(sources: Vec<crate::authz::QuerySource>) -> Vec<crate::authz::QuerySource> {
let mut unique = Vec::with_capacity(sources.len());
for source in sources {
if !unique.contains(&source) {
unique.push(source);
}
}
unique
}
/// The outermost explicit LIMIT of a SQL query; None without one or when it does not parse.
fn explicit_limit(sql: &str) -> Option<i64> {
use sqlparser::{
@@ -4208,6 +4366,113 @@ mod tests {
);
}
#[cfg(feature = "enterprise")]
#[test]
fn only_a_new_range_set_or_default_is_a_range_change() {
let (mut link, ..) = ctx_parts();
link.time_ranges = Some(
json!([{ "type": "relative", "secs": 86400 }, { "type": "relative", "secs": 3600 }])
.to_string(),
);
link.default_range_key = Some("r86400".into());
let policy = |ranges: serde_json::Value, default: usize| -> TimeRangePolicy {
let ranges: Vec<TimeRange> = serde_json::from_value(ranges).unwrap();
TimeRangePolicy {
default: ranges[default],
ranges,
}
};
let sorted =
json!([{ "type": "relative", "secs": 3600 }, { "type": "relative", "secs": 86400 }]);
assert!(
!range_changed(&link, &policy(sorted.clone(), 1)),
"a reorder"
);
assert!(range_changed(&link, &policy(sorted, 0)), "a new default");
assert!(range_changed(
&link,
&policy(json!([{ "type": "relative", "secs": 2592000 }]), 0)
));
}
#[cfg(feature = "enterprise")]
#[tokio::test]
async fn a_range_edit_checks_every_panel_with_the_values_its_build_picks() {
let (link, ..) = ctx_parts();
let dash = Dashboard {
version: 8,
v8: Some(
serde_json::from_value(json!({
"tabs": [{ "tabId": "t1", "name": "", "panels": [
panel("b", Some("SELECT * FROM fixed WHERE env = '$env'")),
panel("d", Some("SELECT * FROM \"$pick\"")),
panel("e", Some("SELECT * FROM \"$sub\"")),
panel("f", Some("SELECT * FROM \"$sub\"")),
typed_panel("g", "custom_chart", "sql", "SELECT * FROM never_run", None),
]}],
"variables": { "list": [
{ "type": "custom", "name": "env", "label": "E", "value": "prod" },
{ "type": "query_values", "name": "pick", "label": "P",
"query_data": { "stream_type": "logs", "stream": "names", "field": "n" } },
{ "type": "query_values", "name": "sub", "label": "U",
"query_data": { "stream_type": "logs", "stream": "$pick", "field": "n" } }
]}
}))
.unwrap(),
),
..Default::default()
};
let frozen: BTreeMap<String, serde_json::Value> =
serde_json::from_value(json!({ "env": "prod" })).unwrap();
let ranges: Vec<TimeRange> =
serde_json::from_value(json!([{ "type": "relative", "secs": 3600 }])).unwrap();
let search = FakeSearch {
hits: 1,
..Default::default()
};
let sources = range_edit_sources(&search, &link, &dash, &frozen, &ranges, "editor@x", 0)
.await
.unwrap();
let mut texts: Vec<String> = sources
.into_iter()
.map(|s| match s {
crate::authz::QuerySource::Sql { sql, .. } => sql,
crate::authz::QuerySource::Stream { name, .. } => format!("stream {name}"),
other => panic!("{other:?}"),
})
.collect();
texts.sort();
assert_eq!(
texts,
[
"SELECT * FROM \"v0\"",
"SELECT * FROM fixed WHERE env = 'prod'",
"stream names",
"stream v0",
],
"the dependent values query reads the stream its parent picks"
);
let denied = FakeSearch {
hits: 1,
refuse: Some((NotAvailableReason::Unauthorized, "FROM \"names\"")),
..Default::default()
};
let err = range_edit_sources(&denied, &link, &dash, &frozen, &ranges, "editor@x", 0)
.await
.unwrap_err();
assert!(err.downcast_ref::<Forbidden>().is_some(), "{err}");
// A failed lookup leaves the dependent panels unchecked, so the edit cannot be saved.
let failed = FakeSearch {
hits: 1,
fail_sql: Some("FROM \"names\""),
..Default::default()
};
let err = range_edit_sources(&failed, &link, &dash, &frozen, &ranges, "editor@x", 0)
.await
.unwrap_err();
assert!(err.downcast_ref::<Invalid>().is_some(), "{err}");
}
#[cfg(all(feature = "enterprise", feature = "test-utils"))]
#[tokio::test]
async fn gate04_a_link_variable_edit_selecting_an_unreadable_stream_is_refused() {
@@ -4226,7 +4491,7 @@ mod tests {
let to = |value: &str| -> BTreeMap<String, serde_json::Value> {
BTreeMap::from([("src.p.c".to_string(), json!(value))])
};
let err = authorize_variable_edit(&link, &dash, &to("secret"), &editor)
let err = authorize_variable_edit(&link, &dash, &to("secret"), None, &editor)
.await
.unwrap_err();
assert!(err.downcast_ref::<Forbidden>().is_some());
@@ -4235,7 +4500,7 @@ mod tests {
"Unauthorized Access: no read permission on logs/secret"
);
assert!(
authorize_variable_edit(&link, &dash, &to("safe"), &editor)
authorize_variable_edit(&link, &dash, &to("safe"), None, &editor)
.await
.is_ok()
);
src/core/src/self_profiles/job.rs
+31 -24
View File
@@ -48,9 +48,10 @@ pub async fn run() {
let cpu_secs = cfg.self_profiles.cpu_secs;
let lock = Arc::new(Mutex::new(()));
log::info!(
"[SELF-PROFILES] starting background loop: interval={}s cpu={}s url={}",
"[SELF-PROFILES] starting background loop: interval={}s cpu={}s memory={} url={}",
cfg.self_profiles.interval_secs,
cpu_secs,
cfg!(feature = "profiling"),
if cfg.self_profiles.url.is_empty() {
"local"
} else {
@@ -73,32 +74,38 @@ pub async fn run() {
}
async fn run_cycle(cpu_secs: u64) -> Result<(), String> {
let host = config::get_config().common.instance_name.clone();
let extra = [("host.name", host.as_str())];
let cfg = config::get_config();
let extra = [
("host.name", cfg.common.instance_name.as_str()),
("cluster.name", cfg.common.cluster_name.as_str()),
];
match sample::dump_cpu_profile(cpu_secs, CPU_FREQUENCY_HZ).await {
Ok(bytes) => match convert::pprof_bytes_to_otlp(&bytes, &extra) {
Ok(req) => {
if let Err(e) = sink::ingest(req).await {
log::error!("[SELF-PROFILES] CPU ingest failed: {e}");
}
}
Err(e) => log::error!("[SELF-PROFILES] CPU convert failed: {e}"),
},
Err(e) => log::error!("[SELF-PROFILES] CPU sample failed: {e}"),
}
let cpu = sample::dump_cpu_profile(cpu_secs, CPU_FREQUENCY_HZ).await;
ingest_sample("CPU", cpu, &extra).await;
match sample::dump_memory_pprof().await {
Ok(bytes) => match convert::pprof_bytes_to_otlp(&bytes, &extra) {
Ok(req) => {
if let Err(e) = sink::ingest(req).await {
log::error!("[SELF-PROFILES] memory ingest failed: {e}");
}
}
Err(e) => log::error!("[SELF-PROFILES] memory convert failed: {e}"),
},
Err(e) => log::error!("[SELF-PROFILES] memory sample failed: {e}"),
#[cfg(feature = "profiling")]
{
let memory = sample::dump_memory_pprof().await;
ingest_sample("memory", memory, &extra).await;
}
Ok(())
}
async fn ingest_sample(kind: &str, sampled: Result<Vec<u8>, String>, extra: &[(&str, &str)]) {
let bytes = match sampled {
Ok(bytes) => bytes,
Err(e) => {
log::error!("[SELF-PROFILES] {kind} sample failed: {e}");
return;
}
};
match convert::pprof_bytes_to_otlp(&bytes, extra) {
Ok(req) => {
if let Err(e) = sink::ingest(req).await {
log::error!("[SELF-PROFILES] {kind} ingest failed: {e}");
}
}
Err(e) => log::error!("[SELF-PROFILES] {kind} convert failed: {e}"),
}
}
src/core/src/self_profiles/mod.rs
+4 -3
View File
@@ -13,7 +13,7 @@
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
//! Self CPU/memory profiling into `_meta.self_profiles` (feature `profiling`).
//! Self CPU (feature `cpu-profiling`) and memory (feature `profiling`) profiling.
mod convert;
mod job;
@@ -21,8 +21,9 @@ mod sample;
mod sink;
pub use job::run;
pub use sample::{dump_cpu_profile, dump_memory_flamegraph, dump_memory_pprof, jemalloc_stats};
pub use sample::dump_cpu_profile;
#[cfg(feature = "profiling")]
pub use sample::{dump_memory_flamegraph, dump_memory_pprof, jemalloc_stats};
pub(crate) const STREAM_NAME: &str = "self_profiles";
pub(crate) const SERVICE_NAME: &str = "openobserve";
pub(crate) const CPU_FREQUENCY_HZ: i32 = 100;
src/core/src/self_profiles/sample.rs
+3
View File
@@ -33,6 +33,7 @@ pub async fn dump_cpu_profile(duration_secs: u64, frequency: i32) -> Result<Vec<
}
/// Dump jemalloc heap profile as pprof protobuf bytes.
#[cfg(feature = "profiling")]
pub async fn dump_memory_pprof() -> Result<Vec<u8>, String> {
let prof_ctl = jemalloc_pprof::PROF_CTL
.as_ref()
@@ -52,6 +53,7 @@ pub async fn dump_memory_pprof() -> Result<Vec<u8>, String> {
}
/// Dump jemalloc heap flamegraph SVG bytes (for `/debug/profile/memory`).
#[cfg(feature = "profiling")]
pub async fn dump_memory_flamegraph() -> Result<Vec<u8>, String> {
let prof_ctl = jemalloc_pprof::PROF_CTL
.as_ref()
@@ -71,6 +73,7 @@ pub async fn dump_memory_flamegraph() -> Result<Vec<u8>, String> {
}
/// Read current jemalloc stats (for `/debug/profile/stats`).
#[cfg(feature = "profiling")]
pub fn jemalloc_stats() -> Result<serde_json::Value, String> {
tikv_jemalloc_ctl::epoch::mib()
.map_err(|e| format!("Failed to get jemalloc epoch mib: {e}"))?
src/core/src/self_profiles/sink.rs
+3 -3
View File
@@ -15,11 +15,11 @@
use std::time::Duration;
use config::{META_ORG_ID, get_config, meta::otlp::OtlpRequestType};
use config::{DEFAULT_STREAM_NAME, META_ORG_ID, get_config, meta::otlp::OtlpRequestType};
use ingestion_common::{IngestUser, SystemJobType};
use opentelemetry_proto::tonic::collector::profiles::v1development::ExportProfilesServiceRequest;
use super::{STREAM_NAME, convert};
use super::convert;
const REMOTE_REQUEST_TIMEOUT: Duration = Duration::from_secs(10);
const REMOTE_CONNECT_TIMEOUT: Duration = Duration::from_secs(5);
@@ -40,7 +40,7 @@ async fn ingest_local(request: ExportProfilesServiceRequest) -> Result<(), Strin
META_ORG_ID,
request,
OtlpRequestType::HttpProtobuf,
Some(STREAM_NAME),
Some(DEFAULT_STREAM_NAME),
IngestUser::SystemJob(SystemJobType::SelfReporting),
)
.await
src/core/src/traces/service_graph/api.rs
+226 -18
View File
@@ -21,6 +21,17 @@ use axum::response::Response as HttpResponse;
use common::meta::http::HttpResponse as MetaHttpResponse;
use serde::Deserialize;
#[cfg(feature = "enterprise")]
const EDGE_HISTORY_WINDOW_MICROS: i64 = 24 * 60 * 60 * 1_000_000;
// Allows clock skew while keeping every bound far from i64 overflow in the step alignment.
#[cfg(any(feature = "enterprise", test))]
const MAX_FUTURE_MICROS: i64 = 24 * 60 * 60 * 1_000_000;
// A stream's own retention can outlast the global one, so the cap never drops below the default.
#[cfg(any(feature = "enterprise", test))]
const MIN_MAX_WINDOW_DAYS: i64 = 3650;
#[cfg(any(feature = "enterprise", test))]
const DAY_MICROS: i64 = 24 * 60 * 60 * 1_000_000;
/// Query parameters for service graph API
#[derive(Debug, Deserialize)]
pub struct ServiceGraphQuery {
@@ -53,6 +64,7 @@ pub struct ServiceGraphQuery {
),
responses(
(status = 200, description = "Success", content_type = "application/json", body = Object),
(status = 400, description = "Invalid time window (negative start_time, end_time not after start_time, end_time more than a day ahead, or longer than the data retention, at least 3650 days) or a stream name the metrics query cannot quote", content_type = "application/json", body = String),
(status = 403, description = "Forbidden - Enterprise feature", content_type = "application/json", body = String),
(status = 500, description = "Internal Server Error", content_type = "application/json", body = ()),
),
@@ -65,15 +77,16 @@ pub async fn get_current_topology(
axum::extract::Path(org_id): axum::extract::Path<String>,
axum::extract::Query(query): axum::extract::Query<ServiceGraphQuery>,
) -> HttpResponse {
// Resolve the current window boundaries
let (start_time, end_time) =
if let (Some(start), Some(end)) = (query.start_time, query.end_time) {
(start, end)
} else {
let now = chrono::Utc::now().timestamp_micros();
let window_micros = super::DEFAULT_QUERY_WINDOW_MINUTES * 60 * 1_000_000;
(now - window_micros, now)
};
let (start_time, end_time) = match resolve_window(
query.start_time,
query.end_time,
chrono::Utc::now().timestamp_micros(),
super::DEFAULT_QUERY_WINDOW_MINUTES * 60 * 1_000_000,
max_window_days(config::get_config().compact.data_retention_days),
) {
Ok(window) => window,
Err(e) => return MetaHttpResponse::bad_request(e),
};
let filter = v4_read_filter(&query);
if let Err(e) = filter.validate() {
@@ -152,7 +165,7 @@ pub struct EdgeHistoryQuery {
),
responses(
(status = 200, description = "Success", content_type = "application/json", body = Object),
(status = 400, description = "A service or stream name the metrics query cannot quote", content_type = "application/json", body = String),
(status = 400, description = "Invalid time window (negative start_time, end_time not after start_time, end_time more than a day ahead, or longer than the data retention, at least 3650 days), or a service or stream name the metrics query cannot quote", content_type = "application/json", body = String),
(status = 403, description = "Forbidden - Enterprise feature"),
(status = 500, description = "Internal Server Error", content_type = "application/json", body = ()),
),
@@ -166,13 +179,16 @@ pub async fn get_edge_history(
use super::v4::history::{EdgeFilter, fetch_edge_history};
let (start_time, end_time) =
if let (Some(start), Some(end)) = (query.start_time, query.end_time) {
(start, end)
} else {
let now = chrono::Utc::now().timestamp_micros();
(now - 24 * 60 * 60 * 1_000_000, now)
};
let (start_time, end_time) = match resolve_window(
query.start_time,
query.end_time,
chrono::Utc::now().timestamp_micros(),
EDGE_HISTORY_WINDOW_MICROS,
max_window_days(config::get_config().compact.data_retention_days),
) {
Ok(window) => window,
Err(e) => return MetaHttpResponse::bad_request(e),
};
let non_empty = |v: Option<String>| v.filter(|s| !s.is_empty());
let filter = EdgeFilter {
client: non_empty(query.client_service),
@@ -228,10 +244,137 @@ pub async fn get_current_topology(
MetaHttpResponse::forbidden("Not Supported")
}
#[cfg(all(test, feature = "enterprise"))]
/// The longest window a request may ask for: the global data retention, at least 3650 days.
#[cfg(any(feature = "enterprise", test))]
fn max_window_days(retention_days: i64) -> i64 {
retention_days.max(MIN_MAX_WINDOW_DAYS)
}
#[cfg(any(feature = "enterprise", test))]
fn resolve_window(
start: Option<i64>,
end: Option<i64>,
now: i64,
default_span_micros: i64,
max_days: i64,
) -> Result<(i64, i64), String> {
let (Some(start), Some(end)) = (start, end) else {
return Ok((now - default_span_micros, now));
};
if start < 0 {
return Err("start_time must not be negative".to_string());
}
if end <= start {
return Err("end_time must be after start_time".to_string());
}
if end > now.saturating_add(MAX_FUTURE_MICROS) {
return Err("end_time must be at most one day after the current time".to_string());
}
if end - start > max_days.saturating_mul(DAY_MICROS) {
return Err(format!(
"the window from start_time to end_time must not exceed {max_days} days"
));
}
Ok((start, end))
}
#[cfg(test)]
mod tests {
use super::*;
const NOW: i64 = 1_800_000_000_000_000;
const SPAN: i64 = 3_600_000_000;
const MAX_DAYS: i64 = 3650;
#[test]
fn test_resolve_window_uses_an_explicit_window() {
assert_eq!(
resolve_window(Some(1_000), Some(2_000), NOW, SPAN, MAX_DAYS),
Ok((1_000, 2_000))
);
assert_eq!(
resolve_window(Some(0), Some(1), NOW, SPAN, MAX_DAYS),
Ok((0, 1))
);
}
#[test]
fn test_resolve_window_rejects_a_negative_start() {
assert!(resolve_window(Some(-1), Some(2_000), NOW, SPAN, MAX_DAYS).is_err());
assert!(resolve_window(Some(i64::MIN), Some(i64::MAX), NOW, SPAN, MAX_DAYS).is_err());
}
#[test]
fn test_resolve_window_rejects_an_end_not_after_the_start() {
assert!(resolve_window(Some(2_189_567), Some(-1_358), NOW, SPAN, MAX_DAYS).is_err());
assert!(resolve_window(Some(2_000), Some(2_000), NOW, SPAN, MAX_DAYS).is_err());
assert!(resolve_window(Some(2_000), Some(1_999), NOW, SPAN, MAX_DAYS).is_err());
}
#[test]
fn test_resolve_window_rejects_an_end_beyond_tomorrow_and_near_i64_max() {
assert_eq!(
resolve_window(
Some(NOW),
Some(NOW + MAX_FUTURE_MICROS),
NOW,
SPAN,
MAX_DAYS
),
Ok((NOW, NOW + MAX_FUTURE_MICROS))
);
assert!(
resolve_window(
Some(NOW),
Some(NOW + MAX_FUTURE_MICROS + 1),
NOW,
SPAN,
MAX_DAYS
)
.is_err()
);
assert!(resolve_window(Some(i64::MAX - 1), Some(i64::MAX), NOW, SPAN, MAX_DAYS).is_err());
assert!(resolve_window(Some(0), Some(i64::MAX), NOW, SPAN, MAX_DAYS).is_err());
assert!(resolve_window(Some(0), Some(1), i64::MAX, SPAN, MAX_DAYS).is_ok());
}
#[test]
fn test_resolve_window_caps_the_span_at_the_given_days() {
let max_span = MAX_DAYS * DAY_MICROS;
assert_eq!(
resolve_window(Some(NOW - max_span), Some(NOW), NOW, SPAN, MAX_DAYS),
Ok((NOW - max_span, NOW))
);
let err =
resolve_window(Some(NOW - max_span - 1), Some(NOW), NOW, SPAN, MAX_DAYS).unwrap_err();
assert!(err.contains("3650 days"), "{err}");
let longer = 7300 * DAY_MICROS;
assert!(resolve_window(Some(NOW - longer), Some(NOW), NOW, SPAN, 7300).is_ok());
assert!(resolve_window(Some(0), Some(NOW), NOW, SPAN, i64::MAX).is_ok());
}
#[test]
fn test_max_window_days_follows_a_longer_retention() {
assert_eq!(max_window_days(7300), 7300);
assert_eq!(max_window_days(3650), 3650);
assert_eq!(
max_window_days(30),
3650,
"a stream may keep data longer than the default"
);
assert_eq!(max_window_days(0), 3650, "0 means unbounded retention");
assert_eq!(max_window_days(-1), 3650);
}
#[test]
fn test_resolve_window_defaults_when_a_bound_is_missing() {
let default = Ok((NOW - SPAN, NOW));
assert_eq!(resolve_window(None, None, NOW, SPAN, MAX_DAYS), default);
assert_eq!(resolve_window(Some(-5), None, NOW, SPAN, MAX_DAYS), default);
assert_eq!(resolve_window(None, Some(-5), NOW, SPAN, MAX_DAYS), default);
}
#[cfg(feature = "enterprise")]
#[test]
fn test_v4_read_filter_maps_stream_and_env_only() {
let query = |stream: Option<&str>, env: Option<&str>| ServiceGraphQuery {
@@ -253,4 +396,69 @@ mod tests {
assert_eq!(f.trace_stream.as_deref(), Some("default"));
assert_eq!(f.agent_env.as_deref(), Some("prod"));
}
#[cfg(feature = "enterprise")]
#[tokio::test]
async fn test_handlers_reject_an_invalid_window_with_400() {
use axum::extract::{Path, Query};
let topology = get_current_topology(
Path("default".to_string()),
Query(ServiceGraphQuery {
start_time: Some(2_189_567),
end_time: Some(-1_358),
filter: None,
stream_name: None,
agent_id: None,
agent_name: None,
agent_env: None,
}),
)
.await;
assert_eq!(topology.status(), axum::http::StatusCode::BAD_REQUEST);
let history = get_edge_history(
Path("default".to_string()),
Query(EdgeHistoryQuery {
client_service: None,
server_service: None,
start_time: Some(-1),
end_time: Some(10),
stream_name: None,
}),
)
.await;
assert_eq!(history.status(), axum::http::StatusCode::BAD_REQUEST);
let near_max = get_edge_history(
Path("default".to_string()),
Query(EdgeHistoryQuery {
client_service: None,
server_service: None,
start_time: Some(i64::MAX - 1),
end_time: Some(i64::MAX),
stream_name: None,
}),
)
.await;
assert_eq!(near_max.status(), axum::http::StatusCode::BAD_REQUEST);
let near_max_topology = get_current_topology(
Path("default".to_string()),
Query(ServiceGraphQuery {
start_time: Some(i64::MAX - 1),
end_time: Some(i64::MAX),
filter: None,
stream_name: None,
agent_id: None,
agent_name: None,
agent_env: None,
}),
)
.await;
assert_eq!(
near_max_topology.status(),
axum::http::StatusCode::BAD_REQUEST
);
}
}
src/core/src/traces/service_graph/v4/history.rs
+10 -2
View File
@@ -90,7 +90,7 @@ pub async fn fetch_edge_history(
/// Coarser than a minute only when the range would exceed `MAX_POINTS`; always whole minutes.
fn step_micros(start: i64, end: i64) -> i64 {
let span_secs = ((end - start) / 1_000_000).max(1) as u64;
let span_secs = (end.saturating_sub(start) / 1_000_000).max(1) as u64;
let minutes = span_secs
.div_ceil(MAX_POINTS as u64)
.div_ceil(MIN_STEP_SECS as u64);
@@ -99,7 +99,7 @@ fn step_micros(start: i64, end: i64) -> i64 {
/// Evaluation times sit on step boundaries, where the rollup's samples are, inside the range.
fn eval_bounds(start: i64, end: i64, step: i64) -> Option<(i64, i64)> {
let first = align_down(start + step - 1, step) + step;
let first = align_down(start.checked_add(step - 1)?, step).checked_add(step)?;
let last = align_down(end, step);
(first <= last).then_some((first, last))
}
@@ -217,6 +217,14 @@ mod tests {
assert_eq!(eval_bounds(0, MIN - 1, MIN), None);
}
#[test]
fn test_step_and_bounds_survive_windows_near_i64_max() {
let step = step_micros(i64::MAX - 1, i64::MAX);
assert_eq!(step, MIN);
assert_eq!(eval_bounds(i64::MAX - 1, i64::MAX, step), None);
assert!(step_micros(i64::MIN, i64::MAX) > 0);
}
#[test]
fn test_plan_queries() {
let plan = plan("server=\"b\"", MIN);
src/core/src/traces/service_graph/v4/read.rs
+1 -1
View File
@@ -257,7 +257,7 @@ where
Fut: Future<Output = Result<Vec<InstantValue>, anyhow::Error>>,
{
// Range selectors are left-open; +1ms keeps the `start` sample, not a baseline 1ms before it.
let range_ms = ((end - start) / 1_000).max(1_000) + 1;
let range_ms = (end.saturating_sub(start) / 1_000).max(1_000) + 1;
let plan =
|v4_only| plan_queries(filter, v4_only, range_ms, start, end).map_err(anyhow::Error::msg);
let mut results = run_plan(&plan(true)?, &run).await;
src/core/src/users.rs
+15 -10
View File
@@ -1876,12 +1876,7 @@ mod tests {
}
async fn create_query_history_table() {
use sea_orm::{ConnectionTrait, Schema};
let conn = get_orm_client_rw().await;
let backend = conn.get_database_backend();
let mut stmt = Schema::new(backend)
.create_table_from_entity(infra_table::entity::query_history::Entity);
conn.execute(backend.build(stmt.if_not_exists()))
infra_table::query_history::create_table_for_tests(get_orm_client_rw().await)
.await
.unwrap();
}
@@ -1906,14 +1901,24 @@ mod tests {
};
create_new_user(user).await.unwrap();
for org in ["dummy", "other"] {
infra_table::query_history::record(org, email, "up", serde_json::json!({}), 1)
.await
.unwrap();
infra_table::query_history::record(infra_table::query_history::NewRun {
org_id: org,
user_email: email,
source: "logs",
query: "up",
query_key: None,
streams: &[],
status: "ok",
context: serde_json::json!({}),
now_us: 1,
})
.await
.unwrap();
}
}
async fn history_count(org: &str, email: &str) -> usize {
infra_table::query_history::list(org, email, None, None, 10, 0)
infra_table::query_history::list(org, email, &Default::default(), 10, 0)
.await
.unwrap()
.len()
src/db/src/authz.rs
+28 -16
View File
@@ -90,11 +90,31 @@ pub async fn list_objects_for_user(
permission: &str,
object_type: &str,
) -> anyhow::Result<Option<Vec<String>>> {
let openfga_config = o2_openfga::config::get_config();
if crate::user::is_root_user(user_id)
|| !openfga_config.enabled
|| !openfga_config.list_only_permitted
{
if !o2_openfga::config::get_config().list_only_permitted {
return Ok(None);
}
list_permitted_objects(org_id, user_id, permission, object_type).await
}
#[cfg(not(feature = "enterprise"))]
pub async fn list_objects_for_user(
_org_id: &str,
_user_id: &str,
_permission: &str,
_object_type: &str,
) -> anyhow::Result<Option<Vec<String>>> {
Ok(None)
}
/// [`list_objects_for_user`] whatever `list_only_permitted` says, for lists that carry secrets.
#[cfg(feature = "enterprise")]
pub async fn list_permitted_objects(
org_id: &str,
user_id: &str,
permission: &str,
object_type: &str,
) -> anyhow::Result<Option<Vec<String>>> {
if crate::user::is_root_user(user_id) || !o2_openfga::config::get_config().enabled {
return Ok(None);
}
@@ -113,20 +133,12 @@ pub async fn list_objects_for_user(
)
.await
.map_err(|_| anyhow::anyhow!("Unauthorized Access"))?;
log::debug!("list_objects_for_user for user {user_id} from {org_id} org returns: {objects:#?}");
log::debug!(
"list_permitted_objects for user {user_id} from {org_id} org returns: {objects:#?}"
);
Ok(Some(objects))
}
#[cfg(not(feature = "enterprise"))]
pub async fn list_objects_for_user(
_org_id: &str,
_user_id: &str,
_permission: &str,
_object_type: &str,
) -> anyhow::Result<Option<Vec<String>>> {
Ok(None)
}
/// OpenFGA object and parent type of `obj`; `None` when either type is unmapped.
#[cfg(feature = "enterprise")]
fn ofga_object(obj_type: &str, obj: &Authz) -> Option<(String, &'static str)> {
src/infra/src/table/alerts/mod.rs
+4
View File
@@ -220,6 +220,7 @@ impl TryFrom<alerts::Model> for MetaAlert {
alert.notify_on_recovery = value.notify_on_recovery.unwrap_or(false);
alert.recovery_destinations = recovery_destinations;
alert.keep_firing_for = value.keep_firing_for_seconds.unwrap_or(0);
alert.suppress_repeats = value.suppress_repeats.unwrap_or(false);
Ok(alert)
}
@@ -958,6 +959,7 @@ fn update_mutable_fields(
Some(serde_json::to_string(&alert.recovery_destinations)?)
};
let keep_firing_for_seconds = alert.keep_firing_for;
let suppress_repeats = alert.suppress_repeats;
alert_am.name = Set(name);
alert_am.is_real_time = Set(is_real_time);
@@ -1016,6 +1018,7 @@ fn update_mutable_fields(
alert_am.notify_on_recovery = Set(Some(notify_on_recovery));
alert_am.recovery_destinations = Set(recovery_destinations);
alert_am.keep_firing_for_seconds = Set(Some(keep_firing_for_seconds));
alert_am.suppress_repeats = Set(Some(suppress_repeats));
Ok(())
}
@@ -1096,6 +1099,7 @@ pub(super) mod tests {
notify_on_recovery: None,
recovery_destinations: None,
keep_firing_for_seconds: None,
suppress_repeats: None,
}
}
src/infra/src/table/entity/alerts.rs
+3
View File
@@ -92,6 +92,8 @@ pub struct Model {
/// How long the condition must stay clear before the episode recovers.
/// NULL or 0 recovers on the first clear evaluation.
pub keep_firing_for_seconds: Option<i64>,
/// NULL predates the feature, which is the same as `false`.
pub suppress_repeats: Option<bool>,
}
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
@@ -178,6 +180,7 @@ mod tests {
notify_on_recovery: None,
recovery_destinations: None,
keep_firing_for_seconds: None,
suppress_repeats: None,
};
assert_eq!(m.id, "alert-1");
assert_eq!(m.name, "High Error Rate");
src/infra/src/table/entity/mod.rs
+2
View File
@@ -80,6 +80,8 @@ pub mod oncall_user_contacts;
pub mod online_eval_jobs;
pub mod org_ai_toolsets;
pub mod org_cleanup_tasks;
pub mod org_domain_ownership;
pub mod org_domain_requests;
pub mod org_ingestion_tokens;
pub mod org_storage_providers;
pub mod org_telephony;
src/infra/src/table/entity/org_domain_ownership.rs
+26
View File
@@ -0,0 +1,26 @@
//! `SeaORM` Entity for the domain_org_ownership table.
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, DeriveEntityModel, serde::Serialize, serde::Deserialize)]
#[sea_orm(table_name = "org_domain_ownership")]
pub struct Model {
#[sea_orm(primary_key, auto_increment = false)]
pub id: String,
pub org_id: String,
pub domain: String,
pub verification_token: String,
pub verification_state: i32,
pub verification_failure_reason: Option<i32>,
pub verified_at: Option<i64>,
pub released_at: Option<i64>,
pub last_checked_at: Option<i64>,
pub created_at: i64,
pub updated_at: i64,
pub meta_mapped: bool,
}
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {}
impl ActiveModelBehavior for ActiveModel {}
src/infra/src/table/entity/org_domain_requests.rs
+23
View File
@@ -0,0 +1,23 @@
//! `SeaORM` Entity for the domain_org_requests table.
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, DeriveEntityModel, serde::Serialize, serde::Deserialize)]
#[sea_orm(table_name = "org_domain_requests")]
pub struct Model {
#[sea_orm(primary_key, auto_increment = false)]
pub id: String,
pub org_id: String,
pub domain: String,
pub created_at: i64,
pub updated_at: i64,
pub requested_by: String,
pub answered_by: Option<String>,
pub status: i32,
pub reason: Option<String>,
}
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
pub enum Relation {}
impl ActiveModelBehavior for ActiveModel {}
src/infra/src/table/entity/prelude.rs
+3 -1
View File
@@ -21,7 +21,9 @@ pub use super::{
llm_prompt_label_events::Entity as LlmPromptLabelEvents,
llm_prompt_labels::Entity as LlmPromptLabels, llm_prompt_versions::Entity as LlmPromptVersions,
llm_prompt_webhook_deliveries::Entity as LlmPromptWebhookDeliveries,
llm_prompts::Entity as LlmPrompts, org_ingestion_tokens::Entity as OrgIngestionTokens,
llm_prompts::Entity as LlmPrompts, org_domain_ownership::Entity as OrgDomainOwnership,
org_domain_requests::Entity as OrgDomainRequests,
org_ingestion_tokens::Entity as OrgIngestionTokens,
org_storage_providers::Entity as OrgStorageProviders, org_users::Entity as OrgUsers,
organizations::Entity as Organizations, pipeline_last_errors::Entity as PipelineLastErrors,
re_pattern_stream_map::Entity as RePatternStreamMap, re_patterns::Entity as RePatterns,
src/infra/src/table/entity/query_history.rs
+13
View File
@@ -26,7 +26,20 @@ pub struct Model {
pub query: String,
pub context: Json,
pub starred: bool,
/// First run of this row; never changed after insert.
pub created_at: i64,
#[sea_orm(column_type = "Text")]
pub source: String,
/// Hex sha256 of the merge key; NULL on rows written before the key existed.
pub query_key: Option<String>,
/// Comma-wrapped stream list (`,a,b,`), empty when there are none.
#[sea_orm(column_type = "Text")]
pub streams: String,
pub run_count: i64,
/// Last run of this row.
pub updated_at: i64,
#[sea_orm(column_type = "Text")]
pub status: String,
}
#[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]
src/infra/src/table/entity/status_page_custom_domains.rs
+3 -1
View File
@@ -2,7 +2,9 @@
use sea_orm::entity::prelude::*;
#[derive(Clone, Debug, PartialEq, DeriveEntityModel, serde::Serialize, serde::Deserialize)]
#[derive(
Clone, Default, Debug, PartialEq, DeriveEntityModel, serde::Serialize, serde::Deserialize,
)]
#[sea_orm(table_name = "status_page_custom_domains")]
pub struct Model {
#[sea_orm(primary_key, auto_increment = false)]
src/infra/src/table/entity/synthetics_checks.rs
+1 -1
View File
@@ -31,7 +31,7 @@ pub struct Model {
/// When the scheduler last fanned out this check (microseconds).
pub last_triggered_at: i64,
/// Denormalised status from the most recent completed check.
/// 0=Unknown, 1=Up, 2=Warning, 3=Down
/// 0=Unknown, 1=Up, 2=Warning, 3=Down, 4=Error (no verdict: dispatch/lease failure)
pub last_check_status: i32,
/// Runs that have failed back to back; reset to 0 by a pass. Compared
/// against `settings.alert_if_fails` to decide whether to notify.
src/infra/src/table/migration/m20261025_000005_add_held_action_to_slack_link_requests.rs
-6
View File
@@ -46,12 +46,6 @@ impl MigrationTrait for Migration {
}
}
/// Adds the column to a freshly created table (test setups that skip the migrator).
pub(crate) async fn add_column<C: sea_orm::ConnectionTrait>(db: &C) -> Result<(), DbErr> {
let stmt = db.get_database_backend().build(&add_column_statement());
db.execute(stmt).await.map(|_| ())
}
fn add_column_statement() -> TableAlterStatement {
Table::alter()
.table(SlackLinkRequests::Table)
src/infra/src/table/migration/m20261027_000001_alter_query_history.rs
+644
View File
@@ -0,0 +1,644 @@
// Copyright 2026 OpenObserve Inc.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
//! Adds source, merge key, streams, run count, last run and status to `query_history`.
use sea_orm_migration::prelude::*;
const TABLE: &str = "query_history";
const OLD_ORG_USER_CREATED_AT_IDX: &str = "query_history_org_user_created_at_idx";
const OLD_STARRED_CREATED_AT_IDX: &str = "query_history_starred_created_at_idx";
const ORG_USER_UPDATED_AT_IDX: &str = "query_history_org_user_updated_at_idx";
const ORG_USER_SOURCE_UPDATED_AT_IDX: &str = "query_history_org_user_source_updated_at_idx";
const STARRED_UPDATED_AT_IDX: &str = "query_history_starred_updated_at_idx";
const ORG_USER_SOURCE_KEY_UIDX: &str = "query_history_org_user_source_key_uidx";
#[derive(DeriveMigrationName)]
pub struct Migration;
#[async_trait::async_trait]
impl MigrationTrait for Migration {
async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> {
// NOT NULL DEFAULT keeps v1.1.x nodes inserting during a rolling upgrade.
for mut def in [
ColumnDef::new(QueryHistory::Source)
.text()
.not_null()
.default("metrics")
.to_owned(),
ColumnDef::new(QueryHistory::QueryKey)
.char_len(64)
.null()
.to_owned(),
ColumnDef::new(QueryHistory::Streams)
.text()
.not_null()
.default("")
.to_owned(),
ColumnDef::new(QueryHistory::RunCount)
.big_integer()
.not_null()
.default(1)
.to_owned(),
ColumnDef::new(QueryHistory::UpdatedAt)
.big_integer()
.not_null()
.default(0)
.to_owned(),
ColumnDef::new(QueryHistory::Status)
.text()
.not_null()
.default("ok")
.to_owned(),
] {
add_column(manager, &mut def).await?;
}
// v1.1.x stored each row's last run in created_at; the guard makes a rerun a no-op.
manager
.get_connection()
.execute_unprepared(
"UPDATE query_history SET updated_at = created_at WHERE updated_at < created_at",
)
.await?;
for name in [OLD_ORG_USER_CREATED_AT_IDX, OLD_STARRED_CREATED_AT_IDX] {
drop_index(manager, name).await?;
}
create_index(
manager,
ORG_USER_UPDATED_AT_IDX,
&[
QueryHistory::OrgId,
QueryHistory::UserEmail,
QueryHistory::UpdatedAt,
],
false,
)
.await?;
create_index(
manager,
ORG_USER_SOURCE_UPDATED_AT_IDX,
&[
QueryHistory::OrgId,
QueryHistory::UserEmail,
QueryHistory::Source,
QueryHistory::UpdatedAt,
],
false,
)
.await?;
create_index(
manager,
STARRED_UPDATED_AT_IDX,
&[QueryHistory::Starred, QueryHistory::UpdatedAt],
false,
)
.await?;
// NULL keys on v1.1.x rows are distinct, so they never collide here.
create_index(
manager,
ORG_USER_SOURCE_KEY_UIDX,
&[
QueryHistory::OrgId,
QueryHistory::UserEmail,
QueryHistory::Source,
QueryHistory::QueryKey,
],
true,
)
.await
}
async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> {
for name in [
ORG_USER_SOURCE_KEY_UIDX,
STARRED_UPDATED_AT_IDX,
ORG_USER_SOURCE_UPDATED_AT_IDX,
ORG_USER_UPDATED_AT_IDX,
] {
drop_index(manager, name).await?;
}
// A repeated or partial rollback has already dropped updated_at.
if manager
.has_column(TABLE, &QueryHistory::UpdatedAt.to_string())
.await?
{
manager
.get_connection()
.execute_unprepared(
"UPDATE query_history SET created_at = updated_at WHERE updated_at > created_at",
)
.await?;
}
create_index(
manager,
OLD_ORG_USER_CREATED_AT_IDX,
&[
QueryHistory::OrgId,
QueryHistory::UserEmail,
QueryHistory::CreatedAt,
],
false,
)
.await?;
create_index(
manager,
OLD_STARRED_CREATED_AT_IDX,
&[QueryHistory::Starred, QueryHistory::CreatedAt],
false,
)
.await?;
for column in [
QueryHistory::Status,
QueryHistory::UpdatedAt,
QueryHistory::RunCount,
QueryHistory::Streams,
QueryHistory::QueryKey,
QueryHistory::Source,
] {
if manager.has_column(TABLE, &column.to_string()).await? {
manager
.alter_table(
Table::alter()
.table(QueryHistory::Table)
.drop_column(column)
.to_owned(),
)
.await?;
}
}
Ok(())
}
}
#[derive(DeriveIden, Clone, Copy)]
enum QueryHistory {
Table,
OrgId,
UserEmail,
Starred,
CreatedAt,
Source,
QueryKey,
Streams,
RunCount,
UpdatedAt,
Status,
}
async fn add_column(manager: &SchemaManager<'_>, def: &mut ColumnDef) -> Result<(), DbErr> {
if manager.has_column(TABLE, &def.get_column_name()).await? {
return Ok(());
}
manager
.alter_table(
Table::alter()
.table(QueryHistory::Table)
.add_column(def.to_owned())
.to_owned(),
)
.await
}
async fn drop_index(manager: &SchemaManager<'_>, name: &str) -> Result<(), DbErr> {
manager
.drop_index(
Index::drop()
.if_exists()
.name(name)
.table(QueryHistory::Table)
.to_owned(),
)
.await
}
async fn create_index(
manager: &SchemaManager<'_>,
name: &str,
columns: &[QueryHistory],
unique: bool,
) -> Result<(), DbErr> {
let mut index = Index::create();
index.if_not_exists().name(name).table(QueryHistory::Table);
if unique {
index.unique();
}
for column in columns {
index.col(*column);
}
manager.create_index(index.to_owned()).await
}
#[cfg(test)]
mod tests {
use sea_orm::{ConnectionTrait, Database, DatabaseConnection, DbBackend, Statement};
use sea_orm_migration::{MigrationName, MigrationTrait};
use super::*;
type Row = (
String,
String,
Option<String>,
String,
i64,
i64,
i64,
String,
);
async fn v1_1_table_with_rows() -> DatabaseConnection {
let db = Database::connect("sqlite::memory:").await.unwrap();
seed_v1_1_table(&db).await;
db
}
async fn seed_v1_1_table(db: &DatabaseConnection) {
super::super::m20261004_000001_create_query_history::Migration
.up(&SchemaManager::new(db))
.await
.unwrap();
for (id, starred, created_at) in [("old", false, 100), ("star", true, 200)] {
db.execute_unprepared(&format!(
"INSERT INTO query_history (id, org_id, user_email, query, context, starred, \
created_at) VALUES ('{id}', 'default', 'a@x.com', 'up', '{{\"step\":\"1m\"}}', \
{starred}, {created_at})"
))
.await
.unwrap();
}
}
/// A throwaway schema on the CI Postgres; `None` where none is configured.
async fn postgres_schema() -> Option<(DatabaseConnection, DatabaseConnection, String)> {
let dsn = std::env::var("ZO_META_POSTGRES_DSN").ok()?;
// The SQLite lane sets the variable empty rather than unsetting it.
if dsn.trim().is_empty() {
return None;
}
let schema = format!(
"qh_migration_test_{}",
config::ider::generate().to_ascii_lowercase()
);
let admin = Database::connect(&dsn).await.unwrap();
admin
.execute_unprepared(&format!(r#"CREATE SCHEMA "{schema}""#))
.await
.unwrap();
let mut options = sea_orm::ConnectOptions::new(dsn);
options
.max_connections(1)
.set_schema_search_path(schema.clone());
Some((admin, Database::connect(options).await.unwrap(), schema))
}
async fn rows(db: &DatabaseConnection) -> Vec<Row> {
db.query_all(Statement::from_string(
DbBackend::Sqlite,
"SELECT id, source, query_key, streams, run_count, created_at, updated_at, status \
FROM query_history ORDER BY id",
))
.await
.unwrap()
.iter()
.map(|r| {
(
r.try_get_by_index(0).unwrap(),
r.try_get_by_index(1).unwrap(),
r.try_get_by_index(2).unwrap(),
r.try_get_by_index(3).unwrap(),
r.try_get_by_index(4).unwrap(),
r.try_get_by_index(5).unwrap(),
r.try_get_by_index(6).unwrap(),
r.try_get_by_index(7).unwrap(),
)
})
.collect()
}
async fn index_columns(db: &DatabaseConnection, index: &str) -> Vec<Option<String>> {
db.query_all(Statement::from_string(
DbBackend::Sqlite,
format!("SELECT name FROM pragma_index_info('{index}') ORDER BY seqno"),
))
.await
.unwrap()
.iter()
.map(|r| r.try_get::<Option<String>>("", "name").unwrap())
.collect()
}
#[tokio::test]
async fn test_up_backfills_updated_at_from_created_at_on_a_populated_v1_1_table() {
let db = v1_1_table_with_rows().await;
let manager = SchemaManager::new(&db);
Migration.up(&manager).await.unwrap();
let to_row = |id: &str, created_at: i64, updated_at: i64| -> Row {
(
id.to_string(),
"metrics".to_string(),
None,
String::new(),
1,
created_at,
updated_at,
"ok".to_string(),
)
};
assert_eq!(
rows(&db).await,
vec![to_row("old", 100, 100), to_row("star", 200, 200)],
"every v1.1.x row gets updated_at = created_at"
);
db.execute_unprepared("UPDATE query_history SET updated_at = 900 WHERE id = 'old'")
.await
.unwrap();
Migration.up(&manager).await.expect("idempotent");
assert_eq!(
rows(&db).await,
vec![to_row("old", 100, 900), to_row("star", 200, 200)],
"a rerun leaves a row that ran after the upgrade alone"
);
let listed = crate::table::query_history::list_with(
&db,
"default",
"a@x.com",
&crate::table::query_history::ListFilter {
sources: vec!["metrics".to_string()],
..Default::default()
},
10,
0,
)
.await
.unwrap();
assert_eq!(
listed
.iter()
.map(|r| (r.id.as_str(), r.updated_at))
.collect::<Vec<_>>(),
vec![("old", 900), ("star", 200)]
);
}
#[tokio::test]
async fn test_up_swaps_indexes() {
let db = v1_1_table_with_rows().await;
let manager = SchemaManager::new(&db);
Migration.up(&manager).await.unwrap();
for old in [OLD_ORG_USER_CREATED_AT_IDX, OLD_STARRED_CREATED_AT_IDX] {
assert!(!manager.has_index(TABLE, old).await.unwrap(), "{old}");
}
let names = |cols: &[Option<&str>]| -> Vec<Option<String>> {
cols.iter().map(|c| c.map(str::to_string)).collect()
};
assert_eq!(
index_columns(&db, ORG_USER_UPDATED_AT_IDX).await,
names(&[Some("org_id"), Some("user_email"), Some("updated_at")])
);
assert_eq!(
index_columns(&db, ORG_USER_SOURCE_UPDATED_AT_IDX).await,
names(&[
Some("org_id"),
Some("user_email"),
Some("source"),
Some("updated_at")
])
);
assert_eq!(
index_columns(&db, STARRED_UPDATED_AT_IDX).await,
names(&[Some("starred"), Some("updated_at")])
);
assert_eq!(
index_columns(&db, ORG_USER_SOURCE_KEY_UIDX).await,
names(&[
Some("org_id"),
Some("user_email"),
Some("source"),
Some("query_key")
])
);
}
#[tokio::test]
async fn test_up_leaves_exactly_the_agreed_index_set() {
let db = v1_1_table_with_rows().await;
Migration.up(&SchemaManager::new(&db)).await.unwrap();
let indexes: Vec<(String, bool)> = db
.query_all(Statement::from_string(
DbBackend::Sqlite,
"SELECT name, \"unique\" FROM pragma_index_list('query_history') \
WHERE origin = 'c' ORDER BY name",
))
.await
.unwrap()
.iter()
.map(|r| {
(
r.try_get_by_index(0).unwrap(),
r.try_get_by_index::<i64>(1).unwrap() == 1,
)
})
.collect();
assert_eq!(
indexes,
vec![
(ORG_USER_SOURCE_KEY_UIDX.to_string(), true),
(ORG_USER_SOURCE_UPDATED_AT_IDX.to_string(), false),
(ORG_USER_UPDATED_AT_IDX.to_string(), false),
(STARRED_UPDATED_AT_IDX.to_string(), false),
]
);
}
#[tokio::test]
async fn test_unique_key_allows_null_keys_and_refuses_a_duplicate_key() {
let db = v1_1_table_with_rows().await;
Migration.up(&SchemaManager::new(&db)).await.unwrap();
let insert = |id: &str, key: &str| {
format!(
"INSERT INTO query_history (id, org_id, user_email, query, context, created_at, \
query_key) VALUES ('{id}', 'default', 'a@x.com', 'up', '{{}}', 1, {key})"
)
};
db.execute_unprepared(&insert("legacy", "NULL"))
.await
.expect("a v1.1.x node inserts with no key");
db.execute_unprepared(&insert("k1", "'abc'")).await.unwrap();
assert!(db.execute_unprepared(&insert("k2", "'abc'")).await.is_err());
}
#[tokio::test]
async fn test_down_restores_the_v1_1_table() {
let db = v1_1_table_with_rows().await;
let manager = SchemaManager::new(&db);
Migration.up(&manager).await.unwrap();
Migration.down(&manager).await.unwrap();
for column in [
"source",
"query_key",
"streams",
"run_count",
"updated_at",
"status",
] {
assert!(
!manager.has_column(TABLE, column).await.unwrap(),
"{column}"
);
}
assert!(
manager
.has_index(TABLE, OLD_ORG_USER_CREATED_AT_IDX)
.await
.unwrap()
);
assert!(
manager
.has_index(TABLE, OLD_STARRED_CREATED_AT_IDX)
.await
.unwrap()
);
}
#[tokio::test]
async fn test_down_keeps_the_last_run_of_rows_an_old_node_wrote() {
let db = v1_1_table_with_rows().await;
let manager = SchemaManager::new(&db);
Migration.up(&manager).await.unwrap();
db.execute_unprepared(
"INSERT INTO query_history (id, org_id, user_email, query, context, created_at) \
VALUES ('mid', 'default', 'a@x.com', 'up', '{}', 300)",
)
.await
.unwrap();
db.execute_unprepared("UPDATE query_history SET updated_at = 900 WHERE id = 'old'")
.await
.unwrap();
Migration.down(&manager).await.unwrap();
let created: Vec<(String, i64)> = db
.query_all(Statement::from_string(
DbBackend::Sqlite,
"SELECT id, created_at FROM query_history ORDER BY id",
))
.await
.unwrap()
.iter()
.map(|r| {
(
r.try_get_by_index(0).unwrap(),
r.try_get_by_index(1).unwrap(),
)
})
.collect();
assert_eq!(
created,
vec![
("mid".to_string(), 300),
("old".to_string(), 900),
("star".to_string(), 200)
]
);
}
#[tokio::test]
async fn test_down_runs_twice() {
let db = v1_1_table_with_rows().await;
let manager = SchemaManager::new(&db);
Migration.up(&manager).await.unwrap();
Migration.down(&manager).await.unwrap();
Migration
.down(&manager)
.await
.expect("a second rollback is a no-op");
}
#[tokio::test]
async fn test_postgres_up_and_down_on_a_populated_v1_1_table_when_ci_provides_it() {
let Some((admin, db, schema)) = postgres_schema().await else {
return;
};
seed_v1_1_table(&db).await;
let manager = SchemaManager::new(&db);
Migration.up(&manager).await.unwrap();
Migration.up(&manager).await.expect("idempotent");
for index in [
ORG_USER_UPDATED_AT_IDX,
ORG_USER_SOURCE_UPDATED_AT_IDX,
STARRED_UPDATED_AT_IDX,
ORG_USER_SOURCE_KEY_UIDX,
] {
assert!(manager.has_index(TABLE, index).await.unwrap(), "{index}");
}
let listed = crate::table::query_history::list_with(
&db,
"default",
"a@x.com",
&crate::table::query_history::ListFilter {
sources: vec!["metrics".to_string()],
..Default::default()
},
10,
0,
)
.await
.unwrap();
// Postgres pads the char(27) id.
assert_eq!(
listed
.iter()
.map(|r| (r.id.trim(), r.updated_at))
.collect::<Vec<_>>(),
vec![("star", 200), ("old", 100)]
);
let insert = |id: &str| {
format!(
"INSERT INTO query_history (id, org_id, user_email, query, context, created_at, \
query_key) VALUES ('{id}', 'default', 'a@x.com', 'up', '{{}}', 1, 'abc')"
)
};
db.execute_unprepared(&insert("k1")).await.unwrap();
assert!(db.execute_unprepared(&insert("k2")).await.is_err());
Migration.down(&manager).await.unwrap();
Migration
.down(&manager)
.await
.expect("a second rollback is a no-op");
assert!(!manager.has_column(TABLE, "updated_at").await.unwrap());
assert!(
manager
.has_index(TABLE, OLD_ORG_USER_CREATED_AT_IDX)
.await
.unwrap()
);
db.close().await.unwrap();
admin
.execute_unprepared(&format!(r#"DROP SCHEMA "{schema}" CASCADE"#))
.await
.unwrap();
}
#[test]
fn test_migration_name() {
assert_eq!(Migration.name(), "m20261027_000001_alter_query_history");
}
}
src/infra/src/table/migration/m20261030_000001_add_suppress_repeats_to_alerts.rs
+92
View File
@@ -0,0 +1,92 @@
// Copyright 2026 OpenObserve Inc.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use sea_orm_migration::prelude::*;
const TABLE: &str = "alerts";
const COLUMN: &str = "suppress_repeats";
#[derive(DeriveMigrationName)]
pub struct Migration;
#[async_trait::async_trait]
impl MigrationTrait for Migration {
async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> {
// SQLite emits a plain ADD COLUMN for add_column_if_not_exists, so guard explicitly.
if manager.has_column(TABLE, COLUMN).await? {
return Ok(());
}
manager.alter_table(add_column_statement()).await
}
async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> {
if !manager.has_column(TABLE, COLUMN).await? {
return Ok(());
}
manager
.alter_table(
Table::alter()
.table(Alerts::Table)
.drop_column(Alerts::SuppressRepeats)
.to_owned(),
)
.await
}
}
#[derive(DeriveIden)]
enum Alerts {
Table,
SuppressRepeats,
}
fn add_column_statement() -> TableAlterStatement {
Table::alter()
.table(Alerts::Table)
.add_column(ColumnDef::new(Alerts::SuppressRepeats).boolean().null())
.to_owned()
}
#[cfg(test)]
mod tests {
use sea_orm::{ConnectionTrait, Database};
use super::*;
#[test]
fn postgres() {
assert_eq!(
&add_column_statement().to_string(PostgresQueryBuilder),
r#"ALTER TABLE "alerts" ADD COLUMN "suppress_repeats" bool NULL"#
);
}
#[tokio::test]
async fn test_up_reruns_without_adding_the_column_twice() {
let db = Database::connect("sqlite::memory:").await.unwrap();
db.execute_unprepared("CREATE TABLE alerts (id varchar(27) PRIMARY KEY)")
.await
.unwrap();
let manager = SchemaManager::new(&db);
Migration.up(&manager).await.expect("first run");
Migration.up(&manager).await.expect("second run");
assert!(manager.has_column(TABLE, COLUMN).await.unwrap());
Migration.down(&manager).await.unwrap();
Migration.down(&manager).await.unwrap();
assert!(!manager.has_column(TABLE, COLUMN).await.unwrap());
}
}
src/infra/src/table/migration/m20261030_000001_create_org_domain_ownership_table.rs
+135
View File
@@ -0,0 +1,135 @@
// Copyright 2026 OpenObserve Inc.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use sea_orm_migration::prelude::*;
#[derive(DeriveMigrationName)]
pub struct Migration;
#[async_trait::async_trait]
impl MigrationTrait for Migration {
async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> {
manager.create_table(create_domains_stmt()).await?;
manager.create_index(domain_org_idx()).await?;
Ok(())
}
async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> {
manager
.drop_table(Table::drop().table(OrgDomainOwnership::Table).to_owned())
.await?;
Ok(())
}
}
fn create_domains_stmt() -> TableCreateStatement {
Table::create()
.table(OrgDomainOwnership::Table)
.if_not_exists()
.col(
ColumnDef::new(OrgDomainOwnership::Id)
.string_len(27)
.not_null()
.primary_key(),
)
.col(
ColumnDef::new(OrgDomainOwnership::OrgId)
.string_len(100)
.not_null(),
)
.col(
ColumnDef::new(OrgDomainOwnership::Domain)
.string_len(253)
.not_null(),
)
.col(
ColumnDef::new(OrgDomainOwnership::VerificationToken)
.string_len(128)
.not_null(),
)
// 0 pending, 1 verified, 2 failed.
.col(
ColumnDef::new(OrgDomainOwnership::VerificationState)
.integer()
.not_null()
.default(0),
)
// 0 record-missing, 1 value-mismatch, 2 dns-resolution-failed.
.col(
ColumnDef::new(OrgDomainOwnership::VerificationFailureReason)
.integer()
.null(),
)
.col(
ColumnDef::new(OrgDomainOwnership::VerifiedAt)
.big_integer()
.null(),
)
.col(
ColumnDef::new(OrgDomainOwnership::ReleasedAt)
.big_integer()
.null(),
)
.col(
ColumnDef::new(OrgDomainOwnership::LastCheckedAt)
.big_integer()
.null(),
)
.col(
ColumnDef::new(OrgDomainOwnership::CreatedAt)
.big_integer()
.not_null(),
)
.col(
ColumnDef::new(OrgDomainOwnership::UpdatedAt)
.big_integer()
.not_null(),
)
.col(
ColumnDef::new(OrgDomainOwnership::MetaMapped)
.boolean()
.not_null()
.default(false),
)
.to_owned()
}
fn domain_org_idx() -> IndexCreateStatement {
sea_query::Index::create()
.if_not_exists()
.name("org_domain_ownership_org_domain_idx")
.table(OrgDomainOwnership::Table)
.col(OrgDomainOwnership::OrgId)
.col(OrgDomainOwnership::Domain)
.col(OrgDomainOwnership::VerificationState)
.to_owned()
}
#[derive(DeriveIden)]
enum OrgDomainOwnership {
Table,
Id,
OrgId,
Domain,
VerificationToken,
VerificationState,
VerificationFailureReason,
VerifiedAt,
ReleasedAt,
LastCheckedAt,
CreatedAt,
UpdatedAt,
MetaMapped,
}
src/infra/src/table/migration/m20261030_000002_create_org_domain_ownership_req_table.rs
+98
View File
@@ -0,0 +1,98 @@
// Copyright 2026 OpenObserve Inc.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use sea_orm_migration::prelude::*;
#[derive(DeriveMigrationName)]
pub struct Migration;
#[async_trait::async_trait]
impl MigrationTrait for Migration {
async fn up(&self, manager: &SchemaManager) -> Result<(), DbErr> {
manager.create_table(create_domaim_req_stmt()).await?;
Ok(())
}
async fn down(&self, manager: &SchemaManager) -> Result<(), DbErr> {
manager
.drop_table(Table::drop().table(OrgDomainRequests::Table).to_owned())
.await?;
Ok(())
}
}
fn create_domaim_req_stmt() -> TableCreateStatement {
Table::create()
.table(OrgDomainRequests::Table)
.if_not_exists()
.col(
ColumnDef::new(OrgDomainRequests::Id)
.string_len(27)
.not_null()
.primary_key(),
)
.col(
ColumnDef::new(OrgDomainRequests::OrgId)
.string_len(100)
.not_null(),
)
.col(
ColumnDef::new(OrgDomainRequests::Domain)
.string_len(253)
.not_null(),
)
.col(
ColumnDef::new(OrgDomainRequests::CreatedAt)
.big_integer()
.not_null(),
)
.col(
ColumnDef::new(OrgDomainRequests::UpdatedAt)
.big_integer()
.not_null(),
)
.col(
ColumnDef::new(OrgDomainRequests::RequestedBy)
.string_len(100)
.not_null(),
)
.col(
ColumnDef::new(OrgDomainRequests::AnsweredBy)
.string_len(100)
.null(),
)
.col(ColumnDef::new(OrgDomainRequests::Reason).string().null())
.col(
ColumnDef::new(OrgDomainRequests::Status)
.integer()
.not_null()
.default(0),
)
.to_owned()
}
#[derive(DeriveIden)]
enum OrgDomainRequests {
Table,
Id,
OrgId,
Domain,
CreatedAt,
UpdatedAt,
RequestedBy,
AnsweredBy,
Reason,
Status,
}
src/infra/src/table/migration/mod.rs
+33 -2
View File
@@ -241,8 +241,12 @@ mod m20261025_000003_add_mute_targets_to_downtimes;
mod m20261025_000004_create_slack_alert_messages;
mod m20261025_000005_add_held_action_to_slack_link_requests;
mod m20261026_000001_backfill_v1_rca_pointer;
mod m20261027_000001_alter_query_history;
mod m20261028_000001_add_group_label_set_to_alert_states;
mod m20261029_000001_widen_alert_dedup_fingerprint;
mod m20261030_000001_add_suppress_repeats_to_alerts;
mod m20261030_000001_create_org_domain_ownership_table;
mod m20261030_000002_create_org_domain_ownership_req_table;
/// Shared body of the two `folder_id` migrations above; not a migration itself.
mod workflow_folder_id;
@@ -435,10 +439,13 @@ pub(crate) async fn create_rum_pa_tables(db: &sea_orm::DatabaseConnection) -> Re
/// Apply only the Slack app tables, idempotently, for tests in any crate.
#[cfg(any(test, feature = "test-utils"))]
pub(crate) async fn create_slack_tables<C: sea_orm::ConnectionTrait>(db: &C) -> Result<(), DbErr> {
pub(crate) async fn create_slack_tables(db: &sea_orm::DatabaseConnection) -> Result<(), DbErr> {
use sea_orm_migration::MigrationTrait;
m20261018_000001_create_slack_tables::create_all(db).await?;
m20261025_000004_create_slack_alert_messages::create_all(db).await?;
m20261025_000005_add_held_action_to_slack_link_requests::add_column(db).await
m20261025_000005_add_held_action_to_slack_link_requests::Migration
.up(&SchemaManager::new(db))
.await
}
/// Apply only `alert_acks`, idempotently, for tests in any crate.
@@ -501,6 +508,20 @@ pub(crate) async fn backfill_v1_rca_pointer(db: &sea_orm::DatabaseConnection) ->
.await
}
/// Apply the query_history table and its columns, idempotently, for tests in any crate.
#[cfg(any(test, feature = "test-utils"))]
pub(crate) async fn create_query_history_table(
db: &sea_orm::DatabaseConnection,
) -> Result<(), DbErr> {
let manager = SchemaManager::new(db);
m20261004_000001_create_query_history::Migration
.up(&manager)
.await?;
m20261027_000001_alter_query_history::Migration
.up(&manager)
.await
}
pub struct Migrator;
#[async_trait::async_trait]
@@ -732,6 +753,10 @@ impl MigratorTrait for Migrator {
Box::new(m20261025_000005_add_held_action_to_slack_link_requests::Migration),
Box::new(m20261028_000001_add_group_label_set_to_alert_states::Migration),
Box::new(m20261029_000001_widen_alert_dedup_fingerprint::Migration),
Box::new(m20261027_000001_alter_query_history::Migration),
Box::new(m20261030_000001_add_suppress_repeats_to_alerts::Migration),
Box::new(m20261030_000001_create_org_domain_ownership_table::Migration),
Box::new(m20261030_000002_create_org_domain_ownership_req_table::Migration),
]
}
}
@@ -825,6 +850,12 @@ mod tests {
),
(127, "m20261028_000001_add_group_label_set_to_alert_states"),
(128, "m20261029_000001_widen_alert_dedup_fingerprint"),
(129, "m20261027_000001_alter_query_history"),
(130, "m20261030_000001_add_suppress_repeats_to_alerts"),
(
131,
"m20261030_000002_create_org_domain_ownership_req_table",
),
];
// Two branches that each claim the next version merge cleanly, so a reused number must fail
src/infra/src/table/mod.rs
+2
View File
@@ -82,6 +82,8 @@ pub mod oncall_user_contacts;
pub mod online_eval_jobs;
pub mod org_ai_toolsets;
pub mod org_cleanup_tasks;
pub mod org_domain_ownership;
pub mod org_domain_requests;
pub mod org_ingestion_tokens;
pub mod org_storage_providers;
pub mod org_telephony;
src/infra/src/table/org_domain_ownership.rs
+150
View File
@@ -0,0 +1,150 @@
// Copyright 2026 OpenObserve Inc.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use std::collections::HashMap;
use config::utils::{rand::generate_random_string, time::now_micros};
use sea_orm::{
ActiveModelTrait, ActiveValue::Set, ColumnTrait, EntityTrait, IntoActiveModel, QueryFilter,
QuerySelect, sea_query::Expr,
};
use super::entity::{org_domain_ownership::*, prelude::OrgDomainOwnership};
use crate::db::{get_orm_client_ro, get_orm_client_rw};
pub enum OwnershipState {
Pending = 0,
Verified = 1,
Failed = 2,
}
pub struct OwnershipRecord {
pub org_id: String,
pub domain: String,
pub state: OwnershipState,
}
pub async fn list_active_domain_org_map() -> Result<HashMap<String, String>, anyhow::Error> {
let client = get_orm_client_ro().await;
let records = OrgDomainOwnership::find()
.select_only()
.column(Column::Domain)
.column(Column::OrgId)
.filter(Column::VerificationState.eq(OwnershipState::Verified as i32))
.into_tuple::<(String, String)>()
.all(client)
.await?;
Ok(records.into_iter().collect())
}
pub async fn get_verified_org_for_domain(domain: &str) -> Result<Option<String>, anyhow::Error> {
let client = get_orm_client_ro().await;
let record = OrgDomainOwnership::find()
.select_only()
.column(Column::OrgId)
.filter(Column::Domain.eq(domain))
.filter(Column::VerificationState.eq(OwnershipState::Verified as i32))
.into_tuple::<(String,)>()
.one(client)
.await?;
Ok(record.map(|v| v.0))
}
pub async fn get_domains_for_org(org_id: &str) -> Result<Vec<Model>, anyhow::Error> {
let client = get_orm_client_ro().await;
let records = OrgDomainOwnership::find()
.filter(Column::OrgId.eq(org_id))
.all(client)
.await?;
Ok(records)
}
pub async fn save_org_domain_mapping(
record: OwnershipRecord,
meta_mapped: bool,
) -> Result<(), anyhow::Error> {
let client = get_orm_client_rw().await;
let now = now_micros();
let token = format!("o2v_{}_{}", record.org_id, generate_random_string(32));
let m = ActiveModel {
id: Set(config::ider::generate()),
org_id: Set(record.org_id),
domain: Set(record.domain),
verification_token: Set(token),
verification_state: Set(record.state as i32),
verification_failure_reason: Set(None),
verified_at: Set(None),
released_at: Set(None),
last_checked_at: Set(None),
created_at: Set(now),
updated_at: Set(now),
meta_mapped: Set(meta_mapped),
};
OrgDomainOwnership::insert(m).exec(client).await?;
Ok(())
}
pub async fn delete_linked_domain(org_id: &str, domain: &str) -> Result<(), anyhow::Error> {
let client = get_orm_client_rw().await;
OrgDomainOwnership::delete_many()
.filter(Column::OrgId.eq(org_id))
.filter(Column::Domain.eq(domain))
.exec(client)
.await?;
Ok(())
}
pub async fn get_domain_org_record(
org_id: &str,
domain: &str,
) -> Result<Option<Model>, anyhow::Error> {
let client = get_orm_client_ro().await;
let res = OrgDomainOwnership::find()
.filter(Column::OrgId.eq(org_id))
.filter(Column::Domain.eq(domain))
.one(client)
.await?;
Ok(res)
}
pub async fn save_domain_org_record(model: Model) -> Result<(), anyhow::Error> {
let client = get_orm_client_rw().await;
let am = model.clone().into_active_model().reset_all();
am.update(client).await?;
Ok(())
}
pub async fn list_all() -> Result<Vec<Model>, anyhow::Error> {
let client = get_orm_client_ro().await;
let res = OrgDomainOwnership::find().all(client).await?;
Ok(res)
}
pub async fn mark_meta_mapped(org_id: &str, domain: &str) -> Result<(), anyhow::Error> {
let now = now_micros();
let client = get_orm_client_rw().await;
OrgDomainOwnership::update_many()
.filter(Column::OrgId.eq(org_id))
.filter(Column::Domain.eq(domain))
.col_expr(Column::MetaMapped, Expr::value(true))
.col_expr(Column::LastCheckedAt, Expr::value(now))
.col_expr(
Column::VerificationState,
Expr::value(OwnershipState::Verified as i32),
)
.exec(client)
.await?;
Ok(())
}
src/infra/src/table/org_domain_requests.rs
+107
View File
@@ -0,0 +1,107 @@
// Copyright 2026 OpenObserve Inc.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use config::utils::time::now_micros;
use sea_orm::{ColumnTrait, EntityTrait, IntoActiveModel, QueryFilter, sea_query::Expr};
use super::entity::{org_domain_requests::*, prelude::OrgDomainRequests};
use crate::db::{get_orm_client_ro, get_orm_client_rw};
#[derive(Debug)]
pub enum Status {
Pending = 0,
Approved = 1,
Rejected = 2,
}
impl From<i32> for Status {
fn from(value: i32) -> Self {
match value {
0 => Self::Pending,
1 => Self::Approved,
2 => Self::Rejected,
_ => Self::Pending,
}
}
}
pub async fn save_org_domain_request(req: Model) -> Result<(), anyhow::Error> {
let client = get_orm_client_rw().await;
let m = req.into_active_model();
OrgDomainRequests::insert(m).exec(client).await?;
Ok(())
}
pub async fn list_all() -> Result<Vec<Model>, anyhow::Error> {
let client = get_orm_client_ro().await;
let res = OrgDomainRequests::find().all(client).await?;
Ok(res)
}
pub async fn list_by_org(org_id: &str) -> Result<Vec<Model>, anyhow::Error> {
let client = get_orm_client_ro().await;
let res = OrgDomainRequests::find()
.filter(Column::OrgId.eq(org_id))
.all(client)
.await?;
Ok(res)
}
pub async fn update_status(
id: &str,
user: &str,
new_status: Status,
reason: String,
) -> Result<(), anyhow::Error> {
let now = now_micros();
let client = get_orm_client_rw().await;
OrgDomainRequests::update_many()
.filter(Column::Id.eq(id))
.col_expr(Column::Status, Expr::value(new_status as i32))
.col_expr(Column::AnsweredBy, Expr::value(user))
.col_expr(Column::Reason, Expr::value(reason))
.col_expr(Column::UpdatedAt, Expr::value(now))
.exec(client)
.await?;
Ok(())
}
pub async fn get_by_id(id: &str) -> Result<Option<Model>, anyhow::Error> {
let client = get_orm_client_ro().await;
let res = OrgDomainRequests::find()
.filter(Column::Id.eq(id))
.one(client)
.await?;
Ok(res)
}
pub async fn get_by_org_domain(org_id: &str, domain: &str) -> Result<Option<Model>, anyhow::Error> {
let client = get_orm_client_ro().await;
let res = OrgDomainRequests::find()
.filter(Column::OrgId.eq(org_id))
.filter(Column::Domain.eq(domain))
.filter(Column::Status.ne(Status::Approved as i32))
.one(client)
.await?;
Ok(res)
}
pub async fn delete_by_org(org_id: &str) -> Result<(), anyhow::Error> {
let client = get_orm_client_rw().await;
OrgDomainRequests::delete_many()
.filter(Column::OrgId.eq(org_id))
.exec(client)
.await?;
Ok(())
}
src/infra/src/table/query_history.rs
+1046 -310
View File
File diff suppressed because it is too large Load Diff
src/infra/src/table/slack.rs
+2 -2
View File
@@ -13,9 +13,9 @@
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
use sea_orm::{ConnectionTrait, DbErr};
use sea_orm::{DatabaseConnection, DbErr};
/// Creates the Slack tables, `slack_alert_messages` included, without replaying the migrator.
pub async fn create_tables_for_test<C: ConnectionTrait>(conn: &C) -> Result<(), DbErr> {
pub async fn create_tables_for_test(conn: &DatabaseConnection) -> Result<(), DbErr> {
super::migration::create_slack_tables(conn).await
}
src/infra/src/table/status_pages.rs
+56 -4
View File
@@ -31,21 +31,26 @@ use super::entity::{
status_page_audit_log, status_page_check_snoozes, status_page_component_checks,
status_page_components, status_page_custom_domains, status_page_notice_components,
status_page_notice_updates, status_page_notices, status_page_snapshots, status_pages,
synthetics_checks,
synthetics_checks, synthetics_runs,
};
use crate::{
db::{get_orm_client_ro, get_orm_client_rw},
errors,
};
/// The three columns the auto-incident engine needs per mapped check —
/// The columns the auto-incident engine needs per mapped check —
/// deliberately not the whole row (config/secrets are KB each).
#[derive(Clone, Debug)]
pub struct CheckStateRow {
pub id: String,
pub org_id: String,
pub last_check_status: i32,
pub consecutive_failures: i32,
pub last_triggered_at: i64,
/// `completed_at` of the check's latest completed run, filled by
/// [`latest_run_completions`]; `None` until read or when no run completed
/// inside the lookback window.
pub last_completed_at: Option<i64>,
}
pub async fn get_page_by_slug<C: ConnectionTrait>(
@@ -117,6 +122,7 @@ pub async fn get_check_states<C: ConnectionTrait>(
let mut q = synthetics_checks::Entity::find()
.select_only()
.column(synthetics_checks::Column::Id)
.column(synthetics_checks::Column::OrgId)
.column(synthetics_checks::Column::LastCheckStatus)
.column(synthetics_checks::Column::ConsecutiveFailures)
.column(synthetics_checks::Column::LastTriggeredAt)
@@ -126,22 +132,31 @@ pub async fn get_check_states<C: ConnectionTrait>(
q = q.filter(synthetics_checks::Column::UpdatedAt.gt(watermark));
}
let rows = q
.into_tuple::<(String, i32, i32, i64, i64)>()
.into_tuple::<(String, String, i32, i32, i64, i64)>()
.all(conn)
.await?;
let mut max_updated = since.unwrap_or(0);
let out = rows
.into_iter()
.map(
|(id, last_check_status, consecutive_failures, last_triggered_at, updated_at)| {
|(
id,
org_id,
last_check_status,
consecutive_failures,
last_triggered_at,
updated_at,
)| {
if updated_at > max_updated {
max_updated = updated_at;
}
CheckStateRow {
id,
org_id,
last_check_status,
consecutive_failures,
last_triggered_at,
last_completed_at: None,
}
},
)
@@ -149,6 +164,43 @@ pub async fn get_check_states<C: ConnectionTrait>(
Ok((out, max_updated))
}
/// Latest `synthetics_runs.completed_at` per check, for runs scheduled at or
/// after `scheduled_since`. Keys are check ids; checks with no completed run in
/// the window are absent.
///
/// Region-local like the runs table itself: it sees the runs this region's
/// scheduler enqueued, which is the region that holds the job-cluster claim and
/// therefore runs the incident engine. The `org_id` + `synthetics_id` +
/// `scheduled_ts` predicates match `synthetics_runs_monitor_idx`, so the read
/// is an index range per mapped check, not a table scan.
pub async fn latest_run_completions<C: ConnectionTrait>(
conn: &C,
checks: &[(String, String)],
scheduled_since: i64,
) -> Result<std::collections::HashMap<String, i64>, errors::Error> {
if checks.is_empty() {
return Ok(Default::default());
}
let orgs: HashSet<&str> = checks.iter().map(|(o, _)| o.as_str()).collect();
let ids: HashSet<&str> = checks.iter().map(|(_, id)| id.as_str()).collect();
let rows = synthetics_runs::Entity::find()
.select_only()
.column(synthetics_runs::Column::SyntheticsId)
.column_as(synthetics_runs::Column::CompletedAt.max(), "completed_at")
.filter(synthetics_runs::Column::OrgId.is_in(orgs.into_iter().map(str::to_owned)))
.filter(synthetics_runs::Column::SyntheticsId.is_in(ids.into_iter().map(str::to_owned)))
.filter(synthetics_runs::Column::ScheduledTs.gte(scheduled_since))
.filter(synthetics_runs::Column::CompletedAt.is_not_null())
.group_by(synthetics_runs::Column::SyntheticsId)
.into_tuple::<(String, Option<i64>)>()
.all(conn)
.await?;
Ok(rows
.into_iter()
.filter_map(|(id, at)| at.map(|at| (id, at)))
.collect())
}
/// Notices the engine and snapshots need: not deleted, and open or resolved
/// after `resolved_cutoff` (90d for history; the merge window rides within).
pub async fn list_notices_since<C: ConnectionTrait>(
src/infra/src/table/synthetics_checks.rs
+117
View File
@@ -269,6 +269,17 @@ pub async fn count<C: ConnectionTrait>(
org_id: &str,
params: &ListSyntheticsParams,
) -> Result<u64, errors::Error> {
if has_json_filters(params) {
// location/tag live in JSON text columns: filter in Rust so count and
// page agree.
let models = Entity::find()
.filter(Column::OrgId.eq(org_id))
.apply_filters(params)
.order_by_asc(Column::Name)
.all(conn)
.await?;
return Ok(filter_and_paginate(models, params).1);
}
let q = Entity::find()
.filter(Column::OrgId.eq(org_id))
.apply_filters(params);
@@ -1040,6 +1051,50 @@ async fn get_model<C: ConnectionTrait>(
.await
}
fn has_json_filters(params: &ListSyntheticsParams) -> bool {
params.location.is_some() || params.tag.is_some()
}
fn json_array_contains(v: &serde_json::Value, needle: &str) -> bool {
v.as_array()
.is_some_and(|a| a.iter().any(|x| x.as_str() == Some(needle)))
}
/// Applies the `location` / `tag` exact-match filters (JSON text columns, so
/// no DB operators) and then page/page_size over the filtered set. Returns the
/// requested page and the filtered total; `models` must already be ordered.
fn filter_and_paginate(
models: Vec<synthetics_checks::Model>,
params: &ListSyntheticsParams,
) -> (Vec<synthetics_checks::Model>, u64) {
let filtered: Vec<_> = models
.into_iter()
.filter(|m| {
params
.location
.as_deref()
.is_none_or(|l| json_array_contains(&m.locations, l))
&& params
.tag
.as_deref()
.is_none_or(|t| json_array_contains(&m.tags, t))
})
.collect();
let total = filtered.len() as u64;
if let (Some(page_size), Some(page)) = (params.page_size, params.page)
&& page_size > 0
{
let start = page.saturating_mul(page_size) as usize;
let page_rows = filtered
.into_iter()
.skip(start)
.take(page_size as usize)
.collect();
return (page_rows, total);
}
(filtered, total)
}
async fn list_models<C: ConnectionTrait>(
conn: &C,
org_id: &str,
@@ -1050,6 +1105,11 @@ async fn list_models<C: ConnectionTrait>(
.apply_filters(params)
.order_by_asc(Column::Name);
if has_json_filters(params) {
let models = q.all(conn).await?;
return Ok(filter_and_paginate(models, params).0);
}
if let (Some(page_size), Some(page)) = (params.page_size, params.page)
&& page_size > 0
{
@@ -1227,6 +1287,63 @@ pub(crate) mod tests {
}
}
#[test]
fn filter_and_paginate_combined_filters_and_pages() {
let mk = |id: &str, locs: &[&str], tags: &[&str]| {
let mut m = make_model();
m.id = id.to_string();
m.locations = serde_json::json!(locs);
m.tags = serde_json::json!(tags);
m
};
let models = vec![
mk("a", &["l1"], &["prod"]),
mk("b", &["l1", "l2"], &["prod", "x"]),
mk("c", &["l2"], &["prod"]),
mk("d", &["l1"], &["dev"]),
mk("e", &["l1"], &["prod"]),
];
let params = |loc: Option<&str>, tag: Option<&str>, page, size| ListSyntheticsParams {
location: loc.map(String::from),
tag: tag.map(String::from),
page,
page_size: size,
..Default::default()
};
// location only
let (rows, total) =
filter_and_paginate(models.clone(), &params(Some("l2"), None, None, None));
assert_eq!((rows.len(), total), (2, 2));
// exact match, not substring
let (_, total) = filter_and_paginate(models.clone(), &params(Some("l"), None, None, None));
assert_eq!(total, 0);
// combined
let (rows, total) = filter_and_paginate(
models.clone(),
&params(Some("l1"), Some("prod"), None, None),
);
assert_eq!(total, 3);
assert_eq!(
rows.iter().map(|m| m.id.as_str()).collect::<Vec<_>>(),
["a", "b", "e"]
);
// page boundaries: total stays 3, pages 2 + 1 + empty
let (rows, total) = filter_and_paginate(
models.clone(),
&params(Some("l1"), Some("prod"), Some(0), Some(2)),
);
assert_eq!((rows.len(), total), (2, 3));
let (rows, total) = filter_and_paginate(
models.clone(),
&params(Some("l1"), Some("prod"), Some(1), Some(2)),
);
assert_eq!((rows.len(), total), (1, 3));
assert_eq!(rows[0].id, "e");
let (rows, total) =
filter_and_paginate(models, &params(Some("l1"), Some("prod"), Some(2), Some(2)));
assert_eq!((rows.len(), total), (0, 3));
}
#[test]
fn test_try_from_model() {
let check = Synthetic::try_from(make_model()).unwrap();
src/jobs/Cargo.toml
+2 -1
View File
@@ -40,7 +40,8 @@ vectorscan = [
"search_service/vectorscan",
"stream/vectorscan",
]
profiling = ["openobserve-core/profiling"]
cpu-profiling = ["openobserve-core/cpu-profiling"]
profiling = ["cpu-profiling", "openobserve-core/profiling"]
[dependencies]
audit = { workspace = true, optional = true }
src/jobs/src/job/cloud.rs
+53
View File
@@ -34,6 +34,7 @@ use db::{
use hashbrown::HashSet;
use infra::table::org_users::get_admin;
use o2_enterprise::enterprise::cloud::billings::{MeteringDetail, MeteringProvider};
use openobserve_core::org_domain_ownership;
use stream::get_streams;
use crate::{
@@ -52,6 +53,8 @@ const TRIAL_QUOTA_FLUSH_INTERVAL: u64 = 10;
/// Interval for external contract expiry checks (1 hour).
const EXTERNAL_CONTRACT_CHECK_INTERVAL: u64 = 3600;
const ORG_DOMAIN_OWNERSHIP_CHECK_INTERVAL: u64 = 300;
/// (days-remaining threshold, stage stored after the warning fires).
/// Walked in descending-urgency order so we send at most one warning per tick.
const EXPIRY_WARNING_STAGES: &[(
@@ -99,6 +102,7 @@ pub fn start() {
tokio::spawn(async move { run_org_expiry_daily().await });
tokio::spawn(async move { run_ai_quota_check().await });
tokio::spawn(async move { run_external_contract_expiry_check().await });
tokio::spawn(async move { run_org_domain_ownership_validation().await });
tokio::spawn(async move { run_budget_warning_emails().await });
}
@@ -111,6 +115,55 @@ pub fn start_trial_quota_jobs() {
});
}
async fn run_org_domain_ownership_validation() {
let mut interval = tokio::time::interval(tokio::time::Duration::from_secs(
ORG_DOMAIN_OWNERSHIP_CHECK_INTERVAL,
));
loop {
interval.tick().await;
let records = match infra::table::org_domain_ownership::list_all().await {
Ok(v) => v,
Err(e) => {
log::error!(
"error listing org domain ownership records for verification, skipping : {e}"
);
continue;
}
};
for record in records {
if record.meta_mapped {
continue;
}
let original_state = record.verification_state;
let org_id = record.org_id.clone();
let domain = record.domain.clone();
let updated_record = match org_domain_ownership::verify(record).await {
Ok(v) => v,
Err(e) => {
log::error!(
"error verifying org domain ownership record for {org_id} domain {domain} : {e}"
);
continue;
}
};
let new_state = updated_record.verification_state;
if new_state != original_state {
log::info!(
"org domain ownership state for {org_id} domain {domain} changed from {original_state} to {new_state}"
);
if let Err(e) =
org_domain_ownership::update_state(&org_id, &domain, new_state).await
{
log::error!(
"error updating in memory org domain ownership state for org {org_id} domain {domain} : {e}"
);
continue;
}
}
}
}
}
async fn run_no_ingestion_period() {
let mut interval = tokio::time::interval(tokio::time::Duration::from_secs(
NO_INGESTION_REPORT_INTERVAL,
src/jobs/src/job/mod.rs
+6 -2
View File
@@ -596,7 +596,11 @@ pub async fn init() -> Result<(), anyhow::Error> {
.await?;
#[cfg(feature = "cloud")]
tokio::task::spawn(o2_enterprise::enterprise::cloud::billings::watch());
{
openobserve_core::org_domain_ownership::init().await;
tokio::task::spawn(o2_enterprise::enterprise::cloud::billings::watch());
tokio::task::spawn(openobserve_core::org_domain_ownership::watch());
}
// check version
db::metas::version::set()
@@ -690,7 +694,7 @@ pub async fn init() -> Result<(), anyhow::Error> {
return Ok(());
}
#[cfg(feature = "profiling")]
#[cfg(feature = "cpu-profiling")]
tokio::task::spawn(openobserve_core::self_profiles::run());
// telemetry run
src/jobs/src/job/slack_hooks.rs
+1 -1
View File
@@ -71,7 +71,7 @@ async fn mapped_domains(org_id: &str) -> anyhow::Result<Vec<String>> {
if db::org_status::is_blocked(org_id) {
return Ok(vec![]);
}
let settings = db::organization::get_org_setting(config::META_ORG_ID).await?;
let settings = db::organization::get_org_setting(org_id).await?;
let mappings = settings
.domain_org_mappings
.iter()
src/synthetics/src/callback_endpoint.rs
+260
View File
@@ -0,0 +1,260 @@
// Copyright 2026 OpenObserve Inc.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU Affero General Public License for more details.
//
// You should have received a copy of the GNU Affero General Public License
// along with this program. If not, see <http://www.gnu.org/licenses/>.
//! The callback endpoint probes are told to call back on
//! (`config::meta::synthetics::api_endpoint`), as it can safely appear in a
//! log line or a dead-letter message, plus a startup reachability warning for
//! Lambda venues (o2-enterprise#3165).
use std::{
net::{IpAddr, Ipv4Addr, Ipv6Addr},
time::Duration,
};
/// Placeholder for an endpoint that does not parse, so no raw value (which may
/// carry credentials) is ever echoed.
pub const UNPARSEABLE_ENDPOINT: &str = "<unparseable callback endpoint>";
/// `scheme://host[:port]/path` of `raw`: no userinfo, no query, no fragment.
/// `None` when `raw` is not an absolute URL with a host.
pub fn sanitize_callback_url(raw: &str) -> Option<String> {
let url = reqwest::Url::parse(raw.trim()).ok()?;
let host = url.host_str()?;
let port = url.port().map(|p| format!(":{p}")).unwrap_or_default();
let path = url.path().trim_end_matches('/');
Some(format!("{}://{host}{port}{path}", url.scheme()))
}
/// The endpoint probes are currently told to call back on, sanitized.
pub fn current_callback_endpoint() -> String {
sanitize_callback_url(&config::meta::synthetics::api_endpoint())
.unwrap_or_else(|| UNPARSEABLE_ENDPOINT.to_string())
}
/// Address classes a Lambda function outside the deployment's network usually
/// cannot reach.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AddrClass {
Public,
Loopback,
Unspecified,
LinkLocal,
Private,
}
pub fn classify_ip(ip: IpAddr) -> AddrClass {
match ip {
IpAddr::V4(v4) => classify_v4(v4),
IpAddr::V6(v6) => {
if let Some(v4) = v6.to_ipv4_mapped() {
return classify_v4(v4);
}
classify_v6(v6)
}
}
}
fn classify_v4(ip: Ipv4Addr) -> AddrClass {
let o = ip.octets();
if ip.is_loopback() {
AddrClass::Loopback
} else if ip.is_unspecified() {
AddrClass::Unspecified
} else if ip.is_link_local() {
AddrClass::LinkLocal
} else if ip.is_private() || (o[0] == 100 && (o[1] & 0xc0) == 64) {
// RFC 1918 plus shared address space (100.64.0.0/10, carrier NAT).
AddrClass::Private
} else {
AddrClass::Public
}
}
fn classify_v6(ip: Ipv6Addr) -> AddrClass {
let first = ip.segments()[0];
if ip.is_loopback() {
AddrClass::Loopback
} else if ip.is_unspecified() {
AddrClass::Unspecified
} else if (first & 0xffc0) == 0xfe80 {
AddrClass::LinkLocal
} else if (first & 0xfe00) == 0xfc00 {
// Unique local addresses, fc00::/7.
AddrClass::Private
} else {
AddrClass::Public
}
}
/// The host part of a parsed URL as an IP literal, IPv6 brackets stripped.
fn host_ip(host: &str) -> Option<IpAddr> {
host.trim_start_matches('[')
.trim_end_matches(']')
.parse()
.ok()
}
const DNS_TIMEOUT: Duration = Duration::from_secs(2);
/// Log-only check run when a Lambda venue is enabled: a Lambda function can
/// call back only on an endpoint reachable from its own network. Never rejects
/// anything, because a VPC-connected Lambda can legitimately reach private
/// addresses. Logs only the sanitized endpoint.
pub async fn warn_if_lambda_callback_may_be_unreachable() {
let raw = config::meta::synthetics::api_endpoint();
let Some(shown) = sanitize_callback_url(&raw) else {
tracing::warn!(
"[synthetics] Lambda venue enabled but the callback endpoint \
(ZO_SYNTHETICS_API_ENDPOINT, else ZO_WEB_URL) is empty or not an absolute URL; \
Lambda probes may be unable to report results"
);
return;
};
let Ok(url) = reqwest::Url::parse(raw.trim()) else {
return;
};
let Some(host) = url.host_str() else { return };
let advice = "a Lambda probe can report only if this endpoint is reachable from the \
Lambda function's network (a VPC-connected function can reach private \
addresses; otherwise use a public endpoint)";
if let Some(ip) = host_ip(host) {
let class = classify_ip(ip);
if class != AddrClass::Public {
tracing::warn!(
endpoint = %shown,
address_class = ?class,
"[synthetics] callback endpoint is a {class:?} address; {advice}"
);
}
return;
}
if host.eq_ignore_ascii_case("localhost") || host.to_ascii_lowercase().ends_with(".localhost") {
tracing::warn!(
endpoint = %shown,
"[synthetics] callback endpoint is localhost; {advice}"
);
return;
}
let port = url.port_or_known_default().unwrap_or(443);
let lookup = tokio::time::timeout(
DNS_TIMEOUT,
tokio::net::lookup_host((host.to_string(), port)),
)
.await;
let addrs: Vec<IpAddr> = match lookup {
Ok(Ok(it)) => it.map(|sa| sa.ip()).collect(),
Ok(Err(e)) => {
tracing::warn!(
endpoint = %shown,
"[synthetics] could not resolve the callback endpoint host from this node ({e}); \
Lambda probes may be unable to report results if it does not resolve for them \
either"
);
return;
}
Err(_) => {
tracing::warn!(
endpoint = %shown,
"[synthetics] resolving the callback endpoint host timed out after {}s; \
Lambda probes may be unable to report results if it does not resolve for them \
either",
DNS_TIMEOUT.as_secs()
);
return;
}
};
let non_public = addrs
.iter()
.filter(|ip| classify_ip(**ip) != AddrClass::Public)
.count();
if non_public == 0 {
return;
}
let kind = if non_public == addrs.len() {
"only private, loopback or link-local addresses"
} else {
"a mix of public and private addresses"
};
tracing::warn!(
endpoint = %shown,
"[synthetics] callback endpoint host resolves to {kind}; {advice}"
);
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn sanitize_strips_userinfo_query_and_fragment() {
assert_eq!(
sanitize_callback_url("https://user:secret@o2.example.com:8443/api/?token=abc#x"),
Some("https://o2.example.com:8443/api".to_string())
);
assert_eq!(
sanitize_callback_url("http://o2.example.com"),
Some("http://o2.example.com".to_string())
);
}
#[test]
fn sanitize_keeps_ipv6_brackets() {
assert_eq!(
sanitize_callback_url("http://[fd00::1]:5080/base?sig=1"),
Some("http://[fd00::1]:5080/base".to_string())
);
}
#[test]
fn sanitize_rejects_bad_urls() {
assert_eq!(sanitize_callback_url(""), None);
assert_eq!(sanitize_callback_url("not a url"), None);
assert_eq!(sanitize_callback_url("o2.example.com/api"), None);
assert_eq!(sanitize_callback_url("mailto:a@b.c"), None);
}
#[test]
fn classify_ipv4() {
let c = |s: &str| classify_ip(s.parse().unwrap());
assert_eq!(c("127.0.0.1"), AddrClass::Loopback);
assert_eq!(c("0.0.0.0"), AddrClass::Unspecified);
assert_eq!(c("169.254.169.254"), AddrClass::LinkLocal);
assert_eq!(c("10.1.2.3"), AddrClass::Private);
assert_eq!(c("172.16.0.1"), AddrClass::Private);
assert_eq!(c("192.168.1.1"), AddrClass::Private);
assert_eq!(c("100.64.0.1"), AddrClass::Private);
assert_eq!(c("8.8.8.8"), AddrClass::Public);
assert_eq!(c("172.32.0.1"), AddrClass::Public);
}
#[test]
fn classify_ipv6() {
let c = |s: &str| classify_ip(s.parse().unwrap());
assert_eq!(c("::1"), AddrClass::Loopback);
assert_eq!(c("::"), AddrClass::Unspecified);
assert_eq!(c("fe80::1"), AddrClass::LinkLocal);
assert_eq!(c("fd12:3456::1"), AddrClass::Private);
assert_eq!(c("::ffff:10.0.0.1"), AddrClass::Private);
assert_eq!(c("2606:4700::1111"), AddrClass::Public);
}
#[test]
fn host_ip_parses_bracketed_ipv6() {
assert_eq!(host_ip("[::1]"), Some("::1".parse().unwrap()));
assert_eq!(host_ip("o2.example.com"), None);
}
}
src/synthetics/src/dispatcher.rs
+113 -24
View File
@@ -35,7 +35,22 @@ use infra::{
};
const TICK: Duration = Duration::from_secs(2);
const LEASE_BATCH_SIZE: i64 = 10;
/// Jobs per pool per lease call (`ZO_SYNTHETICS_LEASE_BATCH_SIZE`), read per tick.
fn lease_batch_size() -> i64 {
config::get_config().synthetics.lease_batch_size.max(1)
}
/// Most jobs leased across all pools in one tick (`ZO_SYNTHETICS_MAX_INFLIGHT_PER_TICK`).
fn max_inflight_per_tick() -> i64 {
config::get_config().synthetics.max_inflight_per_tick.max(1)
}
/// Size of the next lease call: a full batch, trimmed so the tick never leases
/// past its cap. Zero means the tick is out of budget.
fn next_batch_limit(batch: i64, leased_this_tick: i64, cap: i64) -> i64 {
batch.min(cap - leased_this_tick).max(0)
}
/// How long a leased job may run before the reaper requeues it
/// (`reaper::requeue_expired`).
///
@@ -95,39 +110,88 @@ static INSTANCE_ID: LazyLock<String> = LazyLock::new(|| {
)
});
pub async fn run() {
static POOL_ROTATION: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0);
pub async fn run(stop: crate::StopToken) {
let instance_id = INSTANCE_ID.as_str();
tracing::info!(instance_id = %instance_id, "[synthetics dispatcher] started");
// o2-enterprise#3165: a Lambda probe can only report to a callback endpoint
// its own network reaches. Log-only; never blocks the dispatcher.
{
let cfg = config::get_config();
if !cfg.synthetics.lambda_browser.is_empty() || !cfg.synthetics.lambda_net.is_empty() {
crate::callback_endpoint::warn_if_lambda_callback_may_be_unreachable().await;
}
}
loop {
tokio::time::sleep(TICK).await;
if stop.sleep(TICK).await {
tracing::info!("[synthetics dispatcher] stopped");
return;
}
let db = get_orm_client_rw().await;
let now_us = config::utils::time::now_micros();
for pool in dispatcher_pools().await {
let rows = match synthetics_jobs::lease_batch(
db,
&pool,
instance_id,
LEASE_BATCH_SIZE,
now_us,
lease_secs(),
None, // public pools are already type-segmented (net-* vs aws-browser)
)
.await
{
Ok(r) => r,
Err(e) => {
tracing::error!(pool = %pool, "[synthetics dispatcher] lease_batch: {e}");
continue;
// Round-robin across pools, one batch each per round. A pool whose batch
// came back full may have more due, so it stays in for another round;
// the per-tick cap bounds the total, and no pool gets a second batch
// before every other active pool has had its turn.
let batch = lease_batch_size();
let cap = max_inflight_per_tick();
let mut leased_this_tick: i64 = 0;
let mut active = dispatcher_pools().await;
// Rotate the starting pool every tick: with the cap reached, a fixed
// order would let the first pools take the whole budget every tick.
if !active.is_empty() {
let start =
POOL_ROTATION.fetch_add(1, std::sync::atomic::Ordering::Relaxed) % active.len();
active.rotate_left(start);
}
while !active.is_empty() {
let mut still_active = Vec::with_capacity(active.len());
for pool in active {
// Ownership lost mid-tick: lease nothing more.
if stop.is_cancelled() {
return;
}
};
let limit = next_batch_limit(batch, leased_this_tick, cap);
if limit == 0 {
break;
}
let rows = match synthetics_jobs::lease_batch(
db,
&pool,
instance_id,
limit,
now_us,
lease_secs(),
None, // public pools are already type-segmented (net-* vs aws-browser)
)
.await
{
Ok(r) => r,
Err(e) => {
tracing::error!(pool = %pool, "[synthetics dispatcher] lease_batch: {e}");
continue;
}
};
for row in rows {
tokio::spawn(invoke_probe(row));
let full = rows.len() as i64 >= limit;
leased_this_tick += rows.len() as i64;
for row in rows {
tokio::spawn(invoke_probe(row, stop.clone()));
}
if full {
still_active.push(pool);
}
}
if leased_this_tick >= cap {
break;
}
active = still_active;
}
}
}
@@ -304,7 +368,13 @@ async fn mark_failure(row: &synthetics_jobs::LeasedRow, reason: Option<&str>) ->
// nothing, while the first failure after a pass sends one message. Without
// it the other regions' LIST would keep showing the last status they
// themselves ran, or "Unknown".
match synthetics_checks::update_last_check_status(db, &row.synthetics_id, 4).await {
match synthetics_checks::update_last_check_status(
db,
&row.synthetics_id,
config::meta::status_pages::CHECK_STATUS_ERROR,
)
.await
{
Ok(true) => {
#[cfg(feature = "enterprise")]
if o2_enterprise::enterprise::common::config::get_config()
@@ -449,7 +519,13 @@ async fn report_to_stream(
}
/// Invokes the browser probe Lambda for a leased check row.
async fn invoke_probe(row: synthetics_jobs::LeasedRow) {
async fn invoke_probe(row: synthetics_jobs::LeasedRow, stop: crate::StopToken) {
// Leased just before ownership moved: do not start the probe here. The
// lease expires and the new owner's reaper re-queues the row.
if stop.is_cancelled() {
tracing::info!(job_id = %row.id, "[synthetics dispatcher] ownership lost — not invoking");
return;
}
// Derive AWS region from the job's location field.
let region = match aws_region_from_location(&row.location) {
Some(r) => r.to_owned(),
@@ -691,3 +767,16 @@ async fn handle_dispatch_failure(
}
}
}
#[cfg(test)]
mod lease_batch_tests {
use super::next_batch_limit;
#[test]
fn batch_limit_is_trimmed_to_the_tick_cap() {
assert_eq!(next_batch_limit(10, 0, 100), 10);
assert_eq!(next_batch_limit(10, 95, 100), 5);
assert_eq!(next_batch_limit(10, 100, 100), 0);
assert_eq!(next_batch_limit(10, 120, 100), 0);
}
}
src/synthetics/src/lib.rs
+248 -41
View File
@@ -60,6 +60,7 @@ pub(crate) static CONFIG_SWAP_LOCK: tokio::sync::Mutex<()> = tokio::sync::Mutex:
pub const BUILT_WITH_CLOUD: bool = cfg!(feature = "cloud");
pub mod alerting;
pub mod callback_endpoint;
pub mod dispatcher;
pub mod job_api;
pub mod pool;
@@ -157,52 +158,200 @@ pub async fn init(step_pool: Option<pool::StepPoolHooks>) {
// super-cluster KV client is built and no NATS read is issued on this path
// (spec §14 — super cluster off must be byte-identical).
#[cfg(not(feature = "enterprise"))]
spawn_workers();
drop(spawn_workers(StopToken::never()));
#[cfg(feature = "enterprise")]
if !o2_enterprise::enterprise::common::config::get_config()
.super_cluster
.enabled
{
spawn_workers();
drop(spawn_workers(StopToken::never()));
return;
}
#[cfg(feature = "enterprise")]
tokio::spawn(async move {
let mut announced = false;
while !holds_job_cluster_claim().await {
if !announced {
// Once, not per poll: a region that legitimately does not hold
// the claim is the steady state for the whole life of the
// process, and an every-10s line would bury it.
tracing::info!(
"[synthetics] another cluster holds the job-cluster claim (or none is \
registered yet) — workers not started, re-checking every {}s",
JOB_CLUSTER_POLL.as_secs()
);
announced = true;
}
tokio::time::sleep(JOB_CLUSTER_POLL).await;
}
tracing::info!("[synthetics] this cluster holds the job-cluster claim — starting workers");
spawn_workers();
});
tokio::spawn(ownership_supervisor());
}
fn spawn_workers() {
tokio::spawn(scheduler::run());
tokio::spawn(dispatcher::run());
tokio::spawn(reaper::run());
// POC scope: single cluster. Multi-region needs this split — the
// incident-engine half claim-gated, the snapshot half per region.
// Status pages ships with synthetics — no separate toggle.
tokio::spawn(status_pages::run());
tokio::spawn(status_pages::run_domain_verifier());
// Its own task, not a step on the reaper's tick: a pass is up to a thousand
// rows of outbound HTTP, and the reaper's lease bookkeeping cannot wait
// behind it. Its own kill switch too, checked per pass rather than here.
tokio::spawn(reaper::orphan::run());
/// Cooperative stop signal handed to every claim-gated worker.
///
/// A minimal `CancellationToken` on `tokio::sync::watch` (the crate has no
/// `tokio-util` dependency, and adding one is not worth a manifest change).
/// A dropped sender reads as cancelled, so a supervisor that goes away fails
/// closed rather than leaving workers running unsupervised.
#[derive(Clone)]
pub struct StopToken {
rx: tokio::sync::watch::Receiver<bool>,
// Only set by [`StopToken::never`]: keeps the channel open forever.
_keep: Option<std::sync::Arc<tokio::sync::watch::Sender<bool>>>,
}
impl StopToken {
/// A token plus the sender that cancels it.
pub fn new() -> (tokio::sync::watch::Sender<bool>, Self) {
let (tx, rx) = tokio::sync::watch::channel(false);
(tx, Self { rx, _keep: None })
}
/// A token that is never cancelled: the single-cluster path, where there
/// is no ownership to lose.
pub fn never() -> Self {
let (tx, rx) = tokio::sync::watch::channel(false);
Self {
rx,
_keep: Some(std::sync::Arc::new(tx)),
}
}
pub fn is_cancelled(&self) -> bool {
*self.rx.borrow() || (self._keep.is_none() && self.rx.has_changed().is_err())
}
pub async fn cancelled(&self) {
let mut rx = self.rx.clone();
// Err = sender dropped, which also means stop.
let _ = rx.wait_for(|v| *v).await;
}
/// Sleeps for `d`; returns `true` when cancelled first (the caller exits
/// its loop at this tick boundary, never mid-pass).
pub async fn sleep(&self, d: std::time::Duration) -> bool {
tokio::select! {
_ = self.cancelled() => true,
_ = tokio::time::sleep(d) => self.is_cancelled(),
}
}
}
/// Result of one read of the job-cluster claim. A failed read is `Unknown`,
/// never "held": the supervisor fails closed.
#[cfg_attr(not(feature = "enterprise"), allow(dead_code))]
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum ClaimRead {
Held,
NotHeld,
Unknown,
}
#[cfg_attr(not(feature = "enterprise"), allow(dead_code))]
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum SupervisorAction {
Start,
Stop,
Hold,
}
/// The supervisor's whole decision: a pure function of whether a worker
/// generation is running and what the claim read returned. Workers run exactly
/// while the claim is provably held here; `NotHeld` and `Unknown` both stop a
/// running generation (same fail-closed rule as the anomaly supervisor in
/// `jobs::job::mod.rs`).
#[cfg_attr(not(feature = "enterprise"), allow(dead_code))]
fn supervisor_action(running: bool, claim: ClaimRead) -> SupervisorAction {
match (running, claim) {
(false, ClaimRead::Held) => SupervisorAction::Start,
(true, ClaimRead::NotHeld | ClaimRead::Unknown) => SupervisorAction::Stop,
_ => SupervisorAction::Hold,
}
}
/// Start/Stop supervisor for the claim-gated workers (o2-enterprise#3190).
///
/// Replaces the old start-only latch, which spawned the workers once on
/// gaining the claim and never stopped them when the claim moved away, so two
/// regions could schedule, dispatch and reap the same checks. Each generation
/// gets its own [`StopToken`]; Stop cancels it and JOINS every worker before
/// the next Start can happen, so there are never two generations at once.
#[cfg(feature = "enterprise")]
async fn ownership_supervisor() {
let mut generation: u64 = 0;
let mut running: Option<(
tokio::sync::watch::Sender<bool>,
Vec<tokio::task::JoinHandle<()>>,
)> = None;
let mut last_claim: Option<ClaimRead> = None;
loop {
let claim = read_job_cluster_claim().await;
if last_claim != Some(claim) && claim == ClaimRead::NotHeld && running.is_none() {
tracing::info!(
"[synthetics] another cluster holds the job-cluster claim (or none is registered \
yet) — workers not running here, re-checking every {}s",
JOB_CLUSTER_POLL.as_secs()
);
}
last_claim = Some(claim);
match supervisor_action(running.is_some(), claim) {
SupervisorAction::Start => {
generation += 1;
tracing::info!(
generation,
"[synthetics] ownership acquired — starting workers"
);
let (tx, token) = StopToken::new();
running = Some((tx, spawn_workers(token)));
}
SupervisorAction::Stop => {
if claim == ClaimRead::Unknown {
tracing::warn!(
generation,
"[synthetics] ownership unknown (claim read failed) — stopping workers"
);
} else {
tracing::info!(generation, "[synthetics] ownership lost — stopping workers");
}
if let Some((tx, handles)) = running.take() {
let _ = tx.send(true);
for h in handles {
if let Err(e) = h.await {
tracing::error!(generation, "[synthetics] worker join failed: {e}");
}
}
}
tracing::info!(generation, "[synthetics] workers stopped");
}
SupervisorAction::Hold => {}
}
tokio::time::sleep(JOB_CLUSTER_POLL).await;
}
}
#[cfg(feature = "enterprise")]
async fn read_job_cluster_claim() -> ClaimRead {
match holds_job_cluster_claim().await {
Some(true) => ClaimRead::Held,
Some(false) => ClaimRead::NotHeld,
None => ClaimRead::Unknown,
}
}
/// Spawns one generation of workers. Every worker takes `stop` and leaves its
/// loop at the next tick boundary once it is cancelled.
///
/// All of them are claim-gated, as they were under the old latch: the status
/// page task is still one loop (incident engine and snapshot together — the
/// per-region snapshot split is future work), and the domain verifier writes
/// shared verification state, so both stay owner-only.
fn spawn_workers(stop: StopToken) -> Vec<tokio::task::JoinHandle<()>> {
let verifier_stop = stop.clone();
vec![
tokio::spawn(scheduler::run(stop.clone())),
tokio::spawn(dispatcher::run(stop.clone())),
tokio::spawn(reaper::run(stop.clone())),
// Status pages ships with synthetics — no separate toggle.
tokio::spawn(status_pages::run(stop.clone())),
// The verifier lives in the enterprise crate and takes no token, so it
// is dropped at its next await point on cancel.
tokio::spawn(async move {
tokio::select! {
_ = verifier_stop.cancelled() => {}
_ = status_pages::run_domain_verifier() => {}
}
}),
// Its own task, not a step on the reaper's tick: a pass is up to a
// thousand rows of outbound HTTP, and the reaper's lease bookkeeping
// cannot wait behind it. Its own kill switch too, checked per pass.
tokio::spawn(reaper::orphan::run(stop)),
]
}
/// One evaluation of [`should_start`] against the super-cluster KV.
@@ -217,7 +366,7 @@ fn spawn_workers() {
/// increments `SYNTHETICS_ORPHAN_SCANS_TOTAL` anywhere, which is exactly the
/// dead-region signal `reaper::orphan` documents.
#[cfg(feature = "enterprise")]
async fn holds_job_cluster_claim() -> bool {
async fn holds_job_cluster_claim() -> Option<bool> {
use o2_enterprise::enterprise::super_cluster::kv;
let local = config::get_cluster_name();
@@ -225,27 +374,27 @@ async fn holds_job_cluster_claim() -> bool {
Ok(name) => name,
Err(e) => {
tracing::error!("[synthetics] could not read the job cluster, not starting yet: {e}");
return false;
return None;
}
};
// Both settled without the cluster list, which is the more expensive read.
// Must agree with `should_start`, which is where the rule is tested.
if claim.is_empty() {
return false;
return Some(false);
}
if claim == local {
return true;
return Some(true);
}
let live = match kv::cluster::list_by_role_group(None).await {
Ok(clusters) => clusters.into_iter().map(|c| c.name).collect::<Vec<_>>(),
Err(e) => {
tracing::error!("[synthetics] could not list clusters, not starting yet: {e}");
return false;
return None;
}
};
should_start(true, &claim, &local, &live)
Some(should_start(true, &claim, &local, &live))
}
/// Whether this cluster may start the synthetics workers.
@@ -344,4 +493,62 @@ mod job_cluster_gate_tests {
}
#[cfg(test)]
mod tests {}
mod tests {
use super::{ClaimRead::*, StopToken, SupervisorAction::*, supervisor_action};
#[test]
fn starts_only_on_a_held_claim() {
assert_eq!(supervisor_action(false, Held), Start);
assert_eq!(supervisor_action(false, NotHeld), Hold);
assert_eq!(supervisor_action(false, Unknown), Hold);
}
#[test]
fn a_running_generation_stops_on_loss_or_read_error() {
assert_eq!(supervisor_action(true, NotHeld), Stop);
assert_eq!(supervisor_action(true, Unknown), Stop);
assert_eq!(supervisor_action(true, Held), Hold);
}
/// Flapping claim: replay the reads through the decision function and
/// check the generation never overlaps and every flip is acted on.
#[test]
fn flapping_claim_alternates_start_and_stop_without_overlap() {
let reads = [Held, Held, NotHeld, Held, Unknown, Unknown, Held, NotHeld];
let mut running = false;
let mut starts = 0;
let mut stops = 0;
for r in reads {
match supervisor_action(running, r) {
Start => {
assert!(!running, "never two generations");
running = true;
starts += 1;
}
Stop => {
assert!(running);
running = false;
stops += 1;
}
Hold => {}
}
}
assert_eq!((starts, stops), (3, 3));
assert!(!running);
}
#[tokio::test]
async fn stop_token_cancels_sleep_and_dropped_sender_fails_closed() {
let (tx, token) = StopToken::new();
assert!(!token.is_cancelled());
tx.send(true).unwrap();
assert!(token.sleep(std::time::Duration::from_secs(60)).await);
let (tx, token) = StopToken::new();
drop(tx);
assert!(token.sleep(std::time::Duration::from_secs(60)).await);
let never = StopToken::never();
assert!(!never.sleep(std::time::Duration::from_millis(1)).await);
}
}
src/synthetics/src/reaper/mod.rs
+65 -18
View File
@@ -48,11 +48,14 @@ use infra::{
use crate::{MAX_DISPATCH_ATTEMPTS as MAX_ATTEMPTS, alerting::ERROR_SOURCE_DISPATCH};
const TICK: Duration = Duration::from_secs(30);
pub async fn run() {
pub async fn run(stop: crate::StopToken) {
tracing::info!("[synthetics reaper] started");
loop {
tokio::time::sleep(TICK).await;
if stop.sleep(TICK).await {
tracing::info!("[synthetics reaper] stopped");
return;
}
let db = get_orm_client_rw().await;
@@ -181,7 +184,13 @@ async fn handle_dead_letter(
// check that is already in Error dead-letters silently, which is the common
// case when a private agent has gone away and every run of every check it
// served is expiring.
match synthetics_checks::update_last_check_status(db, &row.synthetics_id, 4).await {
match synthetics_checks::update_last_check_status(
db,
&row.synthetics_id,
config::meta::status_pages::CHECK_STATUS_ERROR,
)
.await
{
Ok(true) => {
#[cfg(feature = "enterprise")]
if o2_enterprise::enterprise::common::config::get_config()
@@ -218,21 +227,16 @@ async fn handle_dead_letter(
// points at a dead agent or a location nothing polls; the other two point at
// a probe that took the job and stopped talking. They need different
// responses, so they should not read identically in the record.
let error_msg = match row.reason {
synthetics_jobs::DeadLetterReason::AttemptsExhausted => format!(
"probe did not respond after {} dispatch attempts",
row.dispatch_attempts
),
synthetics_jobs::DeadLetterReason::Expired => format!(
"probe held the check but did not respond before its schedule window closed \
(after {} dispatch attempt(s)); the next scheduled run will retry",
row.dispatch_attempts
),
synthetics_jobs::DeadLetterReason::NeverDispatched => {
"no probe claimed the check for this location before its schedule window closed"
.to_string()
}
};
// Phase, attempt count and the callback endpoint (o2-enterprise#3165): a
// timeout says the probe did not report, not why, so the message names
// where it was told to report and stops short of a cause.
//
// The endpoint is the one resolved NOW, not at dispatch: the job row does
// not store the dispatch-time endpoint. Capturing it needs a column
// (deferred); until then a config change between dispatch and reaping
// would show the new value.
let callback = crate::callback_endpoint::current_callback_endpoint();
let error_msg = dead_letter_message(row.reason, row.dispatch_attempts, &callback);
// Above the token lookup, which returns early: the triggers half needs no token.
usage_reporting::publish_triggers_usage(dead_letter_trigger(row, now_us, &error_msg));
@@ -264,6 +268,32 @@ async fn handle_dead_letter(
write_results_stream(row, &ingest_token, &api_endpoint, now_us, &error_msg).await;
}
/// The dead-letter text for one reason. `callback` must already be sanitized.
fn dead_letter_message(
reason: synthetics_jobs::DeadLetterReason,
dispatch_attempts: i32,
callback: &str,
) -> String {
match reason {
synthetics_jobs::DeadLetterReason::AttemptsExhausted => format!(
"probe did not respond after {dispatch_attempts} dispatch attempts \
(phase: claimed, no result before the lease ran out); the probe may be unable to reach \
the callback endpoint {callback}"
),
synthetics_jobs::DeadLetterReason::Expired => format!(
"probe held the check but did not respond before its schedule window closed \
(phase: claimed, lease expired without a result; after {dispatch_attempts} dispatch attempt(s)); \
the probe may be unable to reach the callback endpoint {callback}; \
the next scheduled run will retry"
),
synthetics_jobs::DeadLetterReason::NeverDispatched => format!(
"no probe claimed the check for this location before its schedule window closed \
(phase: never claimed; {dispatch_attempts} dispatch attempt(s)); a probe may be \
unable to reach the callback endpoint {callback}"
),
}
}
async fn write_results_stream(
row: &synthetics_jobs::DeadLetteredRow,
ingest_token: &str,
@@ -393,4 +423,21 @@ mod tests {
"an alert rule matches the SERIALIZED value, and this row writes `failed` today",
);
}
#[test]
fn dead_letter_messages_name_phase_attempts_and_endpoint() {
use infra::table::synthetics_jobs::DeadLetterReason as R;
let ep = "https://o2.example.com/api";
for (reason, phase) in [
(R::AttemptsExhausted, "phase: claimed, no result"),
(R::Expired, "phase: claimed, lease expired"),
(R::NeverDispatched, "phase: never claimed"),
] {
let msg = super::dead_letter_message(reason, 3, ep);
assert!(msg.contains(phase), "{msg}");
assert!(msg.contains("3 dispatch attempt"), "{msg}");
assert!(msg.contains(ep), "{msg}");
assert!(msg.contains("may be unable to reach"), "{msg}");
}
}
}

Some files were not shown because too many files have changed in this diff Show More