mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-09-07 08:02:40 +00:00
* fix(alerts): identify server replica in low-disk alert + per-host dedup tag The server-mode low-disk alert keyed its dedup tag on the mountpoint alone, so `simple_alert_helper` mapped every server replica onto a single alert row per mountpoint. With more than one replica that row flaps every monitor pass: a replica seeing low disk raises the alert while a replica seeing healthy disk recovers it. The alert text also could not say which replica tripped. The fix lives in windmill-ee-private (`low_disk_alerts` in windmill-common/src/ee.rs) and appends the hostname to both the message and the dedup tag, mirroring the worker branch. Also add a regression test pinning the server tag as per-host, and correct the monitor cadence comments: iterations are LISTEN_NEW_EVENTS_INTERVAL_SEC (10s by default), not 30s, so "~60s (2 iterations * 30s)" was wrong on both factors. * fix(alerts): widen healthchecks.check_type so per-host disk tags fit Alert tags embed a mountpoint and a hostname, both unbounded, but check_type was varchar(50). create_alert only logs the insert error while the notification still fires, so an overflowing tag re-alerts every monitor pass and never records recovery state. The server tag overflows for ordinary pod-length hostnames, and the existing worker tag already overflows for every tracked mount except "/". Widening the column fixes both; bounding the hostname would not, since the mountpoint alone can consume the budget. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore: update ee-repo-ref to b3d01f2c0d2c0714ae95b8a348af22b0fcc30ee4 This commit updates the EE repository reference after PR #666 was merged in windmill-ee-private. Previous ee-repo-ref: ccd1e42cf6b2d051ca17074fbdf5b80a46cffe0f New ee-repo-ref: b3d01f2c0d2c0714ae95b8a348af22b0fcc30ee4 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
58 lines
2.3 KiB
Rust
58 lines
2.3 KiB
Rust
//! Regression test for the server-mode low-disk alert dedup tag.
|
|
//!
|
|
//! ## Requirements
|
|
//!
|
|
//! - PostgreSQL database running locally
|
|
//! - Enterprise features enabled
|
|
//!
|
|
//! ## Running the tests
|
|
//!
|
|
//! ```bash
|
|
//! cargo test -p windmill-common --test low_disk_alerts --features private,enterprise -- --ignored --nocapture
|
|
//! ```
|
|
|
|
#[cfg(all(feature = "private", feature = "enterprise"))]
|
|
mod tests {
|
|
use sqlx::{Pool, Postgres};
|
|
use windmill_common::ee::low_disk_alerts;
|
|
use windmill_common::utils::HOSTNAME;
|
|
|
|
/// The server tag must carry the hostname: `simple_alert_helper` keys one alert row per
|
|
/// tag, so a host-less tag lets a replica seeing low disk and a replica seeing free disk
|
|
/// raise and recover the same row every monitor pass.
|
|
///
|
|
/// The hostname is forced to a pod-length name so the tag runs past 50 chars, which
|
|
/// `check_type` must stay wide enough to hold: `create_alert` only logs the insert
|
|
/// error while the notification still fires, so a tag that does not fit re-alerts every
|
|
/// pass and never recovers. Asserting the row persists pins the width and the shape.
|
|
#[ignore = "requires database setup - run with --ignored flag"]
|
|
#[sqlx::test(migrations = "../migrations")]
|
|
async fn server_low_disk_tag_is_per_host(db: Pool<Postgres>) {
|
|
// Both statics are lazy and read on first access inside the call below.
|
|
std::env::set_var("FORCE_HOSTNAME", "windmill-server-7d9f8b6c4d-x2k9p");
|
|
// Force every mount to read as low so the server branch raises.
|
|
std::env::set_var("MIN_FREE_DISK_SPACE_MB", "999999999999");
|
|
|
|
low_disk_alerts(&db, true, false, vec![]).await;
|
|
|
|
let tags: Vec<String> = sqlx::query_scalar(
|
|
"SELECT check_type FROM healthchecks WHERE check_type LIKE 'low-disk-v2-server@%'",
|
|
)
|
|
.fetch_all(&db)
|
|
.await
|
|
.unwrap();
|
|
|
|
assert!(
|
|
!tags.is_empty(),
|
|
"expected at least one server low-disk alert; an alert whose tag does not fit \
|
|
check_type is dropped here while its notification still fires"
|
|
);
|
|
for tag in &tags {
|
|
assert!(
|
|
tag.ends_with(&format!("@{}", *HOSTNAME)),
|
|
"server tag {tag} is not per-host; replicas would share one alert row"
|
|
);
|
|
}
|
|
}
|
|
}
|