feat: run a workspace handler when a variable's value expires (#10837)

* feat: run a workspace handler when a secret variable expires

Secret variables gain an optional `value_expires_at`, and a workspace can point
at a script or flow to run once that date passes, so a rotation is driven by the
platform instead of tracked out of band.

The secret never leaves the database. The sweep selects the path, description and
expiry and nothing else, so the handler is given a reference to rotate rather than
a value to handle. Job results are stored in cleartext and shown in Runs, which is
what makes that a boundary rather than a preference; a test asserts the job's args
never contain the value.

Each row is claimed by writing `expiration_dispatched_for` in the same transaction
as the push, so a failed push un-claims it and a lost race claims nothing. Moving
the date re-arms the row; re-saving the same date does not. A pass takes at most
100 rows under a transaction-scoped advisory lock, and orders so that one broken
workspace cannot starve the rest.

Dispatch runs as `g/variable_expiration_handler`, exempt from the CE group limit
and from non-premium exec metering the way the error and schedule handlers already
are. A handler that keeps failing backs off and raises a critical alert instead of
retrying every pass, and the back-off ends when the handler is repointed.

The expiry travels with the variable through workspace clone, tarball export and
workspace comparison. The dispatch marker deliberately does not, so a fork re-arms
on its own copy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(frontend): expose variable value expiration in the UI

The variable drawer takes an optional expiry on a secret value, and the variables
list badges it, turning red once the date has passed. A workspace settings tab
picks the handler, its extra args, and whether variables under `u/` are skipped.

That mute flag is stored inside the handler setting, so the toggle stays disabled
until a handler is picked: saving it on its own would clear the setting it lives
in and still report success.

The expiry is carried as a tri-state through the draft deploy and copilot paths —
absent leaves the remote value alone, an explicit null clears it — so deploying a
draft cannot silently drop a date someone set elsewhere.

The settings tab names the one case where the configured identity does not apply:
a handler script carrying `on_behalf_of` runs as that user rather than as
`g/variable_expiration_handler`, which matters because the same tab asks the admin
to grant that group write access to the variables it rotates.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(cli): sync variable expiry and the workspace expiration handler

`wmill sync` carries `value_expires_at` on variables in both directions, and
pushes the workspace handler alongside the error and success handlers in the
grouped settings format. Dropping the key from a variable's yaml clears the
remote expiry, so the file stays canonical rather than accumulating fields that
can only ever be added.

The handler follows the same rule, with the caveat recorded where the type is
declared: an omitted handler only reaches the clear when some other setting
differs, because the push short-circuits on a comparison that walks the local
object's keys and so never sees a missing one. That gate is shared with the two
existing handlers and is left as it is here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(frontend): open the variable expiration handler template in a new tab

The "Create a handler" button hands its script template to /scripts/add through
the URL hash, which `load` reads from `window.location`. A client-side navigation
leaves the previous page's hash in place, so the payload was dropped and the
editor opened on the default template at a `u/` path. `target="_blank"` forces a
full page load, matching the success handler's "Create from template" button.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(frontend): make the variable expiry field state its own status

The expiry field rendered a DateTimeInput whose time defaults to 12:00 when it
holds no value, so a variable with no expiry looked like one already configured,
and "cleared" looked identical to "never set". Setting an expiry is optional, so
the field has to say which it is.

Rebuild it on the shape ScriptAdvancedSettings already uses for Timeout, Cache and
Concurrency limits: a toggle whose checked state is the stored value being present,
seeding a date on enable rather than revealing an empty picker, with the input
under its own label. Drop `clearable` with it, so the toggle is the single way to
mean "no expiry" instead of competing with an X.

Stop the surrounding copy promising a run that may not happen. A handler is only
dispatched if the workspace has one configured and does not skip u/ paths, so the
list tooltip now states the date alone, and the form hedges both conditions,
naming the u/ mute only on personal paths where it can apply.

Also anchor the mute toggle's tooltip: Popover's trigger is a span around a
block-level label, which stretched to the full width of its SettingCard and left
the popup floating beside the card rather than under the control.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(rust-client): initialize value_expires_at in the variable models

The generated `CreateVariable` / `EditVariable` structs gain the field from
openapi.yaml, and client.rs builds both with exhaustive literals.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(frontend): drop extra_args when the variable expiration handler is cleared

Clearing the handler nulls the whole setting server-side, extra_args included,
but the tab kept its loaded copy. A handler picked afterwards without reloading
was persisted with the cleared one's arguments, which the tab does not display.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: back the variable expiration handler's built-in Slack and Teams destinations

Backfills the Slack bot token grant for g/variable_expiration_handler, mirroring
20231016142659 for g/error_handler: the slack_bot folder's extra_perms is written
once at Slack connect and never refreshed, and disconnecting leaves the folder
behind, so workspaces that already connected Slack cannot pick the grant up from
the creation-time list alone.

Adds run_variable_expiration_test_job so an admin can exercise the handler on
demand rather than waiting for a real expiry, which may be months away. It goes
through push_variable_expiration_handler so the test run proves what matters:
the handler resolves, and the group can read what it needs, under the same
identity and argument shape the sweep uses.

Also covers extra_args on the edit endpoint, which the settings tab now writes
for the built-ins' channel. The endpoint rebuilds the setting from the request,
so an omitted extra_args erases it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Do7wJTnL6MG6ngFpCGW4DG

* feat(frontend): offer Slack and Teams as variable expiration handlers

Gives the variable expiration tab the same no-code destinations the workspace
error handler has, so "tell #ops before this credential expires" no longer
requires writing a script.

A sibling component rather than a fourth mode of ErrorOrRecoveryHandler: that
one is shared with the workspace error handler and the three schedule handlers,
and its stripped-args list, hub path suffixes and Email tab are all
error-shaped. This reuses the pieces that are already generic
(SlackConnectionStatus, TeamsConnectionStatus, ChannelSelector, ScriptPicker,
SchemaForm) and leaves those five call sites untouched.

extra_args stops being an API/CLI-only escape hatch now that the built-ins write
their channel into it, so it joins the unsaved-changes comparison and the
discard path, normalised through the existing helper that drops the two
auto-managed keys.

The two hub ids are placeholders until the scripts are pushed to the hub.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Do7wJTnL6MG6ngFpCGW4DG

* fix(frontend): scope the handler identity notice to the chosen destination

The notice told every admin to grant g/variable_expiration_handler write access
to the variables it rotates. That is right for a custom handler and wrong for
the built-ins, which only post a message and are granted the Slack bot token by
migration. Following it for a Slack handler would hand a group write access to
the workspace's secrets for no reason.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Do7wJTnL6MG6ngFpCGW4DG

* feat(frontend): flag the Slack and Teams handlers as EE

Connecting a workspace to Slack or Teams is EE-only (oauth2_oss.rs returns
"Slack only available on enterprise"), so on CE those two tabs can never be
completed. Says so up front, the way the error/success handler tab does,
instead of leaving the user at a "not connected" badge with no working path.

Only on those two tabs: a custom script or flow handler is CE.

Drops the "Connect Slack first" alert, now that the banner and the connection
badge above it already say it twice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Do7wJTnL6MG6ngFpCGW4DG

* chore(backend): regenerate the sqlx cache after the merge

The merged workspace_settings column list in change_workspace_id is a new
query text, so it needs a new cached entry. Drops the two pre-merge entries
it supersedes; both had been left with conflict markers by the merge, since
rename/rename markers are eight characters and escape the usual scan.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Do7wJTnL6MG6ngFpCGW4DG

* fix(cli): narrow the expiry-clear check to a variable that exists remotely

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Do7wJTnL6MG6ngFpCGW4DG

* feat(frontend): point the built-in handlers at their published hub scripts

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* chore(backend): regenerate the sqlx cache after the merge

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(frontend): disable the Slack and Teams handler controls without a license

Matches the workspace error handler, which greys out its built-in destinations
and hides the test button on an unlicensed instance. The custom destination stays
editable: unlike the error handler, the variable expiration handler itself is not
an EE feature.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: record which destination a variable expiration handler uses

handler_configured now counts slack, teams, custom or cleared instead of
set or cleared, so usage shows whether the built-in destinations are picked.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(frontend): let the variable expiration handler template run on any worker

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: guard the variable expiration test endpoint and correct its help text

* fix: reuse the error handler's slack arg and built-in path helpers

* docs: trim variable expiration sweep and backfill comments

* feat: put the variable value expiration under an advanced section

* fix: restrict the expiration test job to built-ins and keep expiry on conflict reload

* fix: link the expiry hint to the variable's workspace and document built-in test paths

* fix: give the expiration sweep its own lock id and reuse double_option

* fix: pin the variable expiration presets so the hub-paths test accepts them

* feat: let the variable expiration handler set how far ahead it runs

* fix: give the notify before expiry setting its own card

* feat: remind the variable expiration handler at a set interval

* fix: tighten the gap between the Slack channel field and its test button

* feat: refuse a value expiry on variables linked to an OAuth account

* refactor: reuse the shared handler form for variable expiration

* refactor: sweep expiring variables round-robin and check the handler exists

* chore: trim comments to the constraints they record

* fix: compare expiry dates by moment when deploying a variable draft

* refactor: rename value_expires_at to value_valid_until

* fix: neutralize variable fields in expiration handler args

* fix: reserve the variable_expiration_handler group name for admins

* fix: send first expiry notices before reminders

* test: pin that a handler failing to start leaves its rows unclaimed

* chore: say any hub handler path skips the existence check

* fix: empty a pre-existing handler group before granting it the Slack token

* docs: correct the registered feature usage counts

* fix: let a workspace whose handler failed to start sit out for an hour

* fix: send first notices before reminders across workspaces

* fix: end a failed handler's wait as soon as it is redeployed

* fix: show an expiry-only variable change in the AI fork diff

* chore: regenerate workspace tool schemas

* test: keep one variable expiration guard per behavior

* fix: use the renamed hub expiration handlers and template

* feat: carry the variable expiry in windmill-utils-internal 1.8.4

* fix: keep the variable expiry through MCP drafts

* chore: bump windmill-utils-internal to 1.8.4

* fix: keep the variable expiry through mcp deploys

* fix: disable saving a slack or teams expiry handler without ee

* fix: skip the failure mark when the expiry handler was saved mid-sweep

* fix: count expiry telemetry only where the expiry changed

* test: assert the exact variable expiry in export and cli push tests

* fix: stamp an expiry handler failure with the pass's read time

* fix: wait for a pending handler lock instead of marking it failed

* test: isolate expiry handler tests from the deployed-script cache

* fix: retry a failing expiry handler every pass instead of waiting an hour

* chore: update ee-repo-ref to b6e23d229966cf0019c0a1926a254fda658d8def

This commit updates the EE repository reference after PR #744 was merged in windmill-ee-private.

Previous ee-repo-ref: c52c06d145b13a6fb04459e85ca5818bf3c1bff4

New ee-repo-ref: b6e23d229966cf0019c0a1926a254fda658d8def

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: hugocasa <hugo@casademont.ch>

backend/windmill-queue/src/jobs.rs

@@ -170,12 +170,15 @@ const ERROR_HANDLER_USERNAME: &str = "error_handler";
 pub const SCHEDULE_ERROR_HANDLER_USERNAME: &str = "schedule_error_handler";
 const GLOBAL_ERROR_HANDLER_USERNAME: &str = "global";
 const SUCCESS_HANDLER_USERNAME: &str = "success_handler";
+const VARIABLE_EXPIRATION_HANDLER_USERNAME: &str = "variable_expiration_handler";
 
 pub const ERROR_HANDLER_USER_GROUP: &str = "g/error_handler";
 pub const ERROR_HANDLER_USER_EMAIL: &str = "error_handler@windmill.dev";
 pub const SCHEDULE_ERROR_HANDLER_USER_EMAIL: &str = "schedule_error_handler@windmill.dev";
 pub const SUCCESS_HANDLER_USER_GROUP: &str = "g/success_handler";
 pub const SUCCESS_HANDLER_USER_EMAIL: &str = "success_handler@windmill.dev";
+pub const VARIABLE_EXPIRATION_HANDLER_USER_GROUP: &str = "g/variable_expiration_handler";
+pub const VARIABLE_EXPIRATION_HANDLER_USER_EMAIL: &str = "variable_expiration_handler@windmill.dev";
 
 #[derive(Clone, Debug, Serialize, Deserialize)]
 pub struct CanceledBy {
@@ -3599,6 +3602,100 @@ pub async fn push_error_handler<'a, 'c, T: Serialize + Send + Sync>(
     return Ok(uuid);
 }
 
+/// Handler arguments, so the field names are wire format. Never add the value: job args and
+/// results are stored in cleartext and shown in Runs.
+#[derive(Serialize)]
+pub struct ExpiringVariable {
+    pub workspace_id: String,
+    pub variable_path: String,
+    pub description: String,
+    pub value_valid_until: DateTime<Utc>,
+    pub is_secret: bool,
+}
+
+/// Takes and returns the caller's transaction so the claim and the push commit together.
+pub(crate) async fn push_variable_expiration_handler<'c>(
+    db: &Pool<Postgres>,
+    tx: Transaction<'c, Postgres>,
+    w_id: &str,
+    (payload, tag, on_behalf_of): (
+        JobPayload,
+        Option<String>,
+        Option<windmill_common::jobs::OnBehalfOf>,
+    ),
+    expiring: &ExpiringVariable,
+    extra_args: Option<Json<Box<RawValue>>>,
+) -> windmill_common::error::Result<(Uuid, Transaction<'c, Postgres>)> {
+    // Serialized whole so a new `ExpiringVariable` field reaches handlers. Any writer of the
+    // variable controls these, so only they are neutralized: the admin's `extra_args` keep their
+    // `$res:` references, which is how the handler gets its token.
+    let mut extra = omit_untrusted_interpolation(
+        serde_json::from_value(serde_json::to_value(expiring)?)?,
+        "The following resource has been omitted for security reasons: ",
+    );
+
+    if let Some(args_v) = extra_args {
+        if let Ok(args_m) = serde_json::from_str::<HashMap<String, Box<RawValue>>>(args_v.get()) {
+            extra.extend(args_m);
+        } else {
+            return Err(error::Error::ExecutionErr(
+                "args of scripts needs to be dict".to_string(),
+            ));
+        }
+    }
+
+    let (email, permissioned_as) = if let Some(on_behalf_of) = on_behalf_of.as_ref() {
+        (
+            on_behalf_of.email.as_str(),
+            on_behalf_of.permissioned_as.clone(),
+        )
+    } else {
+        (
+            VARIABLE_EXPIRATION_HANDLER_USER_EMAIL,
+            VARIABLE_EXPIRATION_HANDLER_USER_GROUP.to_string(),
+        )
+    };
+
+    let empty: HashMap<String, Box<RawValue>> = HashMap::new();
+    let args = PushArgs { extra: Some(extra), args: &empty };
+    push(
+        db,
+        PushIsolationLevel::Transaction(tx),
+        w_id,
+        payload,
+        args,
+        VARIABLE_EXPIRATION_HANDLER_USERNAME,
+        email,
+        permissioned_as,
+        Some(&format!(
+            "variable.expiration.{}",
+            expiring.variable_path.replace('/', ".")
+        )),
+        None,
+        None,
+        None,
+        None,
+        None,
+        None,
+        None,
+        false,
+        false,
+        None,
+        true,
+        tag,
+        None,
+        None,
+        None,
+        None,
+        false,
+        None,
+        None,
+        None,
+        None,
+    )
+    .await
+}
+
 pub async fn push_success_handler<'a, 'c, T: Serialize + Send + Sync>(
     db: &Pool<Postgres>,
     job_id: Uuid,
@@ -6357,6 +6454,7 @@ async fn push_inner<'c, 'd>(
                 if !team_plan_status.premium
                     && email != ERROR_HANDLER_USER_EMAIL
                     && email != SCHEDULE_ERROR_HANDLER_USER_EMAIL
+                    && email != VARIABLE_EXPIRATION_HANDLER_USER_EMAIL
                     && email != recovery_email
                     && email != "worker@windmill.dev"
                     && email != SUPERADMIN_SECRET_EMAIL
@@ -8886,7 +8984,8 @@ mod preset_handler_path_tests {
         let hub_paths: std::collections::HashMap<String, String> =
             serde_json::from_str(include_str!("../../../frontend/src/lib/hubPaths.json")).unwrap();
         for (name, path) in hub_paths {
-            if path.contains("-handler-") {
+            // Variable expiration handlers run as their own group whatever their path.
+            if path.contains("-handler-") && !path.contains("variable-expiration") {
                 assert!(
                     is_preset_handler_path(&path),
                     "{name}: {path} is not pinned"

backend/windmill-queue/src/lib.rs

@@ -34,6 +34,7 @@ pub mod jobs;
 pub mod jobs_ee;
 pub mod jobs_oss;
 pub mod schedule;
+pub mod variable_expiration;
 pub use jobs::*;
 pub mod flow_status;
 pub mod tags;

backend/windmill-queue/src/variable_expiration.rs

@@ -0,0 +1,309 @@
+/*
+ * Author: Ruben Fiszel
+ * Copyright: Windmill Labs, Inc 2022
+ * This file and its contents are licensed under the AGPLv3 License.
+ * Please see the included NOTICE for copyright information and
+ * LICENSE-AGPL for a copy of the license.
+ */
+
+//! Dispatches the workspace variable expiration handler for variables coming due.
+
+use std::collections::{HashMap, HashSet};
+
+use serde_json::value::RawValue;
+use sqlx::types::Json;
+use windmill_common::jobs::get_payload_tag_from_prefixed_path;
+use windmill_common::utils::{report_critical_error, report_recovered_critical_error};
+use windmill_common::DB;
+
+use crate::jobs::{push_variable_expiration_handler, ExpiringVariable};
+
+/// Lead time when the handler setting has no `notify_before_secs`.
+const VARIABLE_EXPIRATION_DEFAULT_NOTIFY_BEFORE_SECS: i64 = 24 * 3600;
+
+/// Subtracted from `remind_every_secs` so a reminder whose interval is a multiple of the
+/// sweep's period is not skipped by tick jitter.
+const VARIABLE_EXPIRATION_REMINDER_SLACK_SECS: i64 = 30;
+
+/// Advisory lock id ensuring only one server replica sweeps at a time. Must not reuse one of
+/// the other monitor pass locks (737_483_920..=737_483_925): both sides try-lock and skip.
+const VARIABLE_EXPIRATION_LOCK_ID: i64 = 737_483_926;
+
+/// System caller only: dispatches across every workspace. A handler job that fails is not
+/// re-dispatched before its next reminder, as re-running it would repeat its side effects
+/// against the third party it rotates; retrying is the handler's job.
+pub async fn dispatch_expiring_variables(db: &DB) {
+    // Transaction-scoped, not session-scoped: a session lock would go back to the pool still
+    // held if this future were dropped mid-pass, wedging the sweep on every replica.
+    let mut lock_tx = match db.begin().await {
+        Ok(tx) => tx,
+        Err(e) => {
+            tracing::error!("variable expiration: failed to begin lock tx: {e:#}");
+            return;
+        }
+    };
+    let locked: bool = match sqlx::query_scalar("SELECT pg_try_advisory_xact_lock($1)")
+        .bind(VARIABLE_EXPIRATION_LOCK_ID)
+        .fetch_one(&mut *lock_tx)
+        .await
+    {
+        Ok(v) => v,
+        Err(e) => {
+            tracing::error!("variable expiration: advisory lock failed: {e:#}");
+            return;
+        }
+    };
+    if !locked {
+        return;
+    }
+
+    dispatch_expiring_variables_inner(db).await;
+
+    if let Err(e) = lock_tx.rollback().await {
+        tracing::error!("variable expiration: releasing lock failed: {e:#}");
+    }
+}
+
+async fn dispatch_expiring_variables_inner(db: &DB) {
+    resolve_removed_handlers(db).await;
+
+    // No lower bound on `value_valid_until`, so a downed instance catches up. No cap either:
+    // everything due is tried every pass, so no workspace or kind of notice can be crowded out.
+    let due = sqlx::query!(
+        r#"SELECT v.workspace_id, v.path, v.description, v.value_valid_until AS "value_valid_until!",
+                  v.is_secret, v.expiration_dispatched_for, v.expiration_notified_at,
+                  ws.variable_expiration_handler->>'path' AS "handler_path!",
+                  (ws.variable_expiration_handler->'extra_args')::text::json
+                      AS "extra_args: Json<Box<RawValue>>"
+           FROM variable v
+           JOIN workspace_settings ws ON ws.workspace_id = v.workspace_id
+           JOIN workspace w ON w.id = v.workspace_id
+           WHERE NOT w.deleted
+             AND v.value_valid_until IS NOT NULL
+             AND (v.value_valid_until IS DISTINCT FROM v.expiration_dispatched_for
+                  OR (v.value_valid_until > now()
+                      AND v.expiration_notified_at <= now() - make_interval(secs =>
+                          NULLIF((ws.variable_expiration_handler->>'remind_every_secs')::bigint, 0) - $2)))
+             AND v.value_valid_until <= now() + make_interval(secs =>
+                 COALESCE((ws.variable_expiration_handler->>'notify_before_secs')::bigint, $1))
+             AND ws.variable_expiration_handler->>'path' IS NOT NULL
+             AND (NOT COALESCE((ws.variable_expiration_handler->>'muted_on_user_path')::boolean, false)
+                  OR v.path NOT LIKE 'u/%')"#,
+        VARIABLE_EXPIRATION_DEFAULT_NOTIFY_BEFORE_SECS,
+        VARIABLE_EXPIRATION_REMINDER_SLACK_SECS,
+    )
+    .fetch_all(db)
+    .await;
+
+    let due = match due {
+        Ok(due) => due,
+        Err(e) => {
+            tracing::error!("Error listing expiring variables: {e:#}");
+            return;
+        }
+    };
+
+    let mut dispatched = 0usize;
+    // A handler fails alike for every variable of its workspace: try it once per pass.
+    let mut failed_this_pass: HashSet<String> = HashSet::new();
+    // Hub handlers are shared by every workspace using them, so a hub outage is fetched once
+    // per pass rather than once per workspace.
+    let mut failed_hub_handlers: HashMap<String, String> = HashMap::new();
+
+    for row in due {
+        if failed_this_pass.contains(&row.workspace_id) {
+            continue;
+        }
+        // Resolved before the claim's transaction opens, so a pass never holds one connection
+        // while waiting on another.
+        let handler = match failed_hub_handlers.get(&row.handler_path) {
+            Some(error) => Err(error.clone()),
+            None => get_payload_tag_from_prefixed_path(&row.handler_path, db, &row.workspace_id)
+                .await
+                .map_err(|e| {
+                    let error = format!("{e:#}");
+                    if row.handler_path.starts_with("script/hub/")
+                        || row.handler_path.starts_with("flow/hub/")
+                    {
+                        failed_hub_handlers.insert(row.handler_path.clone(), error.clone());
+                    }
+                    error
+                }),
+        };
+        let handler = match handler {
+            Ok(handler) => handler,
+            Err(error) => {
+                failed_this_pass.insert(row.workspace_id.clone());
+                record_failure(db, &row.workspace_id, &row.handler_path, &error).await;
+                continue;
+            }
+        };
+
+        // Claim and push in one transaction. The claim matches the row as it was read, so it
+        // loses both to a concurrent re-arm and to whichever replica got there first.
+        let mut tx = match db.begin().await {
+            Ok(tx) => tx,
+            Err(e) => {
+                tracing::error!("Error opening tx for expiring variable {}: {e:#}", row.path);
+                continue;
+            }
+        };
+
+        let claimed = sqlx::query_scalar!(
+            "UPDATE variable SET expiration_dispatched_for = $3, expiration_notified_at = now()
+             WHERE workspace_id = $1 AND path = $2 AND value_valid_until = $3
+               AND expiration_dispatched_for IS NOT DISTINCT FROM $4
+               AND expiration_notified_at IS NOT DISTINCT FROM $5
+             RETURNING 1",
+            row.workspace_id,
+            row.path,
+            row.value_valid_until,
+            row.expiration_dispatched_for,
+            row.expiration_notified_at,
+        )
+        .fetch_optional(&mut *tx)
+        .await;
+
+        match claimed {
+            Ok(Some(_)) => {}
+            Ok(None) => continue,
+            Err(e) => {
+                tracing::error!("Error claiming expiring variable {}: {e:#}", row.path);
+                continue;
+            }
+        }
+
+        let expiring = ExpiringVariable {
+            workspace_id: row.workspace_id.clone(),
+            variable_path: row.path.clone(),
+            description: row.description,
+            value_valid_until: row.value_valid_until,
+            is_secret: row.is_secret,
+        };
+
+        let pushed = push_variable_expiration_handler(
+            db,
+            tx,
+            &row.workspace_id,
+            handler,
+            &expiring,
+            row.extra_args,
+        )
+        .await;
+
+        let tx = match pushed {
+            Ok((_job_id, tx)) => tx,
+            Err(e) => {
+                // Dropping the tx un-claims the variable, so the next pass retries it.
+                failed_this_pass.insert(row.workspace_id.clone());
+                record_failure(db, &row.workspace_id, &row.handler_path, &format!("{e:#}")).await;
+                continue;
+            }
+        };
+
+        if let Err(e) = tx.commit().await {
+            tracing::error!("Error committing dispatch for {}: {e:#}", row.path);
+            continue;
+        }
+        windmill_common::feature_usage::log_feature_usage(
+            "variable_expiration",
+            "dispatched",
+            if row.is_secret { "secret" } else { "plain" },
+        );
+        dispatched += 1;
+        recover(db, &row.workspace_id, &row.handler_path).await;
+    }
+
+    if dispatched > 0 {
+        tracing::info!("Dispatched {dispatched} variable expiration handler job(s)");
+    }
+}
+
+/// Alerts once per failure, not per pass: the alert is resolved when the handler next starts.
+async fn record_failure(db: &DB, w_id: &str, handler_path: &str, error: &str) {
+    let newly_failing = sqlx::query_scalar!(
+        "UPDATE workspace_settings SET variable_expiration_failed_at = now()
+         WHERE workspace_id = $1 AND variable_expiration_failed_at IS NULL
+         RETURNING 1",
+        w_id,
+    )
+    .fetch_optional(db)
+    .await;
+    match newly_failing {
+        Ok(Some(_)) => {
+            report_critical_error(
+                format!(
+                    "The variable expiration handler {handler_path} of workspace {w_id} failed to \
+                     start, so variables in this workspace are not being notified. It is retried \
+                     on every check, and this alert is resolved once it starts: {error}"
+                ),
+                db.clone(),
+                Some(w_id),
+                Some(&alert_resource(w_id)),
+            )
+            .await
+        }
+        Ok(None) => {}
+        Err(e) => {
+            tracing::error!("Error recording variable expiration handler failure of {w_id}: {e:#}")
+        }
+    }
+}
+
+/// Tags a failing handler's critical alerts, so they can be resolved together.
+fn alert_resource(w_id: &str) -> String {
+    format!("variable_expiration_handler:{w_id}")
+}
+
+/// A removed handler has nothing left to retry, so its failure would never be resolved.
+async fn resolve_removed_handlers(db: &DB) {
+    let removed = sqlx::query_scalar!(
+        "UPDATE workspace_settings SET variable_expiration_failed_at = NULL
+         WHERE variable_expiration_failed_at IS NOT NULL AND variable_expiration_handler IS NULL
+         RETURNING workspace_id",
+    )
+    .fetch_all(db)
+    .await;
+    match removed {
+        Ok(removed) => {
+            for w_id in removed {
+                report_recovered_critical_error(
+                    format!("The variable expiration handler of workspace {w_id} was removed"),
+                    db.clone(),
+                    Some(&w_id),
+                    Some(&alert_resource(&w_id)),
+                )
+                .await;
+            }
+        }
+        Err(e) => tracing::error!("Error resolving removed variable expiration handlers: {e:#}"),
+    }
+}
+
+async fn recover(db: &DB, w_id: &str, handler_path: &str) {
+    let cleared = sqlx::query_scalar!(
+        "UPDATE workspace_settings SET variable_expiration_failed_at = NULL
+         WHERE workspace_id = $1 AND variable_expiration_failed_at IS NOT NULL
+         RETURNING 1",
+        w_id,
+    )
+    .fetch_optional(db)
+    .await;
+    match cleared {
+        Ok(Some(_)) => {
+            report_recovered_critical_error(
+                format!(
+                "The variable expiration handler {handler_path} of workspace {w_id} started again"
+            ),
+                db.clone(),
+                Some(w_id),
+                Some(&alert_resource(w_id)),
+            )
+            .await
+        }
+        Ok(None) => {}
+        Err(e) => {
+            tracing::error!("Error clearing variable expiration handler failure of {w_id}: {e:#}")
+        }
+    }
+}