Files
windmill/backend/windmill-api/src/workspaces_export.rs
T
7a29abc7ee feat: run a workspace handler when a variable's value expires (#10837)
* feat: run a workspace handler when a secret variable expires

Secret variables gain an optional `value_expires_at`, and a workspace can point
at a script or flow to run once that date passes, so a rotation is driven by the
platform instead of tracked out of band.

The secret never leaves the database. The sweep selects the path, description and
expiry and nothing else, so the handler is given a reference to rotate rather than
a value to handle. Job results are stored in cleartext and shown in Runs, which is
what makes that a boundary rather than a preference; a test asserts the job's args
never contain the value.

Each row is claimed by writing `expiration_dispatched_for` in the same transaction
as the push, so a failed push un-claims it and a lost race claims nothing. Moving
the date re-arms the row; re-saving the same date does not. A pass takes at most
100 rows under a transaction-scoped advisory lock, and orders so that one broken
workspace cannot starve the rest.

Dispatch runs as `g/variable_expiration_handler`, exempt from the CE group limit
and from non-premium exec metering the way the error and schedule handlers already
are. A handler that keeps failing backs off and raises a critical alert instead of
retrying every pass, and the back-off ends when the handler is repointed.

The expiry travels with the variable through workspace clone, tarball export and
workspace comparison. The dispatch marker deliberately does not, so a fork re-arms
on its own copy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(frontend): expose variable value expiration in the UI

The variable drawer takes an optional expiry on a secret value, and the variables
list badges it, turning red once the date has passed. A workspace settings tab
picks the handler, its extra args, and whether variables under `u/` are skipped.

That mute flag is stored inside the handler setting, so the toggle stays disabled
until a handler is picked: saving it on its own would clear the setting it lives
in and still report success.

The expiry is carried as a tri-state through the draft deploy and copilot paths —
absent leaves the remote value alone, an explicit null clears it — so deploying a
draft cannot silently drop a date someone set elsewhere.

The settings tab names the one case where the configured identity does not apply:
a handler script carrying `on_behalf_of` runs as that user rather than as
`g/variable_expiration_handler`, which matters because the same tab asks the admin
to grant that group write access to the variables it rotates.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat(cli): sync variable expiry and the workspace expiration handler

`wmill sync` carries `value_expires_at` on variables in both directions, and
pushes the workspace handler alongside the error and success handlers in the
grouped settings format. Dropping the key from a variable's yaml clears the
remote expiry, so the file stays canonical rather than accumulating fields that
can only ever be added.

The handler follows the same rule, with the caveat recorded where the type is
declared: an omitted handler only reaches the clear when some other setting
differs, because the push short-circuits on a comparison that walks the local
object's keys and so never sees a missing one. That gate is shared with the two
existing handlers and is left as it is here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(frontend): open the variable expiration handler template in a new tab

The "Create a handler" button hands its script template to /scripts/add through
the URL hash, which `load` reads from `window.location`. A client-side navigation
leaves the previous page's hash in place, so the payload was dropped and the
editor opened on the default template at a `u/` path. `target="_blank"` forces a
full page load, matching the success handler's "Create from template" button.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(frontend): make the variable expiry field state its own status

The expiry field rendered a DateTimeInput whose time defaults to 12:00 when it
holds no value, so a variable with no expiry looked like one already configured,
and "cleared" looked identical to "never set". Setting an expiry is optional, so
the field has to say which it is.

Rebuild it on the shape ScriptAdvancedSettings already uses for Timeout, Cache and
Concurrency limits: a toggle whose checked state is the stored value being present,
seeding a date on enable rather than revealing an empty picker, with the input
under its own label. Drop `clearable` with it, so the toggle is the single way to
mean "no expiry" instead of competing with an X.

Stop the surrounding copy promising a run that may not happen. A handler is only
dispatched if the workspace has one configured and does not skip u/ paths, so the
list tooltip now states the date alone, and the form hedges both conditions,
naming the u/ mute only on personal paths where it can apply.

Also anchor the mute toggle's tooltip: Popover's trigger is a span around a
block-level label, which stretched to the full width of its SettingCard and left
the popup floating beside the card rather than under the control.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(rust-client): initialize value_expires_at in the variable models

The generated `CreateVariable` / `EditVariable` structs gain the field from
openapi.yaml, and client.rs builds both with exhaustive literals.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(frontend): drop extra_args when the variable expiration handler is cleared

Clearing the handler nulls the whole setting server-side, extra_args included,
but the tab kept its loaded copy. A handler picked afterwards without reloading
was persisted with the cleared one's arguments, which the tab does not display.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* feat: back the variable expiration handler's built-in Slack and Teams destinations

Backfills the Slack bot token grant for g/variable_expiration_handler, mirroring
20231016142659 for g/error_handler: the slack_bot folder's extra_perms is written
once at Slack connect and never refreshed, and disconnecting leaves the folder
behind, so workspaces that already connected Slack cannot pick the grant up from
the creation-time list alone.

Adds run_variable_expiration_test_job so an admin can exercise the handler on
demand rather than waiting for a real expiry, which may be months away. It goes
through push_variable_expiration_handler so the test run proves what matters:
the handler resolves, and the group can read what it needs, under the same
identity and argument shape the sweep uses.

Also covers extra_args on the edit endpoint, which the settings tab now writes
for the built-ins' channel. The endpoint rebuilds the setting from the request,
so an omitted extra_args erases it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Do7wJTnL6MG6ngFpCGW4DG

* feat(frontend): offer Slack and Teams as variable expiration handlers

Gives the variable expiration tab the same no-code destinations the workspace
error handler has, so "tell #ops before this credential expires" no longer
requires writing a script.

A sibling component rather than a fourth mode of ErrorOrRecoveryHandler: that
one is shared with the workspace error handler and the three schedule handlers,
and its stripped-args list, hub path suffixes and Email tab are all
error-shaped. This reuses the pieces that are already generic
(SlackConnectionStatus, TeamsConnectionStatus, ChannelSelector, ScriptPicker,
SchemaForm) and leaves those five call sites untouched.

extra_args stops being an API/CLI-only escape hatch now that the built-ins write
their channel into it, so it joins the unsaved-changes comparison and the
discard path, normalised through the existing helper that drops the two
auto-managed keys.

The two hub ids are placeholders until the scripts are pushed to the hub.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Do7wJTnL6MG6ngFpCGW4DG

* fix(frontend): scope the handler identity notice to the chosen destination

The notice told every admin to grant g/variable_expiration_handler write access
to the variables it rotates. That is right for a custom handler and wrong for
the built-ins, which only post a message and are granted the Slack bot token by
migration. Following it for a Slack handler would hand a group write access to
the workspace's secrets for no reason.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Do7wJTnL6MG6ngFpCGW4DG

* feat(frontend): flag the Slack and Teams handlers as EE

Connecting a workspace to Slack or Teams is EE-only (oauth2_oss.rs returns
"Slack only available on enterprise"), so on CE those two tabs can never be
completed. Says so up front, the way the error/success handler tab does,
instead of leaving the user at a "not connected" badge with no working path.

Only on those two tabs: a custom script or flow handler is CE.

Drops the "Connect Slack first" alert, now that the banner and the connection
badge above it already say it twice.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Do7wJTnL6MG6ngFpCGW4DG

* chore(backend): regenerate the sqlx cache after the merge

The merged workspace_settings column list in change_workspace_id is a new
query text, so it needs a new cached entry. Drops the two pre-merge entries
it supersedes; both had been left with conflict markers by the merge, since
rename/rename markers are eight characters and escape the usual scan.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Do7wJTnL6MG6ngFpCGW4DG

* fix(cli): narrow the expiry-clear check to a variable that exists remotely

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Do7wJTnL6MG6ngFpCGW4DG

* feat(frontend): point the built-in handlers at their published hub scripts

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* chore(backend): regenerate the sqlx cache after the merge

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(frontend): disable the Slack and Teams handler controls without a license

Matches the workspace error handler, which greys out its built-in destinations
and hides the test button on an unlicensed instance. The custom destination stays
editable: unlike the error handler, the variable expiration handler itself is not
an EE feature.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* feat: record which destination a variable expiration handler uses

handler_configured now counts slack, teams, custom or cleared instead of
set or cleared, so usage shows whether the built-in destinations are picked.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix(frontend): let the variable expiration handler template run on any worker

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

* fix: guard the variable expiration test endpoint and correct its help text

* fix: reuse the error handler's slack arg and built-in path helpers

* docs: trim variable expiration sweep and backfill comments

* feat: put the variable value expiration under an advanced section

* fix: restrict the expiration test job to built-ins and keep expiry on conflict reload

* fix: link the expiry hint to the variable's workspace and document built-in test paths

* fix: give the expiration sweep its own lock id and reuse double_option

* fix: pin the variable expiration presets so the hub-paths test accepts them

* feat: let the variable expiration handler set how far ahead it runs

* fix: give the notify before expiry setting its own card

* feat: remind the variable expiration handler at a set interval

* fix: tighten the gap between the Slack channel field and its test button

* feat: refuse a value expiry on variables linked to an OAuth account

* refactor: reuse the shared handler form for variable expiration

* refactor: sweep expiring variables round-robin and check the handler exists

* chore: trim comments to the constraints they record

* fix: compare expiry dates by moment when deploying a variable draft

* refactor: rename value_expires_at to value_valid_until

* fix: neutralize variable fields in expiration handler args

* fix: reserve the variable_expiration_handler group name for admins

* fix: send first expiry notices before reminders

* test: pin that a handler failing to start leaves its rows unclaimed

* chore: say any hub handler path skips the existence check

* fix: empty a pre-existing handler group before granting it the Slack token

* docs: correct the registered feature usage counts

* fix: let a workspace whose handler failed to start sit out for an hour

* fix: send first notices before reminders across workspaces

* fix: end a failed handler's wait as soon as it is redeployed

* fix: show an expiry-only variable change in the AI fork diff

* chore: regenerate workspace tool schemas

* test: keep one variable expiration guard per behavior

* fix: use the renamed hub expiration handlers and template

* feat: carry the variable expiry in windmill-utils-internal 1.8.4

* fix: keep the variable expiry through MCP drafts

* chore: bump windmill-utils-internal to 1.8.4

* fix: keep the variable expiry through mcp deploys

* fix: disable saving a slack or teams expiry handler without ee

* fix: skip the failure mark when the expiry handler was saved mid-sweep

* fix: count expiry telemetry only where the expiry changed

* test: assert the exact variable expiry in export and cli push tests

* fix: stamp an expiry handler failure with the pass's read time

* fix: wait for a pending handler lock instead of marking it failed

* test: isolate expiry handler tests from the deployed-script cache

* fix: retry a failing expiry handler every pass instead of waiting an hour

* chore: update ee-repo-ref to b6e23d229966cf0019c0a1926a254fda658d8def

This commit updates the EE repository reference after PR #744 was merged in windmill-ee-private.

Previous ee-repo-ref: c52c06d145b13a6fb04459e85ca5818bf3c1bff4

New ee-repo-ref: b6e23d229966cf0019c0a1926a254fda658d8def

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: hugocasa <hugo@casademont.ch>
2026-10-23 18:12:55 +02:00

1833 lines
70 KiB
Rust

/*
* Author: Ruben Fiszel
* Copyright: Windmill Labs, Inc 2022
* This file and its contents are licensed under the AGPLv3 License.
* Please see the included NOTICE for copyright information and
* LICENSE-AGPL for a copy of the license.
*/
use std::collections::HashMap;
use crate::db::ApiAuthed;
use crate::{apps::AppWithLastVersion, db::DB, folders::Folder};
use windmill_api_auth::check_scopes;
#[cfg(any(
feature = "http_trigger",
feature = "websocket",
feature = "postgres_trigger",
feature = "mqtt_trigger",
feature = "amqp_trigger",
all(
feature = "enterprise",
any(
feature = "kafka",
feature = "sqs_trigger",
feature = "gcp_trigger",
feature = "azure_trigger",
feature = "nats",
feature = "smtp",
),
feature = "private"
)
))]
use crate::triggers::TriggerCrud;
use axum::{
extract::{Extension, Path, Query},
response::IntoResponse,
};
use http::HeaderName;
use itertools::Itertools;
use windmill_common::runnable_settings::{ConcurrencySettings, DebouncingSettings};
use windmill_common::scripts::ScriptRunnableSettingsHandle;
use windmill_common::utils::require_admin;
use windmill_common::variables::decrypt;
use windmill_common::worker::WINDMILL_DIR;
use windmill_common::{
db::UserDB,
error::{to_anyhow, Error, Result},
flows::Flow,
schedule::Schedule,
scripts::{Schema, Script, ScriptLang},
variables::{build_crypt, ExportableListableVariable},
workspace_dependencies::WorkspaceDependencies,
};
use hyper::header;
use serde::{Deserialize, Serialize};
use serde_json::Value;
use tempfile::TempDir;
use tokio::fs::File;
use tokio_util::io::ReaderStream;
use windmill_store::resources::{Resource, ResourceType};
#[derive(Serialize)]
struct ScriptMetadata {
summary: String,
description: String,
schema: Option<Schema>,
lock: Option<String>,
kind: String,
#[serde(skip_serializing_if = "Option::is_none")]
envs: Option<Vec<String>>,
#[serde(skip_serializing_if = "Option::is_none")]
cache_ttl: Option<i32>,
#[serde(skip_serializing_if = "Option::is_none")]
cache_ignore_s3_path: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
dedicated_worker: Option<bool>,
#[serde(skip_serializing_if = "is_none_or_false")]
ws_error_handler_muted: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
priority: Option<i16>,
#[serde(skip_serializing_if = "Option::is_none")]
tag: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub timeout: Option<i32>,
#[serde(skip_serializing_if = "Option::is_none")]
pub delete_after_secs: Option<i32>,
#[serde(skip_serializing_if = "Option::is_none")]
pub restart_unless_cancelled: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub visible_to_runner_only: Option<bool>,
// auto_kind is intentionally excluded from export — it is auto-detected by the
// parser at deploy time from the script content (workflow/task patterns for "wac",
// no main function for "lib").
#[serde(skip_serializing)]
#[allow(dead_code)]
pub auto_kind: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub codebase: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub has_preprocessor: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub on_behalf_of_email: Option<String>,
/// Emitted instead of the address from syncBehavior v1 on, where the CLI only needs to
/// know *that* the runnable has an identity — it re-reads the value itself on push.
#[serde(skip_serializing_if = "Option::is_none")]
pub has_on_behalf_of: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub modules: Option<std::collections::HashMap<String, windmill_common::scripts::ScriptModule>>,
#[serde(flatten)]
pub concurrency_settings: ConcurrencySettings,
#[serde(flatten)]
pub debouncing_settings: DebouncingSettings,
#[serde(skip_serializing_if = "Option::is_none")]
pub labels: Option<Vec<String>>,
#[serde(skip_serializing_if = "Option::is_none")]
pub job_token_scopes: Option<Vec<String>>,
#[serde(skip_serializing_if = "is_empty_extra_perms")]
pub extra_perms: serde_json::Value,
}
/// `syncBehavior` as the CLI writes it (`"v1"`); anything unparseable is v0, matching
/// `parseSyncBehavior` on the client.
fn parse_sync_behavior_version(value: Option<&str>) -> u32 {
value
.and_then(|v| v.strip_prefix('v'))
.and_then(|v| v.parse().ok())
.unwrap_or(0)
}
/// The address a runnable's principal resolves to, memoised for the duration of one export.
///
/// `on_behalf_of` is sparse — most runnables carry none, and those that do
/// usually share a handful of identities — so this collapses to roughly one lookup per
/// distinct principal instead of one per row.
async fn derive_email(
cache: &mut HashMap<String, String>,
db: &DB,
w_id: &str,
permissioned_as: Option<&str>,
) -> Result<Option<String>> {
let Some(permissioned_as) = permissioned_as else {
return Ok(None);
};
if let Some(hit) = cache.get(permissioned_as) {
return Ok(Some(hit.clone()));
}
// The memo above holds this to one query per distinct principal per export.
let email =
windmill_common::users::get_email_from_permissioned_as_uncached(permissioned_as, w_id, db)
.await?;
cache.insert(permissioned_as.to_string(), email.clone());
Ok(Some(email))
}
fn is_empty_extra_perms(value: &serde_json::Value) -> bool {
value.as_object().is_some_and(|o| o.is_empty()) || value.is_null()
}
pub fn is_none_or_false(val: &Option<bool>) -> bool {
match val {
Some(val) => !val,
None => true,
}
}
/// A fork's git-sync export rewrites each trigger's `mode` (and each schedule's
/// `enabled`) to the *parent* workspace's value, instead of emitting the fork's
/// own (clone-disabled / locally-toggled) state. This keeps the fork's synced
/// file byte-identical to the parent on the operational-state field, so a
/// normal-git PR merge has nothing to resolve — no dropped `mode:` line, no
/// flipped parent trigger. Fork-only paths (absent from the parent) keep the
/// fork's own value: there's no parent state to defer to, so the trigger lands
/// with whatever the fork creator set. The write half of the same rule lives in
/// `windmill-trigger::handler::workspace_is_fork`.
///
/// Maps trigger `path` → parent `mode` (as the lowercase enum text that matches
/// `TriggerMode`'s serde representation). Empty when not a fork.
#[cfg(any(
feature = "http_trigger",
feature = "websocket",
feature = "postgres_trigger",
feature = "mqtt_trigger",
feature = "amqp_trigger",
feature = "native_trigger",
all(
feature = "enterprise",
any(
feature = "kafka",
feature = "sqs_trigger",
feature = "gcp_trigger",
feature = "azure_trigger",
feature = "nats",
feature = "smtp",
),
feature = "private"
)
))]
async fn fork_parent_trigger_modes(
db: &DB,
table_name: &str,
parent_workspace_id: Option<&str>,
) -> Result<HashMap<String, String>> {
let Some(parent) = parent_workspace_id else {
return Ok(HashMap::new());
};
// Read the parent's rows on the non-RLS pool (like `workspace_is_fork`): the
// substitution must be complete regardless of the exporter's folder perms,
// otherwise a parent path the exporter can't read would fall back to the
// fork's own value and silently re-introduce the divergence we're fixing.
// No leak: only values for paths the fork already has (it's a clone) are used.
// SAFETY: `table_name` is a compile-time `TriggerCrud::TABLE_NAME` constant.
let rows: Vec<(String, String)> = sqlx::query_as(&format!(
"SELECT path, mode::text FROM {} WHERE workspace_id = $1",
table_name
))
.bind(parent)
.fetch_all(db)
.await?;
Ok(rows.into_iter().collect())
}
/// Build the `{ "mode": <parent value> }` override for a single trigger, or
/// `None` (keep the fork's own value) when the path is fork-only.
#[cfg(any(
feature = "http_trigger",
feature = "websocket",
feature = "postgres_trigger",
feature = "mqtt_trigger",
feature = "amqp_trigger",
feature = "native_trigger",
all(
feature = "enterprise",
any(
feature = "kafka",
feature = "sqs_trigger",
feature = "gcp_trigger",
feature = "azure_trigger",
feature = "nats",
feature = "smtp",
),
feature = "private"
)
))]
fn trigger_mode_override(
parent_modes: &HashMap<String, String>,
path: &str,
) -> Option<serde_json::Map<String, Value>> {
parent_modes.get(path).map(|mode| {
let mut o = serde_json::Map::new();
o.insert("mode".to_string(), Value::String(mode.clone()));
o
})
}
/// Schedule analog of [`fork_parent_trigger_modes`]: maps schedule `path` →
/// parent `enabled`. Empty when not a fork.
async fn fork_parent_schedule_enabled(
db: &DB,
parent_workspace_id: Option<&str>,
) -> Result<HashMap<String, bool>> {
let Some(parent) = parent_workspace_id else {
return Ok(HashMap::new());
};
// Non-RLS pool, same rationale as `fork_parent_trigger_modes`.
let rows: Vec<(String, bool)> =
sqlx::query_as("SELECT path, enabled FROM schedule WHERE workspace_id = $1")
.bind(parent)
.fetch_all(db)
.await?;
Ok(rows.into_iter().collect())
}
enum ArchiveImpl {
#[cfg(feature = "zip")]
Zip(async_zip::tokio::write::ZipFileWriter<tokio::fs::File>),
Tar(tokio_tar::Builder<File>),
}
/// Entry paths come from item paths stored in the workspace; an absolute or
/// drive-prefixed path, or a parent segment, would make extraction write outside
/// the target directory. Win32 strips trailing dots and spaces from a segment,
/// so `.. ` resolves to `..` there: any segment made only of those is refused.
/// A colon matters only in the first segment (a drive prefix); later segments,
/// such as a data table name, may carry one.
fn check_archive_entry_path(path: &str) -> Result<()> {
let is_dot_segment = |seg: &str| !seg.is_empty() && seg.chars().all(|c| c == '.' || c == ' ');
let first = path.split(['/', '\\']).next().unwrap_or_default();
if path.starts_with(['/', '\\'])
|| first.contains(':')
|| path.split(['/', '\\']).any(is_dot_segment)
{
return Err(Error::internal_err(format!(
"refusing to write archive entry with path traversal: {path}"
)));
}
Ok(())
}
impl ArchiveImpl {
async fn write_to_archive(&mut self, content: &str, path: &str) -> Result<()> {
check_archive_entry_path(path)?;
match self {
ArchiveImpl::Tar(t) => {
let bytes = content.as_bytes();
let mut header = tokio_tar::Header::new_gnu();
header.set_size(bytes.len() as u64);
header.set_mtime(0);
header.set_uid(0);
header.set_gid(0);
header.set_mode(0o777);
header.set_cksum();
t.append_data(&mut header, path, bytes).await?;
}
#[cfg(feature = "zip")]
ArchiveImpl::Zip(z) => {
let header =
async_zip::ZipEntryBuilder::new(path.into(), async_zip::Compression::Deflate)
.last_modification_date(Default::default())
.unix_permissions(0o777)
.build();
z.write_entry_whole(header, content.as_bytes())
.await
.map_err(to_anyhow)?;
}
}
Ok(())
}
async fn finish(self) -> Result<()> {
match self {
ArchiveImpl::Tar(t) => t.into_inner().await?,
#[cfg(feature = "zip")]
ArchiveImpl::Zip(z) => z.close().await.map_err(to_anyhow)?.into_inner(),
}
.sync_all()
.await?;
Ok(())
}
}
#[derive(Deserialize)]
pub(crate) struct ArchiveQueryParams {
archive_type: Option<String>,
plain_secret: Option<bool>,
plain_secrets: Option<bool>,
skip_secrets: Option<bool>,
skip_variables: Option<bool>,
skip_resources: Option<bool>,
skip_resource_types: Option<bool>,
include_schedules: Option<bool>,
include_triggers: Option<bool>,
include_users: Option<bool>,
include_groups: Option<bool>,
include_settings: Option<bool>,
/// Opt-in, with `settings_version=v2`: state the data tables' roles in settings.json. Their
/// tenants name this workspace's users and groups, and their roles this instance's catalog.
include_datatable_permissions: Option<bool>,
include_key: Option<bool>,
include_workspace_dependencies: Option<bool>,
/// Default `false`: data table migrations ship in the tarball unless the repo
/// opted out of the `datatablemigration` object type.
skip_datatable_migrations: Option<bool>,
default_ts: Option<String>,
/// Settings format version: "v1" (default) returns legacy flat format, "v2" returns grouped format
settings_version: Option<String>,
/// Opt-in: include `extra_perms` on script / flow / app / variable rows. Default `false`
/// so cross-workspace tarball imports do not carry over ACLs referring to
/// identities that may not exist in the target workspace. `wmill sync pull`
/// passes `true` to surface ACLs in the git-tracked yaml.
preserve_extra_perms: Option<bool>,
/// The caller's `wmill.yaml` `syncBehavior`. From v1 the CLI strips the on-behalf-of
/// address from what it writes and keeps only a `has_on_behalf_of` marker, so the
/// tarball emits that marker directly and skips resolving any address at all.
sync_behavior_version: Option<String>,
}
/// How to handle `extra_perms` in the serialized output.
///
/// * `Drop` — strip the field unconditionally (legacy behavior for
/// types that have never carried ACLs in source).
/// * `KeepEvenEmpty` — always keep the field, even when `{}`. Matches the
/// pre-existing serialization for folders and groups so
/// no customer sees a one-time noisy diff on upgrade.
/// * `KeepIfNonEmpty` — keep when there is at least one entry, drop when `{}`
/// or null. New surface for script / flow / app / variable,
/// which never carried ACLs in source before this change.
#[derive(Clone, Copy)]
pub enum ExtraPermsBehavior {
Drop,
KeepEvenEmpty,
KeepIfNonEmpty,
}
#[inline]
pub fn to_string_without_metadata<T>(
value: &T,
extra_perms: ExtraPermsBehavior,
ignore_keys: Option<Vec<&str>>,
) -> Result<String>
where
T: ?Sized + Serialize,
{
to_string_without_metadata_inner(value, extra_perms, ignore_keys, None)
}
/// Like [`to_string_without_metadata`] but additionally lets the caller
/// override top-level keys after stripping. Used for fork trigger/schedule
/// exports, where `mode`/`enabled` is rewritten to the *parent* workspace's
/// value so the fork's synced file is byte-identical to the parent on those
/// fields — a clean 3-way git merge instead of a dropped line. See the write
/// half of the rule in `windmill-trigger::handler::workspace_is_fork`.
#[inline]
pub fn to_string_without_metadata_inner<T>(
value: &T,
extra_perms: ExtraPermsBehavior,
ignore_keys: Option<Vec<&str>>,
overrides: Option<&serde_json::Map<String, Value>>,
) -> Result<String>
where
T: ?Sized + Serialize,
{
let mut value = serde_json::to_value(value).map_err(to_anyhow)?;
value
.as_object_mut()
.map(|obj| {
let keys = [
vec![
"workspace_id",
"path",
"name",
"versions",
"id",
"created_at",
"updated_at",
"created_by",
"updated_by",
"edited_at",
"edited_by",
"permissioned_as",
"on_behalf_of",
"archived",
"error",
"last_server_ping",
"server_id",
"raw_app",
],
ignore_keys.unwrap_or(vec![]),
]
.concat();
for key in keys {
if obj.contains_key(key) {
obj.remove(key);
}
}
if let Some(o2) = obj.get_mut("policy").and_then(|x| x.as_object_mut()) {
o2.remove("on_behalf_of");
o2.remove("on_behalf_of_email");
}
if obj.contains_key("extra_perms") {
let is_empty_extra_perms = obj
.get("extra_perms")
.map(|v| v.as_object().is_some_and(|o| o.is_empty()) || v.is_null())
.unwrap_or(true);
let drop = match extra_perms {
ExtraPermsBehavior::Drop => true,
ExtraPermsBehavior::KeepEvenEmpty => false,
ExtraPermsBehavior::KeepIfNonEmpty => is_empty_extra_perms,
};
if drop {
obj.remove("extra_perms");
}
}
if obj
.get("default_permissioned_as")
.and_then(|v| v.as_array())
.is_some_and(|a| a.is_empty())
{
obj.remove("default_permissioned_as");
}
if let Some(overrides) = overrides {
for (k, v) in overrides {
obj.insert(k.clone(), v.clone());
}
}
serde_json::to_string_pretty(&obj).ok()
})
.flatten()
.ok_or_else(|| Error::BadRequest("Impossible to serialize value".to_string()))
}
#[derive(Serialize)]
struct SimplifiedUser {
username: String,
role: String,
disabled: bool,
email: String,
}
#[derive(Serialize)]
struct SimplifiedGroup {
name: String,
#[serde(skip_serializing_if = "Option::is_none")]
summary: Option<String>,
members: Vec<String>,
admins: Vec<String>,
}
// V1 format: Legacy flat format for backward compatibility (matches main branch exactly)
#[derive(Serialize)]
struct SimplifiedSettingsLegacy {
auto_invite_enabled: bool,
auto_invite_as: String,
auto_invite_mode: String,
#[serde(skip_serializing_if = "Option::is_none")]
webhook: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
error_handler: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
error_handler_extra_args: Option<Value>,
error_handler_muted_on_cancel: bool,
#[serde(skip_serializing_if = "Option::is_none")]
ai_config: Option<serde_json::Value>,
#[serde(skip_serializing_if = "Option::is_none")]
large_file_storage: Option<Value>,
#[serde(skip_serializing_if = "Option::is_none")]
git_sync: Option<Value>,
#[serde(skip_serializing_if = "Option::is_none")]
default_app: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
default_scripts: Option<Value>,
name: String,
#[serde(skip_serializing_if = "Option::is_none")]
mute_critical_alerts: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
color: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
operator_settings: Option<serde_json::Value>,
#[serde(skip_serializing_if = "Option::is_none")]
datatable: Option<Value>,
#[serde(skip_serializing_if = "Option::is_none")]
slack_team_id: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
slack_name: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
slack_command_script: Option<String>,
}
// Internal struct for querying database
#[derive(sqlx::FromRow)]
struct SettingsRow {
auto_invite: Option<Value>,
webhook: Option<String>,
error_handler: Option<Value>,
ai_config: Option<serde_json::Value>,
large_file_storage: Option<Value>,
git_sync: Option<Value>,
default_app: Option<String>,
default_scripts: Option<Value>,
name: Option<String>,
mute_critical_alerts: Option<bool>,
color: Option<String>,
operator_settings: Option<serde_json::Value>,
datatable: Option<Value>,
slack_team_id: Option<String>,
slack_name: Option<String>,
slack_command_script: Option<String>,
}
pub(crate) async fn tarball_workspace(
authed: ApiAuthed,
Extension(user_db): Extension<UserDB>,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
Query(ArchiveQueryParams {
archive_type,
plain_secret,
plain_secrets,
skip_resources,
skip_resource_types,
skip_secrets,
skip_variables,
include_schedules,
include_triggers,
include_users,
include_groups,
include_settings,
include_datatable_permissions,
include_key,
include_workspace_dependencies,
skip_datatable_migrations,
default_ts,
settings_version,
preserve_extra_perms,
sync_behavior_version,
}): Query<ArchiveQueryParams>,
) -> Result<([(HeaderName, String); 2], impl IntoResponse)> {
tracing::info!(
"tarball_workspace called for workspace {}: include_workspace_dependencies={:?}, skip_variables={:?}, skip_resources={:?}",
w_id,
include_workspace_dependencies,
skip_variables,
skip_resources
);
// The workspace key decrypts every secret offline, so it takes an admin *and* an
// unscoped token. Checked before the item scopes below so that a scoped token
// asking for the key is told about the key rather than about a scope no token
// holding the key would need anyway.
if include_key.unwrap_or(false) {
require_admin(authed.is_admin, &authed.username)?;
windmill_api_auth::forbid_scoped_token_workspace_key(&authed)?;
}
// settings.json carries the admin-managed integration config that `get_settings`
// is admin-only for (ai_config, the webhook URL, git_sync, handler extra_args),
// so it takes the same check. Not a per-field redaction: fields silently dropped
// from settings.json come back as null on the next `wmill sync push`.
if include_settings.unwrap_or(false) && !authed.is_admin {
return Err(Error::PermissionDenied(
"include_settings requires workspace admin".to_string(),
));
}
// The route is gated by workspaces:read, but the tarball also carries the item
// values that the per-item routes gate on their own domain (get_resource_value,
// get_variable). A whole-workspace export cannot be confined to a path, so it
// takes the unrestricted domain scope: a path-scoped token has to skip that kind.
// No-op for unscoped tokens.
if !skip_resources.unwrap_or(false) {
check_scopes(&authed, || "resources:read".to_string())?;
}
if !skip_variables.unwrap_or(false) {
check_scopes(&authed, || "variables:read".to_string())?;
}
// Opt-in behavior for surfacing per-resource ACLs on script/flow/app/variable rows.
// Folder and group rows have always carried `extra_perms` in source and
// continue to do so unconditionally (`KeepEvenEmpty`) so existing
// customer git repos see no one-time noisy diff.
let new_kinds_extra_perms = if preserve_extra_perms.unwrap_or(false) {
ExtraPermsBehavior::KeepIfNonEmpty
} else {
ExtraPermsBehavior::Drop
};
// Resolve workspace dependencies on the pool *before* opening the RLS
// transaction: fetching them mid-transaction would hold a second
// simultaneous connection while `tx` is still checked out.
let workspace_dependencies = if include_workspace_dependencies.unwrap_or(false)
&& require_admin(authed.is_admin, &authed.username).is_ok()
{
Some(WorkspaceDependencies::list(&w_id, &db).await?)
} else {
None
};
let tmp_dir = TempDir::new_in(&*WINDMILL_DIR)?;
let name = match archive_type.as_deref() {
Some("tar") | None => Ok(format!("windmill-{w_id}.tar")),
Some("zip") => Ok(format!("windmill-{w_id}.zip")),
Some(t) => Err(Error::BadRequest(format!("Invalid Archive Type {t}"))),
}?;
let file_path = tmp_dir.path().join(&name);
let mut archive = match archive_type.as_deref() {
Some("tar") | None => {
let file = File::create(&file_path).await?;
Ok(ArchiveImpl::Tar(tokio_tar::Builder::new(file)))
}
#[cfg(feature = "zip")]
Some("zip") => {
let file = tokio::fs::File::create(&file_path).await?;
Ok(ArchiveImpl::Zip(
async_zip::tokio::write::ZipFileWriter::with_tokio(file),
))
}
Some(t) => Err(Error::BadRequest(format!("Invalid Archive Type {t}"))),
}?;
let export_plain_secrets = plain_secret.or(plain_secrets).unwrap_or(false)
&& !skip_secrets.unwrap_or(false)
&& !skip_variables.unwrap_or(false);
// Record what the export is about to disclose, once nothing left can reject the
// request: an entry written before the gates above would claim a disclosure that
// a 403 or an invalid archive type then prevented. On the pool and before the RLS
// transaction opens — `tx` is never committed, so a write through it is rolled
// back with it, and holding both at once would take two connections.
if export_plain_secrets {
// Exporting decrypted secrets in bulk is the same capability as a per-item
// secret read, so record it for parity with variables.decrypt_secret.
windmill_audit::audit_oss::audit_log(
&db,
&authed,
"variables.decrypt_secret",
windmill_audit::ActionKind::Execute,
&w_id,
Some("workspace_tarball_export"),
None,
)
.await?;
}
if include_key.unwrap_or(false) {
windmill_audit::audit_oss::audit_log(
&db,
&authed,
"workspaces.read_encryption_key",
windmill_audit::ActionKind::Execute,
&w_id,
Some("workspace_tarball_export"),
None,
)
.await?;
}
let mut tx = user_db.begin(&authed).await?;
// Source-of-truth for fork-ness: the workspace's parent_workspace_id column.
// The wm-fork-* prefix is a creation-time naming convention that could in
// principle drift (rename, manual SQL); the column is the contract that
// matches what the conflict-warning gates read. The id is also the workspace
// whose trigger `mode` / schedule `enabled` a fork export defers to.
let parent_workspace_id: Option<String> = sqlx::query_scalar::<_, Option<String>>(
"SELECT parent_workspace_id FROM workspace WHERE id = $1",
)
.bind(&w_id)
.fetch_optional(&mut *tx)
.await?
.flatten();
{
let folders = sqlx::query_as::<_, Folder>("SELECT name, workspace_id, display_name, owners, extra_perms, summary, edited_at, created_by, default_permissioned_as, labels FROM folder WHERE workspace_id = $1")
.bind(&w_id)
.fetch_all(&mut *tx)
.await?;
for folder in folders {
archive
.write_to_archive(
&to_string_without_metadata(&folder, ExtraPermsBehavior::KeepEvenEmpty, None)
.unwrap(),
&format!("f/{}/folder.meta.json", folder.name),
)
.await?;
}
}
// From v1 the CLI never writes the address, so resolving it would be pure waste on the
// default sync path; emit the marker it actually keeps instead.
let obo_marker_only = parse_sync_behavior_version(sync_behavior_version.as_deref()) >= 1;
let mut obo_cache: HashMap<String, String> = HashMap::new();
{
let scripts = sqlx::query_as::<_, Script<ScriptRunnableSettingsHandle>>(&format!(
"SELECT {} FROM script as o WHERE workspace_id = $1 AND archived = false
AND created_at = (select max(created_at) from script where path = o.path AND \
workspace_id = $1)",
windmill_common::scripts::SCRIPT_COLUMNS,
))
.bind(&w_id)
.fetch_all(&mut *tx)
.await?;
for script in scripts {
let script =
windmill_common::scripts::prefetch_cached_script_without_email(script, &db).await?;
let ext = match script.language {
ScriptLang::Python3 => "py",
ScriptLang::Deno => {
if default_ts.as_ref().is_some_and(|x| x == "bun") {
"deno.ts"
} else {
"ts"
}
}
ScriptLang::Go => "go",
ScriptLang::Bash => "sh",
ScriptLang::Powershell => "ps1",
ScriptLang::Postgresql => "pg.sql",
ScriptLang::Mysql => "my.sql",
ScriptLang::Bigquery => "bq.sql",
ScriptLang::Snowflake => "sf.sql",
ScriptLang::Mssql => "ms.sql",
ScriptLang::DuckDb => "duckdb.sql",
ScriptLang::Graphql => "gql",
ScriptLang::Nativets => "fetch.ts",
ScriptLang::Bun | ScriptLang::Bunnative => {
if default_ts.as_ref().is_some_and(|x| x == "bun") {
"ts"
} else {
"bun.ts"
}
}
ScriptLang::Php => "php",
ScriptLang::Rust => "rs",
ScriptLang::Ansible => "playbook.yml",
ScriptLang::CSharp => "cs",
ScriptLang::Nu => "nu",
ScriptLang::OracleDB => "odb.sql",
ScriptLang::Java => "java",
ScriptLang::Ruby => "rb",
ScriptLang::Rlang => "r",
// A dbt script's content path is not `<path>.<ext>` — it is the
// descriptor inside the project folder, built below.
ScriptLang::Dbt => "",
// for related places search: ADD_NEW_LANG
};
if script.language == ScriptLang::Dbt {
// The descriptor belongs inside the project it configures, and
// is OPTIONAL: an unmodified dbt project is already a complete
// script. Writing an empty one would put a Windmill file in a
// directory that had none, so a project that names no descriptor
// exports without one.
if !script.content.trim().is_empty() {
archive
.write_to_archive(
&script.content,
&format!("{}__dbt/wm_dbt.yaml", script.path),
)
.await?;
}
} else {
archive
.write_to_archive(&script.content, &format!("{}.{}", script.path, ext))
.await?;
}
let metadata = ScriptMetadata {
summary: script.summary,
description: script.description,
schema: script.schema,
kind: script.kind.to_string(),
lock: script.lock,
envs: script.envs,
concurrency_settings: script.runnable_settings.concurrency_settings,
debouncing_settings: script.runnable_settings.debouncing_settings,
cache_ttl: script.cache_ttl,
cache_ignore_s3_path: script.cache_ignore_s3_path,
dedicated_worker: script.dedicated_worker,
ws_error_handler_muted: script.ws_error_handler_muted,
priority: script.priority,
tag: script.tag,
timeout: script.timeout,
delete_after_secs: script.delete_after_secs,
restart_unless_cancelled: script.restart_unless_cancelled,
visible_to_runner_only: script.visible_to_runner_only,
auto_kind: script.auto_kind,
codebase: script.codebase,
has_preprocessor: script.has_preprocessor,
on_behalf_of_email: if obo_marker_only {
None
} else {
derive_email(&mut obo_cache, &db, &w_id, script.on_behalf_of.as_deref()).await?
},
has_on_behalf_of: (obo_marker_only && script.on_behalf_of.is_some())
.then_some(true),
modules: script.modules,
labels: script.labels,
job_token_scopes: script.job_token_scopes,
// Same opt-in contract as flow/app: the tarball only surfaces
// ACLs when `?preserve_extra_perms=true`. Passing `Null` lets the
// `is_empty_extra_perms` skip-serializer drop the field entirely.
extra_perms: if matches!(new_kinds_extra_perms, ExtraPermsBehavior::KeepIfNonEmpty)
{
script.extra_perms
} else {
serde_json::Value::Null
},
};
let metadata_str = serde_json::to_string_pretty(&metadata).unwrap();
archive
.write_to_archive(&metadata_str, &format!("{}.script.json", script.path))
.await?;
}
}
if !skip_resources.unwrap_or(false) {
let resources = sqlx::query_as!(
Resource,
"SELECT workspace_id, path, value, description, resource_type, extra_perms, created_by, edited_at, labels FROM resource WHERE workspace_id = $1 AND resource_type <> ALL($2)",
&w_id,
&windmill_store::resources::INTERNAL_RESOURCE_TYPES.map(|t| t.to_string())[..]
)
.fetch_all(&mut *tx)
.await?;
for mut resource in resources {
// An agent's identity belongs to this workspace, and whoever writes it elsewhere gets
// their own, as `to_string_without_metadata` leaves out a flow's.
if resource.resource_type == "ai_agent" {
if let Some(obj) = resource.value.as_mut().and_then(|v| v.as_object_mut()) {
obj.remove("on_behalf_of");
}
}
let resource_str =
&to_string_without_metadata(&resource, ExtraPermsBehavior::Drop, None).unwrap();
archive
.write_to_archive(&resource_str, &format!("{}.resource.json", resource.path))
.await?;
}
}
if !skip_resource_types.unwrap_or(false) {
let resource_types = sqlx::query_as!(
ResourceType,
"SELECT workspace_id, name, schema, description, created_by, edited_at, format_extension, is_fileset, display_name FROM resource_type WHERE workspace_id = $1",
&w_id
)
.fetch_all(&mut *tx)
.await?;
for resource_type in resource_types {
let resource_str =
&to_string_without_metadata(&resource_type, ExtraPermsBehavior::Drop, None)
.unwrap();
archive
.write_to_archive(
&resource_str,
&format!("{}.resource-type.json", resource_type.name),
)
.await?;
}
}
{
let flows = sqlx::query_as::<_, Flow>(
"SELECT flow.workspace_id, flow.path, flow.summary, flow.description, flow.archived, flow.extra_perms, flow.dedicated_worker, flow.tag, flow.ws_error_handler_muted, flow.timeout, flow.visible_to_runner_only, flow.on_behalf_of, flow.labels, flow.job_token_scopes, flow_version.schema, flow_version.value, flow_version.created_at as edited_at, flow_version.created_by as edited_by
FROM flow
LEFT JOIN flow_version ON flow_version.id = flow.versions[array_upper(flow.versions, 1)]
WHERE flow.workspace_id = $1 AND flow.archived = false",
)
.bind(&w_id)
.fetch_all(&mut *tx)
.await?;
for mut flow in flows {
let flow_marker = obo_marker_only && flow.on_behalf_of.is_some();
flow.on_behalf_of_email = if obo_marker_only {
None
} else {
derive_email(&mut obo_cache, &db, &w_id, flow.on_behalf_of.as_deref()).await?
};
let mut overrides = serde_json::Map::new();
if flow_marker {
overrides.insert("has_on_behalf_of".to_string(), Value::Bool(true));
}
let flow_str = &to_string_without_metadata_inner(
&flow,
new_kinds_extra_perms,
None,
(!overrides.is_empty()).then_some(&overrides),
)
.unwrap();
archive
.write_to_archive(&flow_str, &format!("{}.flow.json", flow.path))
.await?;
}
}
if !skip_variables.unwrap_or(false) {
// `account` is left out: the variable endpoints ignore it, so a pushed spec carrying it
// would never match the variable it created and every sync would report it changed.
let variables =
sqlx::query_as::<_, ExportableListableVariable>(if !skip_secrets.unwrap_or(false) {
"SELECT workspace_id, path, value::text AS value, is_secret, description, extra_perms, NULL::int AS account, is_oauth, expires_at, value_valid_until, labels FROM variable WHERE workspace_id = $1 AND expires_at IS NULL"
} else {
"SELECT workspace_id, path, value::text AS value, is_secret, description, extra_perms, NULL::int AS account, is_oauth, expires_at, value_valid_until, labels FROM variable WHERE workspace_id = $1 AND is_secret = false AND expires_at IS NULL"
})
.bind(&w_id)
.fetch_all(&mut *tx)
.await?;
let mc = build_crypt(&db, &w_id).await?;
for mut var in variables {
if plain_secret.or(plain_secrets).unwrap_or(false)
&& var.value.is_some()
&& var.is_secret
{
var.value = Some(decrypt(&mc, var.value.unwrap()).map_err(|e| {
Error::internal_err(format!("Error decrypting variable {}: {}", var.path, e))
})?);
}
let var_str = &to_string_without_metadata(&var, new_kinds_extra_perms, None).unwrap();
archive
.write_to_archive(&var_str, &format!("{}.variable.json", var.path))
.await?;
}
}
{
let apps = sqlx::query_as::<_, AppWithLastVersion>(
"SELECT app.id, app.path, app.summary, app.versions, app.policy, app.custom_path,
app.extra_perms, app_version.value,
app_version.created_at, app_version.created_by, app_version.raw_app, app.labels from app, app_version
WHERE app.workspace_id = $1 AND app_version.id = app.versions[array_upper(app.versions, 1)]",
)
.bind(&w_id)
.fetch_all(&mut *tx)
.await?;
for app in apps {
let app_str = &to_string_without_metadata(&app, new_kinds_extra_perms, None).unwrap();
let kind = if app.raw_app { "raw_app" } else { "app" };
archive
.write_to_archive(&app_str, &format!("{}.{}.json", app.path, kind))
.await?;
}
}
if let Some(workspace_dependencies) = workspace_dependencies {
tracing::info!("Including workspace dependencies in tarball export");
tracing::info!(
"Found {} workspace dependencies",
workspace_dependencies.len()
);
for dep in workspace_dependencies {
// let dep_str = &to_string_without_metadata(&dep, ExtraPermsBehavior::Drop, None).unwrap();
let filename = WorkspaceDependencies::to_path(&dep.name, dep.language)?;
tracing::info!(
"Adding workspace dependency: name={:?}, language={:?}, filename={}",
dep.name,
dep.language,
filename
);
archive.write_to_archive(&dep.content, &filename).await?;
}
} else {
tracing::info!(
"Skipping workspace dependencies: include_workspace_dependencies={:?}",
include_workspace_dependencies
);
}
if include_schedules.unwrap_or(false) {
// Managed ducklake-maintenance schedules are excluded: they are
// derived from the workspace ducklake settings (and admins bypass the
// RLS that hides them), so exporting them would drag unsyncable rows
// into git.
let schedules = sqlx::query_as::<_, Schedule>(
"SELECT workspace_id, path, edited_by, edited_at, schedule, timezone, enabled, script_path, is_flow, args, extra_perms, email, permissioned_as, error, on_failure, on_failure_times, on_failure_exact, on_failure_extra_args, on_recovery, on_recovery_times, on_recovery_extra_args, on_success, on_success_extra_args, ws_error_handler_muted, retry, no_flow_overlap, summary, description, tag, paused_until, cron_version, dynamic_skip, labels FROM schedule
WHERE workspace_id = $1 AND NOT starts_with(path, $2)",
)
.bind(&w_id)
.bind(windmill_common::workspaces::DUCKLAKE_MAINTENANCE_PATH_PREFIX)
.fetch_all(&mut *tx)
.await?;
// For a fork, defer each schedule's `enabled` to the parent so the
// synced file matches the parent and the merge doesn't flip it.
let parent_enabled =
fork_parent_schedule_enabled(&db, parent_workspace_id.as_deref()).await?;
for schedule in schedules {
let enabled_override = parent_enabled.get(&schedule.path).map(|enabled| {
let mut o = serde_json::Map::new();
o.insert("enabled".to_string(), Value::Bool(*enabled));
o
});
let app_str = &to_string_without_metadata_inner(
&schedule,
ExtraPermsBehavior::Drop,
None,
enabled_override.as_ref(),
)
.unwrap();
archive
.write_to_archive(&app_str, &format!("{}.schedule.json", schedule.path))
.await?;
}
}
if include_triggers.unwrap_or(false) {
#[cfg(feature = "http_trigger")]
{
use crate::triggers::http::HttpTrigger;
let handler = HttpTrigger;
let http_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?;
let parent_modes = fork_parent_trigger_modes(
&db,
<HttpTrigger as TriggerCrud>::TABLE_NAME,
parent_workspace_id.as_deref(),
)
.await?;
for trigger in http_triggers {
let mode_override = trigger_mode_override(&parent_modes, &trigger.base.path);
let trigger_str = &to_string_without_metadata_inner(
&trigger,
ExtraPermsBehavior::Drop,
None,
mode_override.as_ref(),
)
.unwrap();
archive
.write_to_archive(
&trigger_str,
&format!("{}.http_trigger.json", trigger.base.path),
)
.await?;
}
}
#[cfg(feature = "websocket")]
{
use crate::triggers::websocket::WebsocketTrigger;
let handler = WebsocketTrigger;
let websocket_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?;
let parent_modes = fork_parent_trigger_modes(
&db,
<WebsocketTrigger as TriggerCrud>::TABLE_NAME,
parent_workspace_id.as_deref(),
)
.await?;
for trigger in websocket_triggers {
let mode_override = trigger_mode_override(&parent_modes, &trigger.base.path);
let trigger_str = &to_string_without_metadata_inner(
&trigger,
ExtraPermsBehavior::Drop,
None,
mode_override.as_ref(),
)
.unwrap();
archive
.write_to_archive(
&trigger_str,
&format!("{}.websocket_trigger.json", trigger.base.path),
)
.await?;
}
}
#[cfg(all(feature = "enterprise", feature = "kafka", feature = "private"))]
{
use crate::triggers::kafka::KafkaTrigger;
let handler = KafkaTrigger;
let kafka_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?;
let parent_modes = fork_parent_trigger_modes(
&db,
<KafkaTrigger as TriggerCrud>::TABLE_NAME,
parent_workspace_id.as_deref(),
)
.await?;
for trigger in kafka_triggers {
let mode_override = trigger_mode_override(&parent_modes, &trigger.base.path);
let trigger_str = &to_string_without_metadata_inner(
&trigger,
ExtraPermsBehavior::Drop,
None,
mode_override.as_ref(),
)
.unwrap();
archive
.write_to_archive(
&trigger_str,
&format!("{}.kafka_trigger.json", trigger.base.path),
)
.await?;
}
}
#[cfg(all(feature = "enterprise", feature = "sqs_trigger", feature = "private"))]
{
use crate::triggers::sqs::SqsTrigger;
let handler = SqsTrigger;
let sqs_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?;
let parent_modes = fork_parent_trigger_modes(
&db,
<SqsTrigger as TriggerCrud>::TABLE_NAME,
parent_workspace_id.as_deref(),
)
.await?;
for trigger in sqs_triggers {
let mode_override = trigger_mode_override(&parent_modes, &trigger.base.path);
let trigger_str = &to_string_without_metadata_inner(
&trigger,
ExtraPermsBehavior::Drop,
None,
mode_override.as_ref(),
)
.unwrap();
archive
.write_to_archive(
&trigger_str,
&format!("{}.sqs_trigger.json", trigger.base.path),
)
.await?;
}
}
#[cfg(all(feature = "enterprise", feature = "gcp_trigger", feature = "private"))]
{
use crate::triggers::gcp::GcpTrigger;
let handler = GcpTrigger;
let gcp_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?;
let parent_modes = fork_parent_trigger_modes(
&db,
<GcpTrigger as TriggerCrud>::TABLE_NAME,
parent_workspace_id.as_deref(),
)
.await?;
for trigger in gcp_triggers {
let mode_override = trigger_mode_override(&parent_modes, &trigger.base.path);
let trigger_str = &to_string_without_metadata_inner(
&trigger,
ExtraPermsBehavior::Drop,
None,
mode_override.as_ref(),
)
.unwrap();
archive
.write_to_archive(
&trigger_str,
&format!("{}.gcp_trigger.json", trigger.base.path),
)
.await?;
}
}
#[cfg(all(feature = "enterprise", feature = "azure_trigger", feature = "private"))]
{
use crate::triggers::azure::AzureTrigger;
let handler = AzureTrigger;
let azure_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?;
let parent_modes = fork_parent_trigger_modes(
&db,
<AzureTrigger as TriggerCrud>::TABLE_NAME,
parent_workspace_id.as_deref(),
)
.await?;
for trigger in azure_triggers {
let mode_override = trigger_mode_override(&parent_modes, &trigger.base.path);
let trigger_str = &to_string_without_metadata_inner(
&trigger,
ExtraPermsBehavior::Drop,
None,
mode_override.as_ref(),
)
.unwrap();
archive
.write_to_archive(
&trigger_str,
&format!("{}.azure_trigger.json", trigger.base.path),
)
.await?;
}
}
#[cfg(all(feature = "enterprise", feature = "nats", feature = "private"))]
{
use crate::triggers::nats::NatsTrigger;
let handler = NatsTrigger;
let nats_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?;
let parent_modes = fork_parent_trigger_modes(
&db,
<NatsTrigger as TriggerCrud>::TABLE_NAME,
parent_workspace_id.as_deref(),
)
.await?;
for trigger in nats_triggers {
let mode_override = trigger_mode_override(&parent_modes, &trigger.base.path);
let trigger_str: &String = &to_string_without_metadata_inner(
&trigger,
ExtraPermsBehavior::Drop,
None,
mode_override.as_ref(),
)
.unwrap();
archive
.write_to_archive(
&trigger_str,
&format!("{}.nats_trigger.json", trigger.base.path),
)
.await?;
}
}
#[cfg(feature = "postgres_trigger")]
{
use crate::triggers::postgres::PostgresTrigger;
let handler = PostgresTrigger;
let postgres_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?;
let parent_modes = fork_parent_trigger_modes(
&db,
<PostgresTrigger as TriggerCrud>::TABLE_NAME,
parent_workspace_id.as_deref(),
)
.await?;
for trigger in postgres_triggers {
let mode_override = trigger_mode_override(&parent_modes, &trigger.base.path);
let trigger_str = &to_string_without_metadata_inner(
&trigger,
ExtraPermsBehavior::Drop,
None,
mode_override.as_ref(),
)
.unwrap();
archive
.write_to_archive(
&trigger_str,
&format!("{}.postgres_trigger.json", trigger.base.path),
)
.await?;
}
}
#[cfg(feature = "mqtt_trigger")]
{
use crate::triggers::mqtt::MqttTrigger;
let handler = MqttTrigger;
let mqtt_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?;
let parent_modes = fork_parent_trigger_modes(
&db,
<MqttTrigger as TriggerCrud>::TABLE_NAME,
parent_workspace_id.as_deref(),
)
.await?;
for trigger in mqtt_triggers {
let mode_override = trigger_mode_override(&parent_modes, &trigger.base.path);
let trigger_str = &to_string_without_metadata_inner(
&trigger,
ExtraPermsBehavior::Drop,
None,
mode_override.as_ref(),
)
.unwrap();
archive
.write_to_archive(
&trigger_str,
&format!("{}.mqtt_trigger.json", trigger.base.path),
)
.await?;
}
}
#[cfg(feature = "amqp_trigger")]
{
use crate::triggers::amqp::AmqpTrigger;
let handler = AmqpTrigger;
let amqp_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?;
let parent_modes = fork_parent_trigger_modes(
&db,
<AmqpTrigger as TriggerCrud>::TABLE_NAME,
parent_workspace_id.as_deref(),
)
.await?;
for trigger in amqp_triggers {
let mode_override = trigger_mode_override(&parent_modes, &trigger.base.path);
let trigger_str = &to_string_without_metadata_inner(
&trigger,
ExtraPermsBehavior::Drop,
None,
mode_override.as_ref(),
)
.unwrap();
archive
.write_to_archive(
&trigger_str,
&format!("{}.amqp_trigger.json", trigger.base.path),
)
.await?;
}
}
#[cfg(all(feature = "enterprise", feature = "smtp", feature = "private"))]
{
use crate::triggers::email::EmailTrigger;
let handler = EmailTrigger;
let email_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?;
let parent_modes = fork_parent_trigger_modes(
&db,
<EmailTrigger as TriggerCrud>::TABLE_NAME,
parent_workspace_id.as_deref(),
)
.await?;
for trigger in email_triggers {
let mode_override = trigger_mode_override(&parent_modes, &trigger.base.path);
let trigger_str = &to_string_without_metadata_inner(
&trigger,
ExtraPermsBehavior::Drop,
None,
mode_override.as_ref(),
)
.unwrap();
archive
.write_to_archive(
&trigger_str,
&format!("{}.email_trigger.json", trigger.base.path),
)
.await?;
}
}
#[cfg(feature = "native_trigger")]
{
use crate::native_triggers::{list_native_triggers, ServiceName};
use strum::IntoEnumIterator;
for service_name in ServiceName::iter() {
let native_triggers = list_native_triggers(
&mut *tx,
&w_id,
service_name,
None,
None,
None,
None,
windmill_api_auth::ScopePathFilter::AllowAll,
)
.await?;
// Native triggers (Nextcloud, Google Drive, GitHub) are never
// cloned into a fork — a fork only has one if its owner created
// it there, so it's always "fork-only" and keeps its own mode.
// No parent-value substitution applies; we strip the webhook
// token hash, and `enabled`, which is operational state a sync
// deliberately does not carry — whether a trigger is paused
// belongs to the workspace it runs in, not to the code.
let native_ignore_keys = vec!["webhook_token_hash", "enabled"];
for trigger in native_triggers {
let trigger_str = &to_string_without_metadata(
&trigger,
ExtraPermsBehavior::Drop,
Some(native_ignore_keys.clone()),
)
.unwrap();
archive
.write_to_archive(
&trigger_str,
&format!(
"{}.{}.{}.{}_native_trigger.json",
trigger.script_path,
if trigger.is_flow { "flow" } else { "script" },
trigger.external_id,
service_name.as_str()
),
)
.await?;
}
}
}
}
if include_users.unwrap_or(false) {
let users = sqlx::query!(
"SELECT workspace_id, username, email, is_admin, created_at, operator, disabled, role, added_via FROM usr
WHERE workspace_id = $1",
&w_id
)
.fetch_all(&mut *tx)
.await?;
for user in users {
let user = SimplifiedUser {
username: user.username,
role: if user.is_admin {
"admin".to_string()
} else if user.operator {
"operator".to_string()
} else {
"developer".to_string()
},
disabled: user.disabled,
email: user.email,
};
let user_str =
&to_string_without_metadata(&user, ExtraPermsBehavior::Drop, Some(vec!["email"]))
.unwrap();
archive
.write_to_archive(&user_str, &format!("users/{}.user.json", user.email))
.await?;
}
}
if include_groups.unwrap_or(false) {
let groups = sqlx::query!(
r#"SELECT g_.workspace_id, name, summary, extra_perms, array_agg(u2g.usr) filter (where u2g.usr is not null) as members
FROM usr u
JOIN usr_to_group u2g ON u2g.usr = u.username AND u2g.workspace_id = u.workspace_id
RIGHT JOIN group_ g_ ON g_.workspace_id = u.workspace_id AND g_.name = u2g.group_
WHERE g_.workspace_id = $1 AND g_.name != 'all'
GROUP BY g_.workspace_id, name, summary, extra_perms"#,
&w_id
)
.fetch_all(&mut *tx)
.await?;
for group in groups {
let extra_perms: HashMap<String, bool> = serde_json::from_value(group.extra_perms)
.map_err(|e| {
Error::internal_err(format!(
"Error parsing extra_perms for group {}: {}",
group.name, e
))
})?;
tracing::info!("{:?}", extra_perms);
let members = group.members.unwrap_or(vec![]);
let admins: Vec<String> = extra_perms
.iter()
.filter_map(|(k, v)| {
// only consider extra_perms that concern actual members of the group
if members.contains(&k[2..].to_string()) && *v {
Some(k.clone())
} else {
None
}
})
.sorted()
.collect();
let group = SimplifiedGroup {
name: group.name,
summary: group.summary,
members: members
.iter()
.filter_map(|x| {
// remove members that are also admins as they are already in the admins list
let full_name = format!("u/{}", x);
if !admins.contains(&full_name) {
Some(full_name)
} else {
None
}
})
.collect(),
admins,
};
let group_str =
&to_string_without_metadata(&group, ExtraPermsBehavior::KeepEvenEmpty, None)
.unwrap();
archive
.write_to_archive(&group_str, &format!("groups/{}.group.json", group.name))
.await?;
}
}
if include_settings.unwrap_or(false) {
let row = sqlx::query_as::<_, SettingsRow>(
r#"SELECT
auto_invite,
webhook,
error_handler,
ai_config,
large_file_storage,
git_sync,
default_app,
default_scripts,
workspace.name as name,
mute_critical_alerts,
color,
operator_settings,
datatable,
slack_team_id,
slack_name,
slack_command_script
FROM workspace_settings
LEFT JOIN workspace ON workspace.id = workspace_settings.workspace_id
WHERE workspace_id = $1"#,
)
.bind(&w_id)
.fetch_one(&mut *tx)
.await?;
let settings_str = if settings_version.as_deref() == Some("v2") {
let secrets = if export_plain_secrets {
crate::workspace_settings_document::SecretMode::Plain
} else {
crate::workspace_settings_document::SecretMode::Encrypted
};
let mut doc = crate::workspace_settings_document::load_settings_document(
&mut *tx, &db, &w_id, secrets,
)
.await?;
// A repo syncs its rules through protection-rules.yaml, so settings.yaml leaving
// them out keeps one writer for them.
doc.remove("protection_rules");
if !include_datatable_permissions.unwrap_or(false) {
doc.remove("datatable_permissions");
}
serde_json::to_string_pretty(&doc).ok()
} else {
// V1 (legacy) format: convert JSONB to flat fields (matches main branch exactly)
let (auto_invite_enabled, auto_invite_as, auto_invite_mode) =
if let Some(ref ai) = row.auto_invite {
let enabled = ai.get("enabled").and_then(|v| v.as_bool()).unwrap_or(false);
let operator = ai
.get("operator")
.and_then(|v| v.as_bool())
.unwrap_or(false);
let mode = ai.get("mode").and_then(|v| v.as_str()).unwrap_or("invite");
(
enabled,
if operator {
"operator".to_string()
} else {
"developer".to_string()
},
mode.to_string(),
)
} else {
(false, "developer".to_string(), "invite".to_string())
};
let (error_handler, error_handler_extra_args, error_handler_muted_on_cancel) =
if let Some(ref eh) = row.error_handler {
let path = eh.get("path").and_then(|v| v.as_str()).map(String::from);
let extra_args = eh.get("extra_args").cloned();
let muted_on_cancel = eh
.get("muted_on_cancel")
.and_then(|v| v.as_bool())
.unwrap_or(false);
(path, extra_args, muted_on_cancel)
} else {
(None, None, false)
};
let settings = SimplifiedSettingsLegacy {
auto_invite_enabled,
auto_invite_as,
auto_invite_mode,
webhook: row.webhook,
error_handler,
error_handler_extra_args,
error_handler_muted_on_cancel,
ai_config: row.ai_config,
large_file_storage: row.large_file_storage,
git_sync: crate::workspace_settings_document::redact_git_sync(row.git_sync),
default_app: row.default_app,
default_scripts: row.default_scripts,
name: row.name.unwrap_or_default(),
mute_critical_alerts: row.mute_critical_alerts,
color: row.color,
operator_settings: row.operator_settings,
datatable: windmill_common::workspaces::strip_datatable_permissions(row.datatable),
slack_team_id: row.slack_team_id,
slack_name: row.slack_name,
slack_command_script: row.slack_command_script,
};
serde_json::to_value(settings)
.map(|v| serde_json::to_string_pretty(&v).ok())
.ok()
.flatten()
}
.ok_or_else(|| Error::internal_err("Error serializing settings".to_string()))?;
archive
.write_to_archive(&settings_str, "settings.json")
.await?;
}
// Gated in the pre-flight block above: admin plus an unscoped token, audited there.
if include_key.unwrap_or(false) {
let key = sqlx::query_scalar!(
"SELECT key FROM workspace_key WHERE workspace_id = $1",
&w_id
)
.fetch_one(&mut *tx)
.await?;
let key_json = serde_json::to_value(key)
.map(|v| serde_json::to_string_pretty(&v).ok())
.ok()
.flatten()
.ok_or_else(|| Error::internal_err("Error serializing enryption key".to_string()))?;
archive
.write_to_archive(&key_json, "encryption_key.json")
.await?;
}
if !skip_datatable_migrations.unwrap_or(false) {
// Data table migrations live in the `datatable_migrations` table; surface
// them in the export as `migrations/datatable/<datatable>/<version>_<name>`
// .up.sql (and .down.sql when present) so `wmill sync` treats them like any
// other workspace item.
let migrations = sqlx::query!(
"SELECT datatable, timestamp, name, code_up, code_down FROM datatable_migrations \
WHERE workspace_id = $1 ORDER BY datatable, timestamp",
&w_id
)
.fetch_all(&mut *tx)
.await?;
for m in migrations {
let base = format!(
"migrations/datatable/{}/{}_{}",
m.datatable, m.timestamp, m.name
);
archive
.write_to_archive(&m.code_up, &format!("{base}.up.sql"))
.await?;
if let Some(code_down) = m.code_down {
archive
.write_to_archive(&code_down, &format!("{base}.down.sql"))
.await?;
}
}
}
archive.finish().await?;
let file = tokio::fs::File::open(&file_path).await?;
let stream = ReaderStream::new(file);
let body = axum::body::Body::from_stream(stream);
let headers = [
(header::CONTENT_TYPE, "application/x-tar".to_string()),
(
header::CONTENT_DISPOSITION,
format!("attachment; filename=\"{name}\""),
),
];
Ok((headers, body))
}
#[cfg(test)]
mod archive_entry_path_tests {
use super::check_archive_entry_path;
#[test]
fn rejects_traversal_and_absolute_paths() {
for ok in [
"u/admin/app.app.json",
"f/x/a..b.script.json",
"settings.yaml",
"migrations/datatable/foo:bar/20260617120000_name.up.sql",
] {
assert!(
check_archive_entry_path(ok).is_ok(),
"{ok} should be accepted"
);
}
for bad in [
"f/x/../../evil.app.json",
"../evil",
"/etc/evil",
"f\\..\\evil",
"f/x/.. /.. /evil.app.json",
"f/x/.../evil",
"C:/evil",
"\\evil",
] {
assert!(
check_archive_entry_path(bad).is_err(),
"{bad} should be rejected"
);
}
}
}
#[cfg(test)]
mod fork_export_tests {
use super::*;
use serde_json::json;
/// A fork export rewrites `mode` to the parent's value: the serialized file
/// carries the parent's state (`enabled`), not the fork's clone-disabled DB
/// value — so a normal-git merge sees no change on that line.
#[test]
fn override_substitutes_parent_mode() {
let fork_trigger = json!({
"path": "f/triggers/x",
"script_path": "f/scripts/x",
"mode": "disabled", // fork's local (clone-disabled) state
"is_flow": false,
});
let mut overrides = serde_json::Map::new();
overrides.insert("mode".to_string(), Value::String("enabled".to_string()));
let out = to_string_without_metadata_inner(
&fork_trigger,
ExtraPermsBehavior::Drop,
None,
Some(&overrides),
)
.unwrap();
let parsed: Value = serde_json::from_str(&out).unwrap();
assert_eq!(parsed["mode"], json!("enabled"), "parent mode substituted");
// `path` is in the metadata strip list, so it should be removed.
assert!(parsed.get("path").is_none());
}
/// A fork-only trigger (no parent counterpart, so no override) keeps the
/// fork creator's chosen state.
#[test]
fn no_override_keeps_fork_value() {
let fork_only = json!({ "mode": "enabled", "script_path": "f/scripts/x" });
let out =
to_string_without_metadata_inner(&fork_only, ExtraPermsBehavior::Drop, None, None)
.unwrap();
let parsed: Value = serde_json::from_str(&out).unwrap();
assert_eq!(parsed["mode"], json!("enabled"));
}
/// `trigger_mode_override` builds an override only when the parent has the
/// path; fork-only paths return `None` (keep the fork's own value).
#[cfg(feature = "http_trigger")]
#[test]
fn trigger_mode_override_defers_to_parent_or_self() {
let mut parent_modes = HashMap::new();
parent_modes.insert("f/triggers/shared".to_string(), "enabled".to_string());
let shared = trigger_mode_override(&parent_modes, "f/triggers/shared");
assert_eq!(
shared.as_ref().and_then(|o| o.get("mode")),
Some(&Value::String("enabled".to_string())),
);
// Fork-only path: no parent entry → no override → keep fork's own value.
assert!(trigger_mode_override(&parent_modes, "f/triggers/fork_only").is_none());
}
}