From 6f4124e9a030ce790e484dd75917a2c04b889bd3 Mon Sep 17 00:00:00 2001 From: hugocasa Date: Fri, 30 Oct 2026 20:23:27 +0100 Subject: [PATCH] feat: reference existing HashiCorp Vault secrets from secret variables (#12078) * feat: reference existing HashiCorp Vault secrets from secret variables A secret variable can hold `$external:vault://#` instead of a value. The reference is resolved from Vault each time the variable is read, so the value is never stored in Windmill. An instance-level allow-list of path prefixes, optionally per workspace, is checked when a reference is saved and again when it is resolved. Other schemes are parsed and refused as unsupported backends. Adds AppRole as a Vault auth method, an "External reference" toggle in the variable editor and an instance settings section for the connection and the allow-list. Co-Authored-By: Claude Opus 5.5 * chore: point ee-repo-ref at the external secret references branch Co-Authored-By: Claude Opus 5.5 * fix: keep external secret references intact on delete, deploy and export - Deleting a reference no longer runs the managed backend cleanup, which could delete the secret it points at. - Deploying a variable to another workspace copies the reference, which the target's allow-list then checks, instead of its resolved value. - A plain-secrets workspace export writes references verbatim. - Reference paths reject backslashes, `#` and control characters, which the URL parser would otherwise normalize past the allow-list. Co-Authored-By: Claude Opus 5.5 * fix: skip external secret references in key rotation and the read cache - Workspace key rotation no longer tries to decrypt references. - `get_value?allow_cache=true` does not cache a resolved reference, so Vault rotation and allow-list changes apply on the next read. - `VaultSettings.mount_path` is optional in the OpenAPI spec, matching its serde default. Co-Authored-By: Claude Opus 5.5 * chore: keep the OSS Vault reader stub public and regenerate workspace tool schemas Co-Authored-By: Claude Opus 5.5 * fix: restrict external secret references to workspace admins and keep them out of the managed Vault storage Creating, repointing or removing a reference now takes a workspace admin, on create and update including already-encrypted pushes and cross-workspace deploys; the variable form disables the toggle and fields for anyone else. A reference into the storage of Windmill's own Vault secret backend (same address, namespace folded into the path) is refused on save and on resolve, whatever the allow-list says. Co-Authored-By: Claude Opus 5.5 * fix: compare external secret references as written in workspace comparison Two references that currently resolve to the same value still point at different secrets, so comparing their resolved values hid a repoint from the diff and from deployment. Co-Authored-By: Claude Opus 5.5 * feat: let external secret references reuse the Vault backend connection Co-Authored-By: Claude Opus 5.5 * feat: hide the external reference toggle from non-admins unless already set Co-Authored-By: Claude Opus 5.5 * feat: read whole Vault secrets, show allowed paths, and pick the variable type in one control Co-Authored-By: Claude Opus 5.5 * fix: mask JSON fields only for whole Vault secrets, and skip PEM armor and JSON lines Co-Authored-By: Claude Opus 5.5 * fix: align the reference editor and settings warning with the form and alert guidelines Co-Authored-By: Claude Opus 5.5 * feat: let a saved reference become non-secret directly, and warn when no prefix is allowed Co-Authored-By: Claude Opus 5.5 * fix: save an empty value when a reference is switched to another type Co-Authored-By: Claude Opus 5.5 * fix: keep a saved reference's baseline through type round trips Co-Authored-By: Claude Opus 5.5 * fix: drop an unused binding in the reference parser Co-Authored-By: Claude Opus 5.5 * fix: offer vault reference only when a vault path is allowed Co-Authored-By: Claude Opus 5.5 * chore: update ee-repo-ref to c0fd22ba77e6618cea7611d72fe2e75c8855c595 This commit updates the EE repository reference after PR #932 was merged in windmill-ee-private. Previous ee-repo-ref: 56cba4e7771e7a87e56a67eb115549a31dd12b1a New ee-repo-ref: c0fd22ba77e6618cea7611d72fe2e75c8855c595 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5.5 Co-authored-by: Ruben Fiszel Co-authored-by: windmill-internal-app[bot] --- ...8be507e8c2fa86c8006bb32b5c0660497c128.json | 23 ++ ...1757af5286183faf888c7313f3a657705850.json} | 4 +- ...0a550a03af01798dacce24d42fbe01b51f36.json} | 4 +- backend/ee-repo-ref.txt | 2 +- .../src/datatable_clone.rs | 2 +- .../windmill-api-workspaces/src/workspaces.rs | 13 +- backend/windmill-api/openapi.yaml | 37 +- backend/windmill-api/src/workspaces_export.rs | 4 + .../windmill-common/src/global_settings.rs | 3 + .../windmill-common/src/instance_config.rs | 5 +- .../src/secret_backend/external_ref.rs | 359 ++++++++++++++++++ .../windmill-common/src/secret_backend/mod.rs | 15 +- .../src/secret_backend/resolver.rs | 3 + .../src/secret_backend/tests.rs | 2 + .../src/secret_backend/vault_oss.rs | 12 + .../src/sensitive_log_masks.rs | 90 ++++- backend/windmill-common/src/variables.rs | 10 +- backend/windmill-common/src/workspaces.rs | 4 +- .../tests/secret_backend_integration.rs | 4 + .../tests/secret_backend_migration.rs | 2 + .../windmill-store/src/secret_backend_ext.rs | 7 +- backend/windmill-store/src/variables.rs | 207 +++++++--- cli/src/commands/variable/variable.ts | 8 +- .../variable_ciphertext_shape_unit.test.ts | 1 + cli/windmill-utils-internal/src/deploy.ts | 12 +- .../lib/components/CompareWorkspaces.svelte | 7 +- .../src/lib/components/InstanceSetting.svelte | 3 + .../lib/components/InstanceSettings.svelte | 25 +- .../src/lib/components/VariableEditor.svelte | 15 +- .../src/lib/components/VariableForm.svelte | 332 +++++++++++----- .../copilot/chat/workspaceToolsZod.gen.ts | 2 +- .../src/lib/components/instanceSettings.ts | 10 + .../ExternalSecretsConfig.svelte | 335 ++++++++++++++++ .../SecretBackendConfig.svelte | 84 +++- .../components/variables/VariablesList.svelte | 5 +- frontend/src/lib/externalSecretRef.ts | 26 ++ 36 files changed, 1462 insertions(+), 215 deletions(-) create mode 100644 backend/.sqlx/query-3e0f2e6febdf289aeccbea6b3258be507e8c2fa86c8006bb32b5c0660497c128.json rename backend/.sqlx/{query-0c8a3eb810c96230ba3a5466c55bf24a94eb8a52ceb82cc29dade173ad87569d.json => query-6c9efac7fba0e3f2f34510bb6e2a1757af5286183faf888c7313f3a657705850.json} (70%) rename backend/.sqlx/{query-0600f2a9179f83502c6b13e8e4284f85ca82636f274f5dce47da5a8320a60088.json => query-e2c7dcf549dba16f2a36d6495b4c0a550a03af01798dacce24d42fbe01b51f36.json} (77%) create mode 100644 backend/windmill-common/src/secret_backend/external_ref.rs create mode 100644 frontend/src/lib/components/instanceSettings/ExternalSecretsConfig.svelte create mode 100644 frontend/src/lib/externalSecretRef.ts diff --git a/backend/.sqlx/query-3e0f2e6febdf289aeccbea6b3258be507e8c2fa86c8006bb32b5c0660497c128.json b/backend/.sqlx/query-3e0f2e6febdf289aeccbea6b3258be507e8c2fa86c8006bb32b5c0660497c128.json new file mode 100644 index 0000000000..3b8f6e370c --- /dev/null +++ b/backend/.sqlx/query-3e0f2e6febdf289aeccbea6b3258be507e8c2fa86c8006bb32b5c0660497c128.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT is_secret AND value NOT LIKE '$external:%' AS \"is_secret!\" FROM variable WHERE path = $1 AND workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "is_secret!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "3e0f2e6febdf289aeccbea6b3258be507e8c2fa86c8006bb32b5c0660497c128" +} diff --git a/backend/.sqlx/query-0c8a3eb810c96230ba3a5466c55bf24a94eb8a52ceb82cc29dade173ad87569d.json b/backend/.sqlx/query-6c9efac7fba0e3f2f34510bb6e2a1757af5286183faf888c7313f3a657705850.json similarity index 70% rename from backend/.sqlx/query-0c8a3eb810c96230ba3a5466c55bf24a94eb8a52ceb82cc29dade173ad87569d.json rename to backend/.sqlx/query-6c9efac7fba0e3f2f34510bb6e2a1757af5286183faf888c7313f3a657705850.json index e68f16713a..02d147af11 100644 --- a/backend/.sqlx/query-0c8a3eb810c96230ba3a5466c55bf24a94eb8a52ceb82cc29dade173ad87569d.json +++ b/backend/.sqlx/query-6c9efac7fba0e3f2f34510bb6e2a1757af5286183faf888c7313f3a657705850.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT path FROM variable WHERE path = ANY($1) AND workspace_id = $2 AND is_secret = true", + "query": "SELECT path FROM variable WHERE path = ANY($1) AND workspace_id = $2 AND is_secret = true AND value NOT LIKE '$external:%'", "describe": { "columns": [ { @@ -19,5 +19,5 @@ false ] }, - "hash": "0c8a3eb810c96230ba3a5466c55bf24a94eb8a52ceb82cc29dade173ad87569d" + "hash": "6c9efac7fba0e3f2f34510bb6e2a1757af5286183faf888c7313f3a657705850" } diff --git a/backend/.sqlx/query-0600f2a9179f83502c6b13e8e4284f85ca82636f274f5dce47da5a8320a60088.json b/backend/.sqlx/query-e2c7dcf549dba16f2a36d6495b4c0a550a03af01798dacce24d42fbe01b51f36.json similarity index 77% rename from backend/.sqlx/query-0600f2a9179f83502c6b13e8e4284f85ca82636f274f5dce47da5a8320a60088.json rename to backend/.sqlx/query-e2c7dcf549dba16f2a36d6495b4c0a550a03af01798dacce24d42fbe01b51f36.json index e5aec65b91..ac9ba13b0a 100644 --- a/backend/.sqlx/query-0600f2a9179f83502c6b13e8e4284f85ca82636f274f5dce47da5a8320a60088.json +++ b/backend/.sqlx/query-e2c7dcf549dba16f2a36d6495b4c0a550a03af01798dacce24d42fbe01b51f36.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT workspace_id, path FROM variable WHERE is_secret = true", + "query": "SELECT workspace_id, path FROM variable WHERE is_secret = true AND value NOT LIKE '$external:%'", "describe": { "columns": [ { @@ -22,5 +22,5 @@ false ] }, - "hash": "0600f2a9179f83502c6b13e8e4284f85ca82636f274f5dce47da5a8320a60088" + "hash": "e2c7dcf549dba16f2a36d6495b4c0a550a03af01798dacce24d42fbe01b51f36" } diff --git a/backend/ee-repo-ref.txt b/backend/ee-repo-ref.txt index 82344b8335..2cbe2ba4a0 100644 --- a/backend/ee-repo-ref.txt +++ b/backend/ee-repo-ref.txt @@ -1 +1 @@ -a445ac5e719cc0bb2bc22bb04bb871b00b1bc808 +c0fd22ba77e6618cea7611d72fe2e75c8855c595 diff --git a/backend/windmill-api-workspaces/src/datatable_clone.rs b/backend/windmill-api-workspaces/src/datatable_clone.rs index 067a248aa7..398766b19e 100644 --- a/backend/windmill-api-workspaces/src/datatable_clone.rs +++ b/backend/windmill-api-workspaces/src/datatable_clone.rs @@ -111,7 +111,7 @@ pub(crate) async fn connection_snapshot( .fetch_optional(&mut *conn) .await?; if let Some((stored, true)) = &row { - if windmill_common::secret_backend::is_external_stored_value(stored) { + if windmill_common::secret_backend::resolves_externally(stored) { let secret = windmill_common::secret_backend::get_secret_value( db, w_id, var, stored, ) diff --git a/backend/windmill-api-workspaces/src/workspaces.rs b/backend/windmill-api-workspaces/src/workspaces.rs index 73477f6306..a7e0612ecc 100644 --- a/backend/windmill-api-workspaces/src/workspaces.rs +++ b/backend/windmill-api-workspaces/src/workspaces.rs @@ -77,7 +77,8 @@ use serde::{Deserialize, Serialize}; use sqlx::{FromRow, Postgres, Row, Transaction}; use windmill_common::oauth2::InstanceEvent; use windmill_common::secret_backend::{ - get_secret_backend, is_external_stored_value, is_vault_backend_configured, + get_secret_backend, is_external_ref_value, is_external_stored_value, + is_vault_backend_configured, }; use windmill_common::utils::not_found_if_none; @@ -6772,7 +6773,9 @@ async fn set_encryption_key( .await?; for variable in all_variables { - if !variable.is_secret { + if !variable.is_secret + || windmill_common::secret_backend::resolves_externally(&variable.value) + { continue; } let decrypted_value = @@ -14935,8 +14938,12 @@ async fn compare_two_variables( { // A secret's ciphertext is randomized on every write, so equal secrets are compared // by plaintext, each decrypted with its own workspace's key. An external backend - // reference is the same text on both sides but names a per-workspace entry. + // reference is the same text on both sides but names a per-workspace entry. An + // `$external:` reference is compared as written: two references resolving to equal + // values still point at different secrets. has_changes = !source.is_secret + || is_external_ref_value(&source.value) + || is_external_ref_value(&target.value) || !matches!( tokio::try_join!( decrypt_secret_value(db, source_workspace_id, path, source.value.clone()), diff --git a/backend/windmill-api/openapi.yaml b/backend/windmill-api/openapi.yaml index ccc73a8acc..14c7d6b8c7 100644 --- a/backend/windmill-api/openapi.yaml +++ b/backend/windmill-api/openapi.yaml @@ -7828,6 +7828,24 @@ paths: schema: type: string + /w/{workspace}/variables/external_secret_prefixes: + get: + summary: list the external secret prefixes allowed for this workspace (admin only) + operationId: listExternalSecretPrefixes + tags: + - variable + parameters: + - $ref: "#/components/parameters/WorkspaceId" + responses: + "200": + description: allowed prefixes, such as vault://secret/team + content: + application/json: + schema: + type: array + items: + type: string + /w/{workspace}/variables/delete/{path}: delete: summary: delete variable @@ -29998,14 +30016,13 @@ components: type: object required: - address - - mount_path properties: address: type: string description: HashiCorp Vault server address (e.g., https://vault.company.com:8200) mount_path: type: string - description: KV v2 secrets engine mount path (e.g., windmill) + description: KV v2 secrets engine mount path (e.g., windmill). Empty to test the connection used for `$external:vault://` references, which checks only that Vault accepts Windmill's token. kv_secret_path_prefix: type: string description: Optional path prefix inserted between the KV data/metadata segment and the workspace id (e.g., "apps/windmill"). When set, secrets are stored at `/data///`, allowing a Vault policy scoped to exactly `/data//*`. @@ -30015,6 +30032,12 @@ components: jwt_mount_path: type: string description: Mount path for the JWT auth method in Vault (optional, defaults to "jwt"). Set this when the JWT auth method is mounted at a non-default path, e.g. via `vault auth enable -path= jwt`. + approle_role_id: + type: string + description: AppRole role_id (optional). When set and no static token is given, Windmill logs in with AppRole; the secret_id is read from the `VAULT_APPROLE_SECRET_ID` environment variable of servers and workers. + approle_mount_path: + type: string + description: Mount path of the AppRole auth method (optional, defaults to "approle"). namespace: type: string description: Vault Enterprise namespace (optional) @@ -32890,7 +32913,13 @@ components: description: The path to the variable value: type: string - description: The value of the variable + description: > + The value of the variable. For a secret, `$external:vault://#` + stores a reference to an existing secret in the instance's external + HashiCorp Vault (path as `vault kv get` takes it) instead of a value; without + `#` it reads the whole secret as a JSON object of its fields. It is + resolved on each read, only under the path prefixes a superadmin allowed for + the workspace, and is returned as-is when the secret is not decrypted. is_secret: type: boolean description: Whether the variable is a secret @@ -32939,7 +32968,7 @@ components: description: The path to the variable value: type: string - description: The new value of the variable + description: The new value of the variable. Accepts an `$external:vault://[#]` reference for a secret, as in CreateVariable. is_secret: type: boolean description: Whether the variable is a secret diff --git a/backend/windmill-api/src/workspaces_export.rs b/backend/windmill-api/src/workspaces_export.rs index 46b7a7a8dd..d98ac700a3 100644 --- a/backend/windmill-api/src/workspaces_export.rs +++ b/backend/windmill-api/src/workspaces_export.rs @@ -986,6 +986,10 @@ pub(crate) async fn tarball_workspace( if plain_secret.or(plain_secrets).unwrap_or(false) && var.value.is_some() && var.is_secret + && !var + .value + .as_deref() + .is_some_and(windmill_common::secret_backend::is_external_ref_value) { var.value = Some(decrypt(&mc, var.value.unwrap()).map_err(|e| { Error::internal_err(format!("Error decrypting variable {}: {}", var.path, e)) diff --git a/backend/windmill-common/src/global_settings.rs b/backend/windmill-common/src/global_settings.rs index 2e4e3788ab..82a807c31b 100644 --- a/backend/windmill-common/src/global_settings.rs +++ b/backend/windmill-common/src/global_settings.rs @@ -180,6 +180,7 @@ pub const APP_WORKSPACED_ROUTE_SETTING: &str = "app_workspaced_route"; pub const HTTP_ROUTE_WORKSPACED_ROUTE_SETTING: &str = "http_route_workspaced_route"; pub const HTTP_ROUTE_DEFAULT_ALLOWED_ORIGINS_SETTING: &str = "http_route_default_allowed_origins"; pub const SECRET_BACKEND_SETTING: &str = "secret_backend"; +pub const EXTERNAL_SECRETS_SETTING: &str = "external_secrets"; pub const MIN_KEEP_ALIVE_VERSION_SETTING: &str = "min_keep_alive_version"; pub const GITHUB_ENTERPRISE_APP_SETTING: &str = "github_enterprise_app"; /// Base URL GitHub delivers git-sync repository webhooks to. Falls back to @@ -516,6 +517,7 @@ pub const AGENT_WORKER_BLOCKED_SETTINGS: &[&str] = &[ SCIM_TOKEN_SETTING, SAML_METADATA_SETTING, SECRET_BACKEND_SETTING, + EXTERNAL_SECRETS_SETTING, GITHUB_ENTERPRISE_APP_SETTING, OBJECT_STORE_CONFIG_SETTING, AI_CONFIG_SETTING, @@ -1435,6 +1437,7 @@ mod tests { SCIM_TOKEN_SETTING, SAML_METADATA_SETTING, SECRET_BACKEND_SETTING, + EXTERNAL_SECRETS_SETTING, GITHUB_ENTERPRISE_APP_SETTING, OBJECT_STORE_CONFIG_SETTING, AI_CONFIG_SETTING, diff --git a/backend/windmill-common/src/instance_config.rs b/backend/windmill-common/src/instance_config.rs index bd5767b786..338afbfac6 100644 --- a/backend/windmill-common/src/instance_config.rs +++ b/backend/windmill-common/src/instance_config.rs @@ -1241,7 +1241,10 @@ fn mask_nested_sensitive(key: &str, value: &serde_json::Value) -> serde_json::Va } } // Settings that are maps-of-objects where each child has a sensitive sub-field. - const NESTED_MAP_SENSITIVE: &[(&str, &[&str])] = &[("oauths", &["secret", "signing_secret"])]; + const NESTED_MAP_SENSITIVE: &[(&str, &[&str])] = &[ + ("oauths", &["secret", "signing_secret"]), + ("external_secrets", &["token"]), + ]; for &(parent_key, child_fields) in NESTED_MAP_SENSITIVE { if key == parent_key { if let serde_json::Value::Object(entries) = value { diff --git a/backend/windmill-common/src/secret_backend/external_ref.rs b/backend/windmill-common/src/secret_backend/external_ref.rs new file mode 100644 index 0000000000..375bbbb14f --- /dev/null +++ b/backend/windmill-common/src/secret_backend/external_ref.rs @@ -0,0 +1,359 @@ +/* + * Author: Ruben Fiszel + * Copyright: Windmill Labs, Inc 2026 + * This file and its contents are licensed under the AGPLv3 License. + * Please see the included NOTICE for copyright information and + * LICENSE-AGPL for a copy of the license. + */ + +//! References from a secret variable to a secret that already exists in an +//! external store: the variable stores `$external:://#` and +//! the value is read from the store each time the variable is read. Unlike the +//! `$vault:`/`$aws_sm:`/`$azure_kv:` markers of a managed secret backend, +//! nothing is ever written to the store. Only `vault://` resolves, with `` +//! written as `vault kv get` takes it. +//! +//! The instance's identity in the store can usually read far more than any one +//! workspace should, so a reference resolves only under a prefix a superadmin +//! allowed for that workspace in the `external_secrets` setting. The check runs +//! on save and again on every read, so narrowing the list takes effect on +//! existing references. + +use serde::{Deserialize, Serialize}; + +use crate::{ + db::DB, + error::{Error, Result}, + global_settings::{ + load_value_from_global_settings, EXTERNAL_SECRETS_SETTING, SECRET_BACKEND_SETTING, + }, + secret_backend::{SecretBackendConfig, VaultSettings}, +}; + +pub const EXTERNAL_REF_MARKER: &str = "$external:"; + +/// The `external_secrets` instance setting: the connections references are +/// read through, and where they may point. +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Default)] +pub struct ExternalSecretsSettings { + #[serde(default, skip_serializing_if = "Option::is_none")] + pub vault: Option, + #[serde(default)] + pub allowed: Vec, +} + +#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] +pub struct ExternalSecretAllowedPrefix { + /// `://`, the path matched on whole segments. + pub prefix: String, + /// Workspaces that may reference under `prefix`; empty means all. + #[serde(default)] + pub workspaces: Vec, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum ExternalSecretBackend { + Vault, +} + +impl ExternalSecretBackend { + fn scheme(self) -> &'static str { + match self { + Self::Vault => "vault", + } + } + + fn from_scheme(scheme: &str) -> Result { + match scheme { + "vault" => Ok(Self::Vault), + _ => Err(Error::BadRequest(format!( + "Unsupported external secret backend `{scheme}://`: only `vault://` is supported" + ))), + } + } +} + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct ExternalSecretRef { + pub backend: ExternalSecretBackend, + /// Normalized: no surrounding or doubled slashes. + pub path: String, + /// `None` reads the whole secret, as a JSON object of its fields. + pub key: Option, +} + +pub fn is_external_ref_value(value: &str) -> bool { + value.starts_with(EXTERNAL_REF_MARKER) +} + +/// A stored secret value whose plaintext is not workspace ciphertext but lives +/// in an external store: a managed backend entry or an `$external:` reference. +pub fn resolves_externally(value: &str) -> bool { + super::is_external_stored_value(value) || is_external_ref_value(value) +} + +fn path_segments(path: &str) -> Result> { + let segments: Vec<&str> = path.trim().trim_matches('/').split('/').collect(); + // The URL parser treats `\` as `/` and drops tabs and newlines, so either would let a + // `..` segment slip past this check and the allow-list. + if segments.iter().any(|s| { + s.is_empty() + || *s == "." + || *s == ".." + || s.contains(['?', '%', '#', '\\']) + || s.contains(char::is_control) + }) { + return Err(Error::BadRequest(format!( + "Invalid external secret path `{path}`: expected segments like `secret/team/app`" + ))); + } + Ok(segments) +} + +fn split_scheme(uri: &str) -> Result<(ExternalSecretBackend, &str)> { + let (scheme, rest) = uri.trim().split_once("://").ok_or_else(|| { + Error::BadRequest(format!( + "Invalid external secret reference `{uri}`: expected `://[#]`, e.g. `vault://secret/team/app#token`" + )) + })?; + Ok((ExternalSecretBackend::from_scheme(scheme)?, rest)) +} + +/// Parses `$external:://[#]` (the marker is optional). +pub fn parse_external_ref(value: &str) -> Result { + let uri = value.strip_prefix(EXTERNAL_REF_MARKER).unwrap_or(value); + let (backend, rest) = split_scheme(uri)?; + let (path, key) = match rest.split_once('#') { + Some((_, key)) if key.trim().is_empty() => { + return Err(Error::BadRequest(format!( + "Invalid external secret reference `{uri}`: the key after `#` is empty" + ))) + } + Some((path, key)) => (path, Some(key.trim().to_string())), + None => (rest, None), + }; + Ok(ExternalSecretRef { backend, path: path_segments(path)?.join("/"), key }) +} + +fn applies_to(a: &ExternalSecretAllowedPrefix, w_id: &str) -> bool { + a.workspaces.is_empty() || a.workspaces.iter().any(|w| w == w_id) +} + +/// The prefixes `w_id` may reference, as a superadmin wrote them. +pub async fn allowed_prefixes(db: &DB, w_id: &str) -> Result> { + Ok(load_settings(db) + .await? + .allowed + .into_iter() + .filter(|a| applies_to(a, w_id)) + .map(|a| a.prefix) + .collect()) +} + +fn is_allowed(settings: &ExternalSecretsSettings, w_id: &str, r: &ExternalSecretRef) -> bool { + let path: Vec<&str> = r.path.split('/').collect(); + settings.allowed.iter().any(|a| { + applies_to(a, w_id) + && split_scheme(&a.prefix).is_ok_and(|(backend, prefix)| { + backend == r.backend && path_segments(prefix).is_ok_and(|p| path.starts_with(&p)) + }) + }) +} + +async fn load_settings(db: &DB) -> Result { + match load_value_from_global_settings(db, EXTERNAL_SECRETS_SETTING).await? { + Some(value) => serde_json::from_value(value).map_err(|e| { + Error::internal_err(format!("Invalid {EXTERNAL_SECRETS_SETTING} setting: {e}")) + }), + None => Ok(ExternalSecretsSettings::default()), + } +} + +fn vault_segments(s: Option<&str>) -> Vec<&str> { + s.unwrap_or_default() + .split('/') + .filter(|s| !s.trim().is_empty()) + .collect() +} + +/// Whether `path`, read through `conn`, lands in the storage of the managed Vault +/// backend `managed`. The namespace is folded into the path because Vault also +/// takes a namespace as a path prefix, so `ns/kv/x` with no namespace header +/// reaches the same secret as `kv/x` under namespace `ns`. +fn is_managed_storage(managed: &VaultSettings, conn: &VaultSettings, path: &str) -> bool { + let address = |s: &VaultSettings| s.address.trim().trim_end_matches('/').to_lowercase(); + let mount = vault_segments(Some(&managed.mount_path)); + if mount.is_empty() || address(managed) != address(conn) { + return false; + } + let mut storage = vault_segments(managed.namespace.as_deref()); + storage.extend(mount); + storage.extend(vault_segments(managed.kv_secret_path_prefix.as_deref())); + let mut target = vault_segments(conn.namespace.as_deref()); + target.extend(vault_segments(Some(path))); + target.starts_with(&storage) +} + +/// Parses a reference, checks it is allowed for `w_id`, and returns the Vault +/// connection to read it through: the `external_secrets` one, or else the +/// managed backend's when that is Vault. +async fn authorize( + db: &DB, + w_id: &str, + value: &str, +) -> Result<(Option, ExternalSecretRef)> { + let r = parse_external_ref(value)?; + let settings = load_settings(db).await?; + if !is_allowed(&settings, w_id, &r) { + let allowed: Vec<_> = settings + .allowed + .iter() + .filter(|a| applies_to(a, w_id)) + .map(|a| format!("`{}/**`", a.prefix.trim_end_matches('/'))) + .collect(); + let target = format!("External secret `{}://{}`", r.backend.scheme(), r.path); + return Err(Error::PermissionDenied(if allowed.is_empty() { + format!("{target} is not allowed: no Vault paths are allowed for workspace {w_id} (instance settings > Secret storage)") + } else { + format!( + "{target} is outside the paths allowed for workspace {w_id}: {}", + allowed.join(", ") + ) + })); + } + let managed = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING) + .await? + .and_then(|v| serde_json::from_value(v).ok()) + { + Some(SecretBackendConfig::HashiCorpVault(managed)) => Some(managed), + _ => None, + }; + let conn = settings.vault.or_else(|| managed.clone()); + // The managed backend's storage holds every workspace's secrets, so no + // allow-list entry may open it. + if let (Some(managed), Some(conn)) = (&managed, &conn) { + if is_managed_storage(managed, conn, &r.path) { + return Err(Error::PermissionDenied(format!( + "External secret `vault://{}` points into the storage of Windmill's own Vault secret backend", + r.path + ))); + } + } + Ok((conn, r)) +} + +/// Write-time check of an `$external:` value. Does not contact the store, so a +/// reference may be saved before its secret exists. +pub async fn check_external_ref(db: &DB, w_id: &str, value: &str) -> Result<()> { + authorize(db, w_id, value).await.map(|_| ()) +} + +pub(crate) async fn resolve_external_ref( + db: &DB, + w_id: &str, + variable_path: &str, + value: &str, +) -> Result { + let (conn, r) = authorize(db, w_id, value).await?; + let read = match r.backend { + ExternalSecretBackend::Vault => { + let vault = conn.as_ref().ok_or_else(|| { + Error::BadRequest( + "No Vault connection is configured for external secrets (instance settings > Secret storage)" + .to_string(), + ) + })?; + super::read_vault_external_secret(db, vault, &r.path, r.key.as_deref()).await + } + }; + read.map_err(|e| match e { + Error::NotFound(m) => Error::NotFound(format!("{m} (variable {variable_path})")), + Error::PermissionDenied(m) => { + Error::PermissionDenied(format!("{m} (variable {variable_path})")) + } + e => e, + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn settings(allowed: &[(&str, &[&str])]) -> ExternalSecretsSettings { + serde_json::from_value(serde_json::json!({ + "allowed": allowed + .iter() + .map(|(p, w)| serde_json::json!({"prefix": p, "workspaces": w})) + .collect::>(), + })) + .unwrap() + } + + fn allowed(s: &ExternalSecretsSettings, w_id: &str, value: &str) -> bool { + is_allowed(s, w_id, &parse_external_ref(value).unwrap()) + } + + #[test] + fn parses_references() { + assert_eq!( + parse_external_ref("$external:vault:///secret/team/app/#token").unwrap(), + ExternalSecretRef { + backend: ExternalSecretBackend::Vault, + path: "secret/team/app".to_string(), + key: Some("token".to_string()) + } + ); + assert_eq!( + parse_external_ref("$external:vault://secret/team/app") + .unwrap() + .key, + None + ); + for bad in [ + "$external:secret/app#k", + "$external:vault://secret/app#", + "$external:vault://secret//app#k", + "$external:vault://secret/../sys#k", + "$external:vault://secret/team/..\\other/app#k", + "$external:vault://secret/app?list=true#k", + "$external:aws-sm://secret/app#k", + ] { + assert!(parse_external_ref(bad).is_err(), "{bad}"); + } + } + + #[test] + fn allow_list_matches_whole_segments_and_workspaces() { + let s = settings(&[ + ("vault://secret/team", &["ws1"]), + ("vault://kv/shared/", &[]), + ]); + assert!(allowed(&s, "ws1", "vault://secret/team/app#k")); + assert!(allowed(&s, "ws1", "vault://secret/team#k")); + assert!(!allowed(&s, "ws1", "vault://secret/team-other/app#k")); + assert!(!allowed(&s, "ws2", "vault://secret/team/app#k")); + assert!(allowed(&s, "ws2", "vault://kv/shared/x#k")); + assert!(!allowed(&s, "ws2", "vault://kv#k")); + } + + #[test] + fn refuses_the_managed_backend_storage() { + let vault = |address: &str, mount: &str, prefix: Option<&str>, ns: Option<&str>| { + serde_json::from_value::(serde_json::json!({ + "address": address, "mount_path": mount, + "kv_secret_path_prefix": prefix, "namespace": ns, + })) + .unwrap() + }; + let managed = vault("https://vault:8200", "kv", Some("/apps/wm/"), Some("ns")); + let same = vault("https://vault:8200/", "", None, Some("ns")); + assert!(is_managed_storage(&managed, &same, "kv/apps/wm/ws1/f/x")); + assert!(is_managed_storage(&managed, &same, "kv/apps/wm")); + assert!(!is_managed_storage(&managed, &same, "kv/apps/other/x")); + assert!(!is_managed_storage(&managed, &same, "secret/apps/wm/x")); + let root = vault("https://vault:8200", "", None, None); + assert!(is_managed_storage(&managed, &root, "ns/kv/apps/wm/ws1/x")); + let other = vault("https://other:8200", "", None, Some("ns")); + assert!(!is_managed_storage(&managed, &other, "kv/apps/wm/ws1/x")); + } +} diff --git a/backend/windmill-common/src/secret_backend/mod.rs b/backend/windmill-common/src/secret_backend/mod.rs index 9fd117c235..a721f0fb39 100644 --- a/backend/windmill-common/src/secret_backend/mod.rs +++ b/backend/windmill-common/src/secret_backend/mod.rs @@ -14,8 +14,10 @@ pub mod database; pub mod resolver; +pub mod external_ref; pub use resolver::*; +pub use external_ref::*; #[cfg(feature = "private")] pub mod vault_ee; @@ -138,7 +140,9 @@ impl Default for SecretBackendConfig { pub struct VaultSettings { /// Vault server address (e.g., "https://vault.company.com:8200") pub address: String, - /// KV v2 mount path (e.g., "windmill") + /// KV v2 mount path (e.g., "windmill"). Empty for the external-secrets + /// connection, which reads whatever mount a reference names. + #[serde(default)] pub mount_path: String, /// Optional path prefix inserted between the KV `data`/`metadata` segment /// and the workspace id, e.g. "apps/windmill". When set, secrets live at @@ -156,6 +160,15 @@ pub struct VaultSettings { /// e.g. via `vault auth enable -path=my-mount jwt`. #[serde(skip_serializing_if = "Option::is_none")] pub jwt_mount_path: Option, + /// AppRole role_id. When set (and no static token), Windmill logs in with + /// AppRole; the secret_id is read from the `VAULT_APPROLE_SECRET_ID` + /// environment variable of servers and workers, never from settings, and + /// is omitted when unset (a role with `bind_secret_id=false`). + #[serde(skip_serializing_if = "Option::is_none")] + pub approle_role_id: Option, + /// Mount path of the AppRole auth method (defaults to "approle"). + #[serde(skip_serializing_if = "Option::is_none")] + pub approle_mount_path: Option, /// Vault Enterprise namespace (optional) #[serde(skip_serializing_if = "Option::is_none")] pub namespace: Option, diff --git a/backend/windmill-common/src/secret_backend/resolver.rs b/backend/windmill-common/src/secret_backend/resolver.rs index 4a9a65bc0d..fff58528c1 100644 --- a/backend/windmill-common/src/secret_backend/resolver.rs +++ b/backend/windmill-common/src/secret_backend/resolver.rs @@ -298,6 +298,9 @@ pub async fn get_secret_value( path: &str, encrypted_value: &str, ) -> Result { + if super::is_external_ref_value(encrypted_value) { + return super::resolve_external_ref(db, workspace_id, path, encrypted_value).await; + } let backend = get_secret_backend(db).await?; match backend.backend_name() { diff --git a/backend/windmill-common/src/secret_backend/tests.rs b/backend/windmill-common/src/secret_backend/tests.rs index 8f04a755b7..d749e8f27e 100644 --- a/backend/windmill-common/src/secret_backend/tests.rs +++ b/backend/windmill-common/src/secret_backend/tests.rs @@ -28,6 +28,8 @@ mod tests { kv_secret_path_prefix: None, jwt_role: Some("windmill-secrets".to_string()), jwt_mount_path: None, + approle_role_id: None, + approle_mount_path: None, namespace: None, token: Some("test-root-token".to_string()), skip_ssl_verify: None, diff --git a/backend/windmill-common/src/secret_backend/vault_oss.rs b/backend/windmill-common/src/secret_backend/vault_oss.rs index a20738f118..8ee486dd11 100644 --- a/backend/windmill-common/src/secret_backend/vault_oss.rs +++ b/backend/windmill-common/src/secret_backend/vault_oss.rs @@ -129,3 +129,15 @@ pub async fn generate_vault_jwt(_db: &DB, _vault_address: &str) -> Result, +) -> Result { + Err(Error::internal_err( + "HashiCorp Vault integration requires Enterprise Edition".to_string(), + )) +} diff --git a/backend/windmill-common/src/sensitive_log_masks.rs b/backend/windmill-common/src/sensitive_log_masks.rs index d9a10e764a..4c1c73e277 100644 --- a/backend/windmill-common/src/sensitive_log_masks.rs +++ b/backend/windmill-common/src/sensitive_log_masks.rs @@ -220,6 +220,31 @@ pub fn unregister_running_job(job_id: Uuid) { } } +/// `secret` and, when it spans several lines, each of them: `handle_child` masks a log +/// line by line, so a multi-line secret such as a PEM key never matches as a whole. Masks +/// apply to every running job, so lines any job may print are left out: PEM armor, and the +/// lines of a JSON document (`"type": "service_account",`). +fn mask_patterns(secret: &str) -> impl Iterator { + let split = secret.contains('\n') && !secret.trim_start().starts_with(['{', '[']); + let lines = split.then(|| { + secret + .lines() + .map(str::trim) + .filter(|l| !l.starts_with("-----")) + }); + std::iter::once(secret) + .chain(lines.into_iter().flatten()) + .filter(|s| s.len() >= MIN_SECRET_LENGTH) +} + +fn insert_masks(job: &mut JobMasks, secret: &str) { + for pattern in mask_patterns(secret) { + if job.secrets.insert(pattern.to_string()) { + job.compiled = None; + } + } +} + /// Register a secret value for ALL currently running jobs. /// Used when a secret is fetched via the embedded server (we don't know which job triggered it). pub fn register_secret_for_all_running_jobs(secret: &str) { @@ -236,13 +261,42 @@ pub fn register_secret_for_all_running_jobs(secret: &str) { let mut masks = SENSITIVE_MASKS.write().unwrap_or_else(|e| e.into_inner()); for job_id in job_ids { if let Some(job) = masks.get_mut(&job_id) { - if job.secrets.insert(secret.to_string()) { - job.compiled = None; - } + insert_masks(job, secret); } } } +/// Like [`register_secret_for_all_running_jobs`], and when `secret` is a JSON object or +/// array also each string in it: `$jsonvar:` hands a job the parsed value, which can then +/// print one field on its own. Only for values whose every field is secret, such as a whole +/// Vault secret: masks apply to every running job, and a JSON variable's other fields +/// (`"user": "postgres"`) would mask unrelated logs. +pub fn register_secret_and_fields_for_all_running_jobs(secret: &str) { + register_secret_for_all_running_jobs(secret); + for field in json_strings(secret) { + register_secret_for_all_running_jobs(&field); + } +} + +/// Every string inside `secret` when it is a JSON object or array. +fn json_strings(secret: &str) -> Vec { + fn walk(v: serde_json::Value, out: &mut Vec) { + match v { + serde_json::Value::String(s) => out.push(s), + serde_json::Value::Array(a) => a.into_iter().for_each(|v| walk(v, out)), + serde_json::Value::Object(o) => o.into_iter().for_each(|(_, v)| walk(v, out)), + _ => {} + } + } + let mut out = vec![]; + if secret.trim_start().starts_with(['{', '[']) { + if let Ok(v) = serde_json::from_str(secret) { + walk(v, &mut out); + } + } + out +} + /// Register a secret value for a specific job. /// Used for the job's own token and for `$encrypted:` args, where we know the job ID. pub fn register_secret_for_job(job_id: Uuid, secret: &str) { @@ -251,9 +305,7 @@ pub fn register_secret_for_job(job_id: Uuid, secret: &str) { } let mut masks = SENSITIVE_MASKS.write().unwrap_or_else(|e| e.into_inner()); if let Some(job) = masks.get_mut(&job_id) { - if job.secrets.insert(secret.to_string()) { - job.compiled = None; - } + insert_masks(job, secret); } } @@ -281,6 +333,32 @@ mod tests { unregister_running_job(job_id); } + /// Logs are masked line by line, so a multi-line secret must be masked one line at a time. + #[test] + fn masks_each_line_of_a_multi_line_secret() { + let job_id = Uuid::new_v4(); + register_running_job(job_id); + register_secret_for_job( + job_id, + "-----BEGIN KEY-----\nfirstlineofkey\nsecondlineofkey\n", + ); + let snap = snapshot(&job_id).expect("secret registered"); + for line in ["firstlineofkey", "secondlineofkey"] { + assert!(!snap.mask(line).contains(line), "{line}"); + } + assert_eq!(snap.mask("-----BEGIN KEY-----"), "-----BEGIN KEY-----"); + unregister_running_job(job_id); + } + + #[test] + fn finds_each_string_of_a_whole_secret() { + assert_eq!( + json_strings(r#"{"a":"secretvalue1","b":["secretvalue2"],"n":12345678}"#), + ["secretvalue1", "secretvalue2"] + ); + assert!(json_strings("not json").is_empty()); + } + /// A line-by-line sink must not repeat the notice on every line, and must still /// pick up a secret registered after the masker was built. #[test] diff --git a/backend/windmill-common/src/variables.rs b/backend/windmill-common/src/variables.rs index f3ee17769b..3e17b5140d 100644 --- a/backend/windmill-common/src/variables.rs +++ b/backend/windmill-common/src/variables.rs @@ -9,7 +9,7 @@ use crate::db::{Authable, UserDB}; use crate::error::{self, Error}; use crate::scripts::ScriptHash; -use crate::secret_backend::{get_secret_value, is_external_stored_value}; +use crate::secret_backend::{get_secret_value, resolves_externally}; use crate::utils::{strip_json_nul, WarnAfterExt}; use crate::worker::{to_raw_value, Connection}; use crate::workspace_crypt::CipherFormat; @@ -490,7 +490,7 @@ pub async fn decrypt_secret_value( ) -> crate::error::Result { if value.is_empty() { Ok(value) - } else if is_external_stored_value(&value) { + } else if resolves_externally(&value) { get_secret_value(db, w_id, path, &value).await } else { let mc = build_crypt(db, w_id).await?; @@ -687,7 +687,7 @@ async fn upgrade_workspace_secret_variables(db: &DB, w_id: &str) -> error::Resul .await?; let last_page = (rows.len() as i64) < LEGACY_SECRET_UPGRADE_BATCH; for (path, value) in rows { - if value.is_empty() || is_external_stored_value(&value) || mc.is_v2(&value) { + if value.is_empty() || resolves_externally(&value) || mc.is_v2(&value) { after = path; continue; } @@ -1029,7 +1029,7 @@ pub async fn get_variable_or_self( if let Some(record) = record { let mut value = record.value; if record.is_secret { - if is_external_stored_value(&value) { + if resolves_externally(&value) { value = get_secret_value(db, w_id, &path, &value).await?; } else { let mc = build_crypt(db, w_id).await?; @@ -1078,7 +1078,7 @@ pub async fn get_variable_or_self_as( if let Some(record) = record { let mut value = record.value; if record.is_secret { - if is_external_stored_value(&value) { + if resolves_externally(&value) { value = get_secret_value(db, w_id, &var_path, &value).await?; } else { let mc = build_crypt(db, w_id).await?; diff --git a/backend/windmill-common/src/workspaces.rs b/backend/windmill-common/src/workspaces.rs index 7f4c60faa3..0a3bf0ac6f 100644 --- a/backend/windmill-common/src/workspaces.rs +++ b/backend/windmill-common/src/workspaces.rs @@ -11,7 +11,7 @@ use crate::{ error::{self, to_anyhow, Error, Result}, get_database_url, roles::{caller_capabilities, AuthorLevel, RoleCapabilities}, - secret_backend::{get_secret_value, is_external_stored_value}, + secret_backend::{get_secret_value, resolves_externally}, utils::{get_custom_pg_instance_password, get_custom_pg_instance_replication_password}, variables::{build_crypt, decrypt}, PgDatabase, DB, @@ -3813,7 +3813,7 @@ async fn transform_json_unchecked( .map_err(to_anyhow)? .ok_or_else(|| Error::NotFound(format!("variable {path} does not exist")))?; let value = if is_secret { - if is_external_stored_value(&value) { + if resolves_externally(&value) { get_secret_value(db, w_id, &s[5..], &value).await? } else { let mc = build_crypt(&db, &w_id).await?; diff --git a/backend/windmill-common/tests/secret_backend_integration.rs b/backend/windmill-common/tests/secret_backend_integration.rs index a94a67e336..1a861a297c 100644 --- a/backend/windmill-common/tests/secret_backend_integration.rs +++ b/backend/windmill-common/tests/secret_backend_integration.rs @@ -79,6 +79,8 @@ mod tests { kv_secret_path_prefix: None, jwt_role: None, // Static token mode jwt_mount_path: None, + approle_role_id: None, + approle_mount_path: None, namespace: None, token: Some( std::env::var("VAULT_TOKEN").unwrap_or_else(|_| "test-root-token".to_string()), @@ -96,6 +98,8 @@ mod tests { kv_secret_path_prefix: None, jwt_role: Some("windmill-secrets".to_string()), // JWT mode jwt_mount_path: None, + approle_role_id: None, + approle_mount_path: None, namespace: None, token: None, // No static token - use JWT skip_ssl_verify: None, diff --git a/backend/windmill-common/tests/secret_backend_migration.rs b/backend/windmill-common/tests/secret_backend_migration.rs index 5a4086fa1b..2a6981794c 100644 --- a/backend/windmill-common/tests/secret_backend_migration.rs +++ b/backend/windmill-common/tests/secret_backend_migration.rs @@ -37,6 +37,8 @@ fn test_vault_settings() -> VaultSettings { kv_secret_path_prefix: None, jwt_role: Some("windmill-secrets".to_string()), jwt_mount_path: None, + approle_role_id: None, + approle_mount_path: None, namespace: None, token: Some(std::env::var("VAULT_TOKEN").unwrap_or_else(|_| "test-root-token".to_string())), skip_ssl_verify: None, diff --git a/backend/windmill-store/src/secret_backend_ext.rs b/backend/windmill-store/src/secret_backend_ext.rs index 16d8ae5a93..af0380a5ab 100644 --- a/backend/windmill-store/src/secret_backend_ext.rs +++ b/backend/windmill-store/src/secret_backend_ext.rs @@ -23,9 +23,10 @@ use windmill_common::{ }; pub use windmill_common::secret_backend::{ - get_secret_backend, get_secret_value, is_aws_sm_stored_value, is_azure_kv_stored_value, - is_external_stored_value, is_http_store_stored_value, is_vault_backend_configured, - is_vault_stored_value, + check_external_ref, get_secret_backend, get_secret_value, is_aws_sm_stored_value, + is_azure_kv_stored_value, is_external_ref_value, is_external_stored_value, + is_http_store_stored_value, is_vault_backend_configured, is_vault_stored_value, + parse_external_ref, resolves_externally, }; /// The HTTP store's key is derived from the variable path, so moving a reference would point it diff --git a/backend/windmill-store/src/variables.rs b/backend/windmill-store/src/variables.rs index b9e4d078d5..ef70e59bb7 100644 --- a/backend/windmill-store/src/variables.rs +++ b/backend/windmill-store/src/variables.rs @@ -14,9 +14,9 @@ use windmill_common::db::{Authable, DB}; use windmill_common::workspaces::{check_deploy_rules, operator_can_draft, RuleCheckResult}; use crate::secret_backend_ext::{ - delete_secret_from_backend, get_secret_value, http_store_rename_error, - is_external_stored_value, is_http_store_stored_value, is_vault_stored_value, - rename_vault_secret, store_managed_secret_value, store_secret_value, + check_external_ref, delete_secret_from_backend, get_secret_value, http_store_rename_error, + is_external_ref_value, is_http_store_stored_value, is_vault_stored_value, parse_external_ref, + rename_vault_secret, resolves_externally, store_managed_secret_value, store_secret_value, }; use windmill_common::utils::{ check_proper_path, escape_ilike_pattern, sanitize_db_error, BulkDeleteRequest, @@ -39,12 +39,13 @@ use windmill_common::{ db::{DbWithOptAuthed, UserDB}, error::{Error, JsonResult, Result}, scripts::ScriptHash, + secret_backend::allowed_prefixes, user_drafts::{ decrypt_draft_secret_value, delete_all_drafts_for_path, delete_own_draft_for_path, fetch_draft_only, fetch_draft_only_list_rows, maybe_overlay_draft, UserDraftItemKind, WithDraftOverlay, ENCRYPTED_DRAFT_PREFIX, }, - utils::{not_found_if_none, paginate, Pagination, StripPath, WarnAfterExt}, + utils::{not_found_if_none, paginate, require_admin, Pagination, StripPath, WarnAfterExt}, variables::{ build_crypt, get_reserved_variables, ContextualVariable, CreateVariable, ListableVariable, }, @@ -76,6 +77,10 @@ pub fn workspaced_service() -> Router { .route("/delete_bulk", delete(delete_variables_bulk)) .route("/create", post(create_variable)) .route("/encrypt", post(encrypt_value)) + .route( + "/external_secret_prefixes", + get(list_external_secret_prefixes), + ) } async fn list_contextual_variables( @@ -139,7 +144,8 @@ async fn list_variables( .fields(&[ "variable.workspace_id", "variable.path", - "CASE WHEN is_secret IS TRUE THEN null ELSE variable.value::text END as value", + // An `$external:` reference names where the secret lives, not the secret. + "CASE WHEN is_secret IS TRUE AND variable.value NOT LIKE '$external:%' THEN null ELSE variable.value::text END as value", "is_secret", "variable.description", "variable.extra_perms", @@ -426,7 +432,7 @@ async fn get_variable( } else if !value.is_empty() && decrypt_secret { // Use secret backend for decryption (supports both DB and Vault) Some(get_secret_value(&db, &w_id, &variable.path, &value).await?) - } else if q.include_encrypted.unwrap_or(false) { + } else if q.include_encrypted.unwrap_or(false) || is_external_ref_value(&value) { Some(value) } else { None @@ -580,7 +586,7 @@ async fn check_path_conflict(db: &DB, w_id: &str, path: &str) -> Result<()> { /// Values stored by an external backend ($vault:/$aws_sm:/$azure_kv:/$http_store: markers) /// are not workspace ciphertext and are passed through untouched. fn validate_already_encrypted_secret(path: &str, value: &str) -> Result<()> { - if is_external_stored_value(value) { + if resolves_externally(value) { return Ok(()); } let looks_like_ciphertext = STANDARD @@ -597,6 +603,71 @@ fn validate_already_encrypted_secret(path: &str, value: &str) -> Result<()> { Ok(()) } +/// Creating, repointing or removing an `$external:` reference takes a workspace +/// admin. Every reference resolves with the instance's one store identity, so +/// the allow-list cannot tell users apart: any writer could point a variable at +/// a secret under an allowed prefix and read it. +fn check_external_ref_change( + is_admin: bool, + old_ref: Option<&str>, + new_ref: Option<&str>, +) -> Result<()> { + if !is_admin && old_ref != new_ref { + return Err(Error::PermissionDenied( + "Only workspace admins can create, repoint or remove an external secret reference" + .to_string(), + )); + } + Ok(()) +} + +/// The `value` column for a secret sent to create or update, replacing +/// `old_ref` when the variable currently holds a reference. An `$external:` +/// reference is checked against the workspace's allowed prefixes and kept +/// verbatim, including when pushed as already encrypted by sync; any other +/// value goes to the configured secret backend, or with `windmill_writes` +/// (an OAuth token Windmill refreshes) to [`store_managed_secret_value`]. +async fn secret_value_to_store( + db: &DB, + w_id: &str, + path: &str, + value: String, + already_encrypted: bool, + op: &'static str, + is_admin: bool, + old_ref: Option<&str>, + windmill_writes: bool, +) -> Result { + if already_encrypted && !is_external_ref_value(&value) { + check_external_ref_change(is_admin, old_ref, None)?; + // Stored verbatim, so it must be ciphertext and not plaintext mislabeled as encrypted. + validate_already_encrypted_secret(path, &value)?; + return Ok(value); + } + // A restored draft sends the `$encrypted:` marker as-is; decrypt it back + // (validating against the workspace key) before storing it. + let value = if value.starts_with(ENCRYPTED_DRAFT_PREFIX) { + decrypt_draft_secret_value(db, w_id, &value).await? + } else { + value + }; + check_external_ref_change( + is_admin, + old_ref, + is_external_ref_value(&value).then_some(value.as_str()), + )?; + if is_external_ref_value(&value) { + check_external_ref(db, w_id, &value).await?; + windmill_common::feature_usage::log_feature_usage("external_secret_ref", "saved", op); + return Ok(value); + } + if windmill_writes { + store_managed_secret_value(db, w_id, path, &value).await + } else { + store_secret_value(db, w_id, path, &value).await + } +} + async fn create_variable( authed: ApiAuthed, Extension(db): Extension, @@ -644,7 +715,7 @@ pub async fn create_variable_with_account( user_db: UserDB, webhook: &WebhookShared, w_id: String, - variable: CreateVariable, + mut variable: CreateVariable, already_encrypted: bool, account: Option, ) -> Result> { @@ -680,25 +751,21 @@ pub async fn create_variable_with_account( let authed = maybe_refresh_folders(&variable.path, &w_id, authed, &db).await; check_path_conflict(&db, &w_id, &variable.path).await?; - let value = if variable.is_secret && !already_encrypted { - // A restored draft sends the `$encrypted:` marker as-is; decrypt it back - // (validating against the workspace key) before the secret backend re-stores it. - let plain = if variable.value.starts_with(ENCRYPTED_DRAFT_PREFIX) { - decrypt_draft_secret_value(&db, &w_id, &variable.value).await? - } else { - variable.value.clone() - }; - if account.is_some() { - store_managed_secret_value(&db, &w_id, &variable.path, &plain).await? - } else { - store_secret_value(&db, &w_id, &variable.path, &plain).await? - } + let value = if variable.is_secret { + let value = std::mem::take(&mut variable.value); + secret_value_to_store( + &db, + &w_id, + &variable.path, + value, + already_encrypted, + "create", + authed.is_admin, + None, + account.is_some(), + ) + .await? } else { - if variable.is_secret { - // already_encrypted == true: value is stored verbatim, so it must be - // ciphertext and not plaintext mislabeled as encrypted. - validate_already_encrypted_secret(&variable.path, &variable.value)?; - } variable.value }; @@ -788,6 +855,15 @@ pub async fn create_variable_with_account( Ok(account_id) } +async fn list_external_secret_prefixes( + authed: ApiAuthed, + Extension(db): Extension, + Path(w_id): Path, +) -> JsonResult> { + require_admin(authed.is_admin, &authed.username)?; + Ok(Json(allowed_prefixes(&db, &w_id).await?)) +} + async fn encrypt_value( Extension(db): Extension, Path(w_id): Path, @@ -821,9 +897,10 @@ async fn delete_variable( return Err(Error::PermissionDenied(msg)); } - // Check if variable is a secret before deleting (for Vault cleanup) + // Check if variable is a secret before deleting (for Vault cleanup). An external reference + // is excluded: its path may name a secret Windmill does not own. let is_secret = sqlx::query_scalar!( - "SELECT is_secret FROM variable WHERE path = $1 AND workspace_id = $2", + r#"SELECT is_secret AND value NOT LIKE '$external:%' AS "is_secret!" FROM variable WHERE path = $1 AND workspace_id = $2"#, path, &w_id ) @@ -995,9 +1072,9 @@ async fn delete_variables_bulk( return Err(Error::PermissionDenied(msg)); } - // Query which paths are secrets before deletion (for Vault cleanup) + // Query which paths are secrets before deletion (for Vault cleanup), external references excluded let secret_paths: Vec = sqlx::query_scalar!( - "SELECT path FROM variable WHERE path = ANY($1) AND workspace_id = $2 AND is_secret = true", + "SELECT path FROM variable WHERE path = ANY($1) AND workspace_id = $2 AND is_secret = true AND value NOT LIKE '$external:%'", &request.paths, &w_id ) @@ -1199,35 +1276,37 @@ async fn update_variable( // Determine the target path for storing secrets (use new path if provided) let target_path = ns.path.as_deref().unwrap_or(path); if let Some(nvalue) = ns.value.clone() { - let is_secret = if ns.is_secret.is_some() { - ns.is_secret.unwrap() - } else { - sqlx::query_scalar!( - "SELECT is_secret from variable WHERE path = $1 AND workspace_id = $2", - &path, - &w_id - ) - .fetch_optional(&db) - .await? - .unwrap_or(false) - }; + let old = sqlx::query!( + "SELECT value::text AS \"value!\", is_secret FROM variable WHERE path = $1 AND workspace_id = $2", + path, + w_id + ) + .fetch_optional(&db) + .await?; + let is_secret = ns + .is_secret + .unwrap_or_else(|| old.as_ref().is_some_and(|o| o.is_secret)); + let old_ref = old + .as_ref() + .filter(|o| o.is_secret && is_external_ref_value(&o.value)) + .map(|o| o.value.as_str()); - let value = if is_secret && !already_encrypted.unwrap_or(false) { - // Decrypt a restored draft's `$encrypted:` marker before re-storing; see create_variable. - let plain = if nvalue.starts_with(ENCRYPTED_DRAFT_PREFIX) { - decrypt_draft_secret_value(&db, &w_id, &nvalue).await? - } else { - nvalue - }; - // Use secret backend for encryption (supports both DB and Vault) - // Store at target_path (new path if renaming, otherwise current path) - store_secret_value(&db, &w_id, target_path, &plain).await? + let value = if is_secret { + // Stored at target_path (the new path when renaming) + secret_value_to_store( + &db, + &w_id, + target_path, + nvalue, + already_encrypted.unwrap_or(false), + "update", + authed.is_admin, + old_ref, + false, + ) + .await? } else { - if is_secret { - // already_encrypted == true: value is stored verbatim, so it must - // be ciphertext and not plaintext mislabeled as encrypted. - validate_already_encrypted_secret(target_path, &nvalue)?; - } + check_external_ref_change(authed.is_admin, old_ref, None)?; nvalue }; sqlb.set_str("value", &value); @@ -1594,6 +1673,10 @@ pub async fn get_value_internal<'a>( unreachable!() }; + // A reference is resolved on every read, so Vault rotation and allow-list changes apply at once. + let cacheable = !is_external_ref_value(&variable.value); + let whole_secret_ref = is_external_ref_value(&variable.value) + && parse_external_ref(&variable.value).is_ok_and(|r| r.key.is_none()); let r = if variable.is_secret { audit_decrypt_secret(db_with_opt_authed, &w_id, &variable.path).await?; @@ -1623,13 +1706,19 @@ pub async fn get_value_internal<'a>( }; if variable.is_secret && !r.is_empty() { - windmill_common::sensitive_log_masks::register_secret_for_all_running_jobs(&r); + if whole_secret_ref { + windmill_common::sensitive_log_masks::register_secret_and_fields_for_all_running_jobs( + &r, + ); + } else { + windmill_common::sensitive_log_masks::register_secret_for_all_running_jobs(&r); + } } // Cache the result when explicitly allowed. Secrets are cached too: their per-read side // effects (audit + running-job registration) are re-run on a hit (see the hit path above), // and `is_secret` is stored so the hit knows to do so. - if let Some(identity) = cache_identity.as_deref() { + if let Some(identity) = cache_identity.as_deref().filter(|_| cacheable) { cache_variable( &w_id, &path, diff --git a/cli/src/commands/variable/variable.ts b/cli/src/commands/variable/variable.ts index e9c3183ea9..803a6031bf 100644 --- a/cli/src/commands/variable/variable.ts +++ b/cli/src/commands/variable/variable.ts @@ -111,15 +111,17 @@ export interface VariableFile { * ciphertext (AES-256-GCM or legacy AES-256-CBC, base64) is standard base64 * decoding to a non-zero * multiple of the 16-byte block size. External secret-backend markers - * ($vault:/$aws_sm:/$azure_kv:/$http_store:) are stored verbatim too, so they - * count as already-encrypted. This is a shape check only — it never decrypts. + * ($vault:/$aws_sm:/$azure_kv:/$http_store:) and `$external:` references to a secret in + * an external store are stored verbatim too, so they count as already-encrypted. + * This is a shape check only — it never decrypts. */ export function looksLikeWorkspaceCiphertext(value: string): boolean { if ( value.startsWith("$vault:") || value.startsWith("$aws_sm:") || value.startsWith("$azure_kv:") || - value.startsWith("$http_store:") + value.startsWith("$http_store:") || + value.startsWith("$external:") ) { return true; } diff --git a/cli/test/variable_ciphertext_shape_unit.test.ts b/cli/test/variable_ciphertext_shape_unit.test.ts index 0113929164..a05d4d225d 100644 --- a/cli/test/variable_ciphertext_shape_unit.test.ts +++ b/cli/test/variable_ciphertext_shape_unit.test.ts @@ -19,6 +19,7 @@ test("treats workspace-ciphertext-shaped values as already-encrypted", () => { "$vault:f/x/cfg", "$aws_sm:f/x/cfg", "$azure_kv:f/x/cfg", + "$external:vault://secret/team/app#token", ]; for (const value of ciphertextShaped) { expect(looksLikeWorkspaceCiphertext(value)).toBe(true); diff --git a/cli/windmill-utils-internal/src/deploy.ts b/cli/windmill-utils-internal/src/deploy.ts index 3478fa8b0b..77d093f74f 100644 --- a/cli/windmill-utils-internal/src/deploy.ts +++ b/cli/windmill-utils-internal/src/deploy.ts @@ -602,11 +602,19 @@ export async function deployItem( } } } else if (kind === "variable") { - const variable = await provider.getVariable({ + // An external secret reference is copied as the reference, never as the value it resolves to. + let variable = await provider.getVariable({ workspace: workspaceFrom, path, - decryptSecret: true, + decryptSecret: false, }); + if (variable.is_secret && !variable.value?.startsWith("$external:")) { + variable = await provider.getVariable({ + workspace: workspaceFrom, + path, + decryptSecret: true, + }); + } if (alreadyExists) { await provider.updateVariable({ workspace: workspaceTo, diff --git a/frontend/src/lib/components/CompareWorkspaces.svelte b/frontend/src/lib/components/CompareWorkspaces.svelte index 06b691679c..4ce7fb1504 100644 --- a/frontend/src/lib/components/CompareWorkspaces.svelte +++ b/frontend/src/lib/components/CompareWorkspaces.svelte @@ -17,6 +17,7 @@ UserPlus } from 'lucide-svelte' import { onDestroy, untrack } from 'svelte' + import { isExternalRef } from '$lib/externalSecretRef' import type { CiTestResult } from '$lib/gen' import { Alert, Badge } from './common' import { @@ -285,7 +286,11 @@ } }) } else { - const v = await VariableService.getVariable({ workspace: from, path, decryptSecret: true }) + // An external secret reference is copied as the reference, never as its resolved value. + let v = await VariableService.getVariable({ workspace: from, path, decryptSecret: false }) + if (v.is_secret && !isExternalRef(v.value)) { + v = await VariableService.getVariable({ workspace: from, path, decryptSecret: true }) + } await VariableService.createVariable({ workspace: to, requestBody: { diff --git a/frontend/src/lib/components/InstanceSetting.svelte b/frontend/src/lib/components/InstanceSetting.svelte index 4a7c614b0b..92e31bcf31 100644 --- a/frontend/src/lib/components/InstanceSetting.svelte +++ b/frontend/src/lib/components/InstanceSetting.svelte @@ -26,6 +26,7 @@ import WorkspaceRolesSetting from './instanceSettings/WorkspaceRolesSetting.svelte' import SmtpSettings from './instanceSettings/SmtpSettings.svelte' import SecretBackendConfig from './instanceSettings/SecretBackendConfig.svelte' + import ExternalSecretsConfig from './instanceSettings/ExternalSecretsConfig.svelte' import ExternalInstancePgSettings from './instanceSettings/ExternalInstancePgSettings.svelte' import InstancePgSettings from './instanceSettings/InstancePgSettings.svelte' import GhesAppSettings from './instanceSettings/GhesAppSettings.svelte' @@ -879,6 +880,8 @@ {:else if setting.fieldType == 'secret_backend'} + {:else if setting.fieldType == 'external_secrets'} + {:else if setting.fieldType == 'external_instance_pg'} void /** Whether the deployed variable being edited is secret. */ deployedSecret?: boolean + /** Whether the deployed variable being edited is an external reference. */ + deployedRef?: boolean /** Workspace the path is validated against; defaults to the nav workspace. */ workspace?: string | undefined /** The user acting in `workspace`, resolved by the editor above. `undefined` while @@ -67,6 +73,7 @@ onLoadSecret, onSecretChange, deployedSecret = false, + deployedRef = false, workspace = undefined, actingUser, oauthLinked = false @@ -96,11 +103,43 @@ let hasStagedValue = $derived(variable.value !== '') // Un-securing has to send the plaintext, and fetching it silently on the toggle would - // reveal the secret without the explicit, audited "Load secret value" action. - let unsecureNeedsLoad = $derived(edit && deployedSecret && variable.is_secret && !hasStagedValue) + // reveal the secret without the explicit, audited "Load secret value" action. A saved + // reference holds no value to load: whatever is typed, even nothing, replaces it. + let unsecureNeedsLoad = $derived( + edit && deployedSecret && !deployedRef && variable.is_secret && !hasStagedValue + ) + + // The API refuses anyone else creating, repointing or removing a reference. + let canEditRef = $derived(can_write && !!actingUser?.is_admin) const MAX_VARIABLE_LENGTH = 10000 + let externalRef = $derived(variable.is_secret && isExternalRef(variable.value)) + let vaultRef = $derived(parseVaultRef(variable.value)) + let kind = $derived(externalRef ? 'ref' : variable.is_secret ? 'secret' : 'plain') + let secretLocked = $derived( + edit && (!actingUser || !caps.code || isEncryptedDraftValue(variable.value)) + ) + + function setKind(next: string | undefined) { + if (!next || next === kind) return + if (kind === 'ref') variable.value = '' + if (variable.is_secret !== (next !== 'plain')) { + variable.is_secret = next !== 'plain' + onSecretChange?.(variable.is_secret) + } + if (next === 'ref') variable.value = formatVaultRef('', '') + } + let allowedPrefixes = resource( + () => (canEditRef ? ws : undefined), + async (ws) => + ws ? await VariableService.listExternalSecretPrefixes({ workspace: ws }) : undefined + ) + let pathPlaceholder = $derived( + (allowedPrefixes.current?.[0]?.replace(/^vault:\/\//, '').replace(/\/*$/, '') ?? + 'secret/team') + '/app' + ) + let editorKind: 'plain' | 'json' | 'yaml' = $state('plain') let editor: any = $state(undefined) @@ -123,29 +162,66 @@ /> -
- Secret + Type
- onSecretChange?.(e.detail)} - bind:checked={variable.is_secret} - disabled={unsecureNeedsLoad || - (edit && (!actingUser || !caps.code || isEncryptedDraftValue(variable.value)))} - /> - {#if unsecureNeedsLoad && can_write && caps.code} - - Load the secret value or enter a new one to make it non-secret - - {/if} + + {#snippet children({ item })} + + + + {#if externalRef || (canEditRef && allowedPrefixes.current?.length)} + + {/if} + {/snippet} +
- {#if variable.is_secret} + {#if externalRef && can_write && !canEditRef} + + Only workspace admins can create or change a reference + + {/if} + {#if deployedRef && kind !== 'ref' && !hasStagedValue} + Enter a value to replace the Vault reference + {:else if unsecureNeedsLoad && can_write && caps.code} + + Load the secret value or enter a new one to make it non-secret + + {/if} + {#if externalRef} + + Only the reference is stored in Windmill. The value is read from HashiCorp Vault each time the + variable is read, and every read generates an audit log whose operation name is: + variables.decrypt_secret + + {:else if variable.is_secret} Every secret is encrypted at rest and in transit with a key specific to this workspace. In addition, any read of a secret variable generates an audit log whose operation name is: @@ -163,96 +239,142 @@ stretch the hit area over the whole row and the alert. The toggle carries its na {/if} -
-