feat: reference existing HashiCorp Vault secrets from secret variables (#12078)

* feat: reference existing HashiCorp Vault secrets from secret variables

A secret variable can hold `$external:vault://<path>#<key>` instead of a
value. The reference is resolved from Vault each time the variable is
read, so the value is never stored in Windmill. An instance-level
allow-list of path prefixes, optionally per workspace, is checked when a
reference is saved and again when it is resolved. Other schemes are
parsed and refused as unsupported backends.

Adds AppRole as a Vault auth method, an "External reference" toggle in
the variable editor and an instance settings section for the connection
and the allow-list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: point ee-repo-ref at the external secret references branch

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep external secret references intact on delete, deploy and export

- Deleting a reference no longer runs the managed backend cleanup, which
  could delete the secret it points at.
- Deploying a variable to another workspace copies the reference, which
  the target's allow-list then checks, instead of its resolved value.
- A plain-secrets workspace export writes references verbatim.
- Reference paths reject backslashes, `#` and control characters, which
  the URL parser would otherwise normalize past the allow-list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: skip external secret references in key rotation and the read cache

- Workspace key rotation no longer tries to decrypt references.
- `get_value?allow_cache=true` does not cache a resolved reference, so
  Vault rotation and allow-list changes apply on the next read.
- `VaultSettings.mount_path` is optional in the OpenAPI spec, matching
  its serde default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: keep the OSS Vault reader stub public and regenerate workspace tool schemas

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: restrict external secret references to workspace admins and keep them out of the managed Vault storage

Creating, repointing or removing a reference now takes a workspace admin, on
create and update including already-encrypted pushes and cross-workspace
deploys; the variable form disables the toggle and fields for anyone else.

A reference into the storage of Windmill's own Vault secret backend (same
address, namespace folded into the path) is refused on save and on resolve,
whatever the allow-list says.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: compare external secret references as written in workspace comparison

Two references that currently resolve to the same value still point at
different secrets, so comparing their resolved values hid a repoint from the
diff and from deployment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: let external secret references reuse the Vault backend connection

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: hide the external reference toggle from non-admins unless already set

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: read whole Vault secrets, show allowed paths, and pick the variable type in one control

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: mask JSON fields only for whole Vault secrets, and skip PEM armor and JSON lines

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: align the reference editor and settings warning with the form and alert guidelines

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat: let a saved reference become non-secret directly, and warn when no prefix is allowed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: save an empty value when a reference is switched to another type

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: keep a saved reference's baseline through type round trips

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: drop an unused binding in the reference parser

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix: offer vault reference only when a vault path is allowed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore: update ee-repo-ref to c0fd22ba77e6618cea7611d72fe2e75c8855c595

This commit updates the EE repository reference after PR #932 was merged in windmill-ee-private.

Previous ee-repo-ref: 56cba4e7771e7a87e56a67eb115549a31dd12b1a

New ee-repo-ref: c0fd22ba77e6618cea7611d72fe2e75c8855c595

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
This commit is contained in:
hugocasa
2026-10-30 20:23:27 +01:00
committed by GitHub
co-authored by Claude Opus 5.5 Ruben Fiszel windmill-internal-app[bot]
parent 6444b20abd
commit 6f4124e9a0
36 changed files with 1462 additions and 215 deletions
@@ -0,0 +1,23 @@
{
"db_name": "PostgreSQL",
"query": "SELECT is_secret AND value NOT LIKE '$external:%' AS \"is_secret!\" FROM variable WHERE path = $1 AND workspace_id = $2",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "is_secret!",
"type_info": "Bool"
}
],
"parameters": {
"Left": [
"Text",
"Text"
]
},
"nullable": [
null
]
},
"hash": "3e0f2e6febdf289aeccbea6b3258be507e8c2fa86c8006bb32b5c0660497c128"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT path FROM variable WHERE path = ANY($1) AND workspace_id = $2 AND is_secret = true",
"query": "SELECT path FROM variable WHERE path = ANY($1) AND workspace_id = $2 AND is_secret = true AND value NOT LIKE '$external:%'",
"describe": {
"columns": [
{
@@ -19,5 +19,5 @@
false
]
},
"hash": "0c8a3eb810c96230ba3a5466c55bf24a94eb8a52ceb82cc29dade173ad87569d"
"hash": "6c9efac7fba0e3f2f34510bb6e2a1757af5286183faf888c7313f3a657705850"
}
@@ -1,6 +1,6 @@
{
"db_name": "PostgreSQL",
"query": "SELECT workspace_id, path FROM variable WHERE is_secret = true",
"query": "SELECT workspace_id, path FROM variable WHERE is_secret = true AND value NOT LIKE '$external:%'",
"describe": {
"columns": [
{
@@ -22,5 +22,5 @@
false
]
},
"hash": "0600f2a9179f83502c6b13e8e4284f85ca82636f274f5dce47da5a8320a60088"
"hash": "e2c7dcf549dba16f2a36d6495b4c0a550a03af01798dacce24d42fbe01b51f36"
}
+1 -1
View File
@@ -1 +1 @@
a445ac5e719cc0bb2bc22bb04bb871b00b1bc808
c0fd22ba77e6618cea7611d72fe2e75c8855c595
@@ -111,7 +111,7 @@ pub(crate) async fn connection_snapshot(
.fetch_optional(&mut *conn)
.await?;
if let Some((stored, true)) = &row {
if windmill_common::secret_backend::is_external_stored_value(stored) {
if windmill_common::secret_backend::resolves_externally(stored) {
let secret = windmill_common::secret_backend::get_secret_value(
db, w_id, var, stored,
)
@@ -77,7 +77,8 @@ use serde::{Deserialize, Serialize};
use sqlx::{FromRow, Postgres, Row, Transaction};
use windmill_common::oauth2::InstanceEvent;
use windmill_common::secret_backend::{
get_secret_backend, is_external_stored_value, is_vault_backend_configured,
get_secret_backend, is_external_ref_value, is_external_stored_value,
is_vault_backend_configured,
};
use windmill_common::utils::not_found_if_none;
@@ -6772,7 +6773,9 @@ async fn set_encryption_key(
.await?;
for variable in all_variables {
if !variable.is_secret {
if !variable.is_secret
|| windmill_common::secret_backend::resolves_externally(&variable.value)
{
continue;
}
let decrypted_value =
@@ -14935,8 +14938,12 @@ async fn compare_two_variables(
{
// A secret's ciphertext is randomized on every write, so equal secrets are compared
// by plaintext, each decrypted with its own workspace's key. An external backend
// reference is the same text on both sides but names a per-workspace entry.
// reference is the same text on both sides but names a per-workspace entry. An
// `$external:` reference is compared as written: two references resolving to equal
// values still point at different secrets.
has_changes = !source.is_secret
|| is_external_ref_value(&source.value)
|| is_external_ref_value(&target.value)
|| !matches!(
tokio::try_join!(
decrypt_secret_value(db, source_workspace_id, path, source.value.clone()),
+33 -4
View File
@@ -7828,6 +7828,24 @@ paths:
schema:
type: string
/w/{workspace}/variables/external_secret_prefixes:
get:
summary: list the external secret prefixes allowed for this workspace (admin only)
operationId: listExternalSecretPrefixes
tags:
- variable
parameters:
- $ref: "#/components/parameters/WorkspaceId"
responses:
"200":
description: allowed prefixes, such as vault://secret/team
content:
application/json:
schema:
type: array
items:
type: string
/w/{workspace}/variables/delete/{path}:
delete:
summary: delete variable
@@ -29998,14 +30016,13 @@ components:
type: object
required:
- address
- mount_path
properties:
address:
type: string
description: HashiCorp Vault server address (e.g., https://vault.company.com:8200)
mount_path:
type: string
description: KV v2 secrets engine mount path (e.g., windmill)
description: KV v2 secrets engine mount path (e.g., windmill). Empty to test the connection used for `$external:vault://` references, which checks only that Vault accepts Windmill's token.
kv_secret_path_prefix:
type: string
description: Optional path prefix inserted between the KV data/metadata segment and the workspace id (e.g., "apps/windmill"). When set, secrets are stored at `<mount>/data/<prefix>/<workspace>/<secret>`, allowing a Vault policy scoped to exactly `<mount>/data/<prefix>/*`.
@@ -30015,6 +30032,12 @@ components:
jwt_mount_path:
type: string
description: Mount path for the JWT auth method in Vault (optional, defaults to "jwt"). Set this when the JWT auth method is mounted at a non-default path, e.g. via `vault auth enable -path=<mount> jwt`.
approle_role_id:
type: string
description: AppRole role_id (optional). When set and no static token is given, Windmill logs in with AppRole; the secret_id is read from the `VAULT_APPROLE_SECRET_ID` environment variable of servers and workers.
approle_mount_path:
type: string
description: Mount path of the AppRole auth method (optional, defaults to "approle").
namespace:
type: string
description: Vault Enterprise namespace (optional)
@@ -32890,7 +32913,13 @@ components:
description: The path to the variable
value:
type: string
description: The value of the variable
description: >
The value of the variable. For a secret, `$external:vault://<path>#<key>`
stores a reference to an existing secret in the instance's external
HashiCorp Vault (path as `vault kv get` takes it) instead of a value; without
`#<key>` it reads the whole secret as a JSON object of its fields. It is
resolved on each read, only under the path prefixes a superadmin allowed for
the workspace, and is returned as-is when the secret is not decrypted.
is_secret:
type: boolean
description: Whether the variable is a secret
@@ -32939,7 +32968,7 @@ components:
description: The path to the variable
value:
type: string
description: The new value of the variable
description: The new value of the variable. Accepts an `$external:vault://<path>[#<key>]` reference for a secret, as in CreateVariable.
is_secret:
type: boolean
description: Whether the variable is a secret
@@ -986,6 +986,10 @@ pub(crate) async fn tarball_workspace(
if plain_secret.or(plain_secrets).unwrap_or(false)
&& var.value.is_some()
&& var.is_secret
&& !var
.value
.as_deref()
.is_some_and(windmill_common::secret_backend::is_external_ref_value)
{
var.value = Some(decrypt(&mc, var.value.unwrap()).map_err(|e| {
Error::internal_err(format!("Error decrypting variable {}: {}", var.path, e))
@@ -180,6 +180,7 @@ pub const APP_WORKSPACED_ROUTE_SETTING: &str = "app_workspaced_route";
pub const HTTP_ROUTE_WORKSPACED_ROUTE_SETTING: &str = "http_route_workspaced_route";
pub const HTTP_ROUTE_DEFAULT_ALLOWED_ORIGINS_SETTING: &str = "http_route_default_allowed_origins";
pub const SECRET_BACKEND_SETTING: &str = "secret_backend";
pub const EXTERNAL_SECRETS_SETTING: &str = "external_secrets";
pub const MIN_KEEP_ALIVE_VERSION_SETTING: &str = "min_keep_alive_version";
pub const GITHUB_ENTERPRISE_APP_SETTING: &str = "github_enterprise_app";
/// Base URL GitHub delivers git-sync repository webhooks to. Falls back to
@@ -516,6 +517,7 @@ pub const AGENT_WORKER_BLOCKED_SETTINGS: &[&str] = &[
SCIM_TOKEN_SETTING,
SAML_METADATA_SETTING,
SECRET_BACKEND_SETTING,
EXTERNAL_SECRETS_SETTING,
GITHUB_ENTERPRISE_APP_SETTING,
OBJECT_STORE_CONFIG_SETTING,
AI_CONFIG_SETTING,
@@ -1435,6 +1437,7 @@ mod tests {
SCIM_TOKEN_SETTING,
SAML_METADATA_SETTING,
SECRET_BACKEND_SETTING,
EXTERNAL_SECRETS_SETTING,
GITHUB_ENTERPRISE_APP_SETTING,
OBJECT_STORE_CONFIG_SETTING,
AI_CONFIG_SETTING,
@@ -1241,7 +1241,10 @@ fn mask_nested_sensitive(key: &str, value: &serde_json::Value) -> serde_json::Va
}
}
// Settings that are maps-of-objects where each child has a sensitive sub-field.
const NESTED_MAP_SENSITIVE: &[(&str, &[&str])] = &[("oauths", &["secret", "signing_secret"])];
const NESTED_MAP_SENSITIVE: &[(&str, &[&str])] = &[
("oauths", &["secret", "signing_secret"]),
("external_secrets", &["token"]),
];
for &(parent_key, child_fields) in NESTED_MAP_SENSITIVE {
if key == parent_key {
if let serde_json::Value::Object(entries) = value {
@@ -0,0 +1,359 @@
/*
* Author: Ruben Fiszel
* Copyright: Windmill Labs, Inc 2026
* This file and its contents are licensed under the AGPLv3 License.
* Please see the included NOTICE for copyright information and
* LICENSE-AGPL for a copy of the license.
*/
//! References from a secret variable to a secret that already exists in an
//! external store: the variable stores `$external:<scheme>://<path>#<key>` and
//! the value is read from the store each time the variable is read. Unlike the
//! `$vault:`/`$aws_sm:`/`$azure_kv:` markers of a managed secret backend,
//! nothing is ever written to the store. Only `vault://` resolves, with `<path>`
//! written as `vault kv get` takes it.
//!
//! The instance's identity in the store can usually read far more than any one
//! workspace should, so a reference resolves only under a prefix a superadmin
//! allowed for that workspace in the `external_secrets` setting. The check runs
//! on save and again on every read, so narrowing the list takes effect on
//! existing references.
use serde::{Deserialize, Serialize};
use crate::{
db::DB,
error::{Error, Result},
global_settings::{
load_value_from_global_settings, EXTERNAL_SECRETS_SETTING, SECRET_BACKEND_SETTING,
},
secret_backend::{SecretBackendConfig, VaultSettings},
};
pub const EXTERNAL_REF_MARKER: &str = "$external:";
/// The `external_secrets` instance setting: the connections references are
/// read through, and where they may point.
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Default)]
pub struct ExternalSecretsSettings {
#[serde(default, skip_serializing_if = "Option::is_none")]
pub vault: Option<VaultSettings>,
#[serde(default)]
pub allowed: Vec<ExternalSecretAllowedPrefix>,
}
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
pub struct ExternalSecretAllowedPrefix {
/// `<scheme>://<path prefix>`, the path matched on whole segments.
pub prefix: String,
/// Workspaces that may reference under `prefix`; empty means all.
#[serde(default)]
pub workspaces: Vec<String>,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ExternalSecretBackend {
Vault,
}
impl ExternalSecretBackend {
fn scheme(self) -> &'static str {
match self {
Self::Vault => "vault",
}
}
fn from_scheme(scheme: &str) -> Result<Self> {
match scheme {
"vault" => Ok(Self::Vault),
_ => Err(Error::BadRequest(format!(
"Unsupported external secret backend `{scheme}://`: only `vault://` is supported"
))),
}
}
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ExternalSecretRef {
pub backend: ExternalSecretBackend,
/// Normalized: no surrounding or doubled slashes.
pub path: String,
/// `None` reads the whole secret, as a JSON object of its fields.
pub key: Option<String>,
}
pub fn is_external_ref_value(value: &str) -> bool {
value.starts_with(EXTERNAL_REF_MARKER)
}
/// A stored secret value whose plaintext is not workspace ciphertext but lives
/// in an external store: a managed backend entry or an `$external:` reference.
pub fn resolves_externally(value: &str) -> bool {
super::is_external_stored_value(value) || is_external_ref_value(value)
}
fn path_segments(path: &str) -> Result<Vec<&str>> {
let segments: Vec<&str> = path.trim().trim_matches('/').split('/').collect();
// The URL parser treats `\` as `/` and drops tabs and newlines, so either would let a
// `..` segment slip past this check and the allow-list.
if segments.iter().any(|s| {
s.is_empty()
|| *s == "."
|| *s == ".."
|| s.contains(['?', '%', '#', '\\'])
|| s.contains(char::is_control)
}) {
return Err(Error::BadRequest(format!(
"Invalid external secret path `{path}`: expected segments like `secret/team/app`"
)));
}
Ok(segments)
}
fn split_scheme(uri: &str) -> Result<(ExternalSecretBackend, &str)> {
let (scheme, rest) = uri.trim().split_once("://").ok_or_else(|| {
Error::BadRequest(format!(
"Invalid external secret reference `{uri}`: expected `<scheme>://<path>[#<key>]`, e.g. `vault://secret/team/app#token`"
))
})?;
Ok((ExternalSecretBackend::from_scheme(scheme)?, rest))
}
/// Parses `$external:<scheme>://<path>[#<key>]` (the marker is optional).
pub fn parse_external_ref(value: &str) -> Result<ExternalSecretRef> {
let uri = value.strip_prefix(EXTERNAL_REF_MARKER).unwrap_or(value);
let (backend, rest) = split_scheme(uri)?;
let (path, key) = match rest.split_once('#') {
Some((_, key)) if key.trim().is_empty() => {
return Err(Error::BadRequest(format!(
"Invalid external secret reference `{uri}`: the key after `#` is empty"
)))
}
Some((path, key)) => (path, Some(key.trim().to_string())),
None => (rest, None),
};
Ok(ExternalSecretRef { backend, path: path_segments(path)?.join("/"), key })
}
fn applies_to(a: &ExternalSecretAllowedPrefix, w_id: &str) -> bool {
a.workspaces.is_empty() || a.workspaces.iter().any(|w| w == w_id)
}
/// The prefixes `w_id` may reference, as a superadmin wrote them.
pub async fn allowed_prefixes(db: &DB, w_id: &str) -> Result<Vec<String>> {
Ok(load_settings(db)
.await?
.allowed
.into_iter()
.filter(|a| applies_to(a, w_id))
.map(|a| a.prefix)
.collect())
}
fn is_allowed(settings: &ExternalSecretsSettings, w_id: &str, r: &ExternalSecretRef) -> bool {
let path: Vec<&str> = r.path.split('/').collect();
settings.allowed.iter().any(|a| {
applies_to(a, w_id)
&& split_scheme(&a.prefix).is_ok_and(|(backend, prefix)| {
backend == r.backend && path_segments(prefix).is_ok_and(|p| path.starts_with(&p))
})
})
}
async fn load_settings(db: &DB) -> Result<ExternalSecretsSettings> {
match load_value_from_global_settings(db, EXTERNAL_SECRETS_SETTING).await? {
Some(value) => serde_json::from_value(value).map_err(|e| {
Error::internal_err(format!("Invalid {EXTERNAL_SECRETS_SETTING} setting: {e}"))
}),
None => Ok(ExternalSecretsSettings::default()),
}
}
fn vault_segments(s: Option<&str>) -> Vec<&str> {
s.unwrap_or_default()
.split('/')
.filter(|s| !s.trim().is_empty())
.collect()
}
/// Whether `path`, read through `conn`, lands in the storage of the managed Vault
/// backend `managed`. The namespace is folded into the path because Vault also
/// takes a namespace as a path prefix, so `ns/kv/x` with no namespace header
/// reaches the same secret as `kv/x` under namespace `ns`.
fn is_managed_storage(managed: &VaultSettings, conn: &VaultSettings, path: &str) -> bool {
let address = |s: &VaultSettings| s.address.trim().trim_end_matches('/').to_lowercase();
let mount = vault_segments(Some(&managed.mount_path));
if mount.is_empty() || address(managed) != address(conn) {
return false;
}
let mut storage = vault_segments(managed.namespace.as_deref());
storage.extend(mount);
storage.extend(vault_segments(managed.kv_secret_path_prefix.as_deref()));
let mut target = vault_segments(conn.namespace.as_deref());
target.extend(vault_segments(Some(path)));
target.starts_with(&storage)
}
/// Parses a reference, checks it is allowed for `w_id`, and returns the Vault
/// connection to read it through: the `external_secrets` one, or else the
/// managed backend's when that is Vault.
async fn authorize(
db: &DB,
w_id: &str,
value: &str,
) -> Result<(Option<VaultSettings>, ExternalSecretRef)> {
let r = parse_external_ref(value)?;
let settings = load_settings(db).await?;
if !is_allowed(&settings, w_id, &r) {
let allowed: Vec<_> = settings
.allowed
.iter()
.filter(|a| applies_to(a, w_id))
.map(|a| format!("`{}/**`", a.prefix.trim_end_matches('/')))
.collect();
let target = format!("External secret `{}://{}`", r.backend.scheme(), r.path);
return Err(Error::PermissionDenied(if allowed.is_empty() {
format!("{target} is not allowed: no Vault paths are allowed for workspace {w_id} (instance settings > Secret storage)")
} else {
format!(
"{target} is outside the paths allowed for workspace {w_id}: {}",
allowed.join(", ")
)
}));
}
let managed = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING)
.await?
.and_then(|v| serde_json::from_value(v).ok())
{
Some(SecretBackendConfig::HashiCorpVault(managed)) => Some(managed),
_ => None,
};
let conn = settings.vault.or_else(|| managed.clone());
// The managed backend's storage holds every workspace's secrets, so no
// allow-list entry may open it.
if let (Some(managed), Some(conn)) = (&managed, &conn) {
if is_managed_storage(managed, conn, &r.path) {
return Err(Error::PermissionDenied(format!(
"External secret `vault://{}` points into the storage of Windmill's own Vault secret backend",
r.path
)));
}
}
Ok((conn, r))
}
/// Write-time check of an `$external:` value. Does not contact the store, so a
/// reference may be saved before its secret exists.
pub async fn check_external_ref(db: &DB, w_id: &str, value: &str) -> Result<()> {
authorize(db, w_id, value).await.map(|_| ())
}
pub(crate) async fn resolve_external_ref(
db: &DB,
w_id: &str,
variable_path: &str,
value: &str,
) -> Result<String> {
let (conn, r) = authorize(db, w_id, value).await?;
let read = match r.backend {
ExternalSecretBackend::Vault => {
let vault = conn.as_ref().ok_or_else(|| {
Error::BadRequest(
"No Vault connection is configured for external secrets (instance settings > Secret storage)"
.to_string(),
)
})?;
super::read_vault_external_secret(db, vault, &r.path, r.key.as_deref()).await
}
};
read.map_err(|e| match e {
Error::NotFound(m) => Error::NotFound(format!("{m} (variable {variable_path})")),
Error::PermissionDenied(m) => {
Error::PermissionDenied(format!("{m} (variable {variable_path})"))
}
e => e,
})
}
#[cfg(test)]
mod tests {
use super::*;
fn settings(allowed: &[(&str, &[&str])]) -> ExternalSecretsSettings {
serde_json::from_value(serde_json::json!({
"allowed": allowed
.iter()
.map(|(p, w)| serde_json::json!({"prefix": p, "workspaces": w}))
.collect::<Vec<_>>(),
}))
.unwrap()
}
fn allowed(s: &ExternalSecretsSettings, w_id: &str, value: &str) -> bool {
is_allowed(s, w_id, &parse_external_ref(value).unwrap())
}
#[test]
fn parses_references() {
assert_eq!(
parse_external_ref("$external:vault:///secret/team/app/#token").unwrap(),
ExternalSecretRef {
backend: ExternalSecretBackend::Vault,
path: "secret/team/app".to_string(),
key: Some("token".to_string())
}
);
assert_eq!(
parse_external_ref("$external:vault://secret/team/app")
.unwrap()
.key,
None
);
for bad in [
"$external:secret/app#k",
"$external:vault://secret/app#",
"$external:vault://secret//app#k",
"$external:vault://secret/../sys#k",
"$external:vault://secret/team/..\\other/app#k",
"$external:vault://secret/app?list=true#k",
"$external:aws-sm://secret/app#k",
] {
assert!(parse_external_ref(bad).is_err(), "{bad}");
}
}
#[test]
fn allow_list_matches_whole_segments_and_workspaces() {
let s = settings(&[
("vault://secret/team", &["ws1"]),
("vault://kv/shared/", &[]),
]);
assert!(allowed(&s, "ws1", "vault://secret/team/app#k"));
assert!(allowed(&s, "ws1", "vault://secret/team#k"));
assert!(!allowed(&s, "ws1", "vault://secret/team-other/app#k"));
assert!(!allowed(&s, "ws2", "vault://secret/team/app#k"));
assert!(allowed(&s, "ws2", "vault://kv/shared/x#k"));
assert!(!allowed(&s, "ws2", "vault://kv#k"));
}
#[test]
fn refuses_the_managed_backend_storage() {
let vault = |address: &str, mount: &str, prefix: Option<&str>, ns: Option<&str>| {
serde_json::from_value::<VaultSettings>(serde_json::json!({
"address": address, "mount_path": mount,
"kv_secret_path_prefix": prefix, "namespace": ns,
}))
.unwrap()
};
let managed = vault("https://vault:8200", "kv", Some("/apps/wm/"), Some("ns"));
let same = vault("https://vault:8200/", "", None, Some("ns"));
assert!(is_managed_storage(&managed, &same, "kv/apps/wm/ws1/f/x"));
assert!(is_managed_storage(&managed, &same, "kv/apps/wm"));
assert!(!is_managed_storage(&managed, &same, "kv/apps/other/x"));
assert!(!is_managed_storage(&managed, &same, "secret/apps/wm/x"));
let root = vault("https://vault:8200", "", None, None);
assert!(is_managed_storage(&managed, &root, "ns/kv/apps/wm/ws1/x"));
let other = vault("https://other:8200", "", None, Some("ns"));
assert!(!is_managed_storage(&managed, &other, "kv/apps/wm/ws1/x"));
}
}
@@ -14,8 +14,10 @@
pub mod database;
pub mod resolver;
pub mod external_ref;
pub use resolver::*;
pub use external_ref::*;
#[cfg(feature = "private")]
pub mod vault_ee;
@@ -138,7 +140,9 @@ impl Default for SecretBackendConfig {
pub struct VaultSettings {
/// Vault server address (e.g., "https://vault.company.com:8200")
pub address: String,
/// KV v2 mount path (e.g., "windmill")
/// KV v2 mount path (e.g., "windmill"). Empty for the external-secrets
/// connection, which reads whatever mount a reference names.
#[serde(default)]
pub mount_path: String,
/// Optional path prefix inserted between the KV `data`/`metadata` segment
/// and the workspace id, e.g. "apps/windmill". When set, secrets live at
@@ -156,6 +160,15 @@ pub struct VaultSettings {
/// e.g. via `vault auth enable -path=my-mount jwt`.
#[serde(skip_serializing_if = "Option::is_none")]
pub jwt_mount_path: Option<String>,
/// AppRole role_id. When set (and no static token), Windmill logs in with
/// AppRole; the secret_id is read from the `VAULT_APPROLE_SECRET_ID`
/// environment variable of servers and workers, never from settings, and
/// is omitted when unset (a role with `bind_secret_id=false`).
#[serde(skip_serializing_if = "Option::is_none")]
pub approle_role_id: Option<String>,
/// Mount path of the AppRole auth method (defaults to "approle").
#[serde(skip_serializing_if = "Option::is_none")]
pub approle_mount_path: Option<String>,
/// Vault Enterprise namespace (optional)
#[serde(skip_serializing_if = "Option::is_none")]
pub namespace: Option<String>,
@@ -298,6 +298,9 @@ pub async fn get_secret_value(
path: &str,
encrypted_value: &str,
) -> Result<String> {
if super::is_external_ref_value(encrypted_value) {
return super::resolve_external_ref(db, workspace_id, path, encrypted_value).await;
}
let backend = get_secret_backend(db).await?;
match backend.backend_name() {
@@ -28,6 +28,8 @@ mod tests {
kv_secret_path_prefix: None,
jwt_role: Some("windmill-secrets".to_string()),
jwt_mount_path: None,
approle_role_id: None,
approle_mount_path: None,
namespace: None,
token: Some("test-root-token".to_string()),
skip_ssl_verify: None,
@@ -129,3 +129,15 @@ pub async fn generate_vault_jwt(_db: &DB, _vault_address: &str) -> Result<String
"HashiCorp Vault integration requires Enterprise Edition".to_string(),
))
}
/// Read `key` of the secret at `path`, or all of it, for an `$external:vault://` reference (OSS stub)
pub async fn read_vault_external_secret(
_db: &DB,
_settings: &VaultSettings,
_path: &str,
_key: Option<&str>,
) -> Result<String> {
Err(Error::internal_err(
"HashiCorp Vault integration requires Enterprise Edition".to_string(),
))
}
@@ -220,6 +220,31 @@ pub fn unregister_running_job(job_id: Uuid) {
}
}
/// `secret` and, when it spans several lines, each of them: `handle_child` masks a log
/// line by line, so a multi-line secret such as a PEM key never matches as a whole. Masks
/// apply to every running job, so lines any job may print are left out: PEM armor, and the
/// lines of a JSON document (`"type": "service_account",`).
fn mask_patterns(secret: &str) -> impl Iterator<Item = &str> {
let split = secret.contains('\n') && !secret.trim_start().starts_with(['{', '[']);
let lines = split.then(|| {
secret
.lines()
.map(str::trim)
.filter(|l| !l.starts_with("-----"))
});
std::iter::once(secret)
.chain(lines.into_iter().flatten())
.filter(|s| s.len() >= MIN_SECRET_LENGTH)
}
fn insert_masks(job: &mut JobMasks, secret: &str) {
for pattern in mask_patterns(secret) {
if job.secrets.insert(pattern.to_string()) {
job.compiled = None;
}
}
}
/// Register a secret value for ALL currently running jobs.
/// Used when a secret is fetched via the embedded server (we don't know which job triggered it).
pub fn register_secret_for_all_running_jobs(secret: &str) {
@@ -236,13 +261,42 @@ pub fn register_secret_for_all_running_jobs(secret: &str) {
let mut masks = SENSITIVE_MASKS.write().unwrap_or_else(|e| e.into_inner());
for job_id in job_ids {
if let Some(job) = masks.get_mut(&job_id) {
if job.secrets.insert(secret.to_string()) {
job.compiled = None;
}
insert_masks(job, secret);
}
}
}
/// Like [`register_secret_for_all_running_jobs`], and when `secret` is a JSON object or
/// array also each string in it: `$jsonvar:` hands a job the parsed value, which can then
/// print one field on its own. Only for values whose every field is secret, such as a whole
/// Vault secret: masks apply to every running job, and a JSON variable's other fields
/// (`"user": "postgres"`) would mask unrelated logs.
pub fn register_secret_and_fields_for_all_running_jobs(secret: &str) {
register_secret_for_all_running_jobs(secret);
for field in json_strings(secret) {
register_secret_for_all_running_jobs(&field);
}
}
/// Every string inside `secret` when it is a JSON object or array.
fn json_strings(secret: &str) -> Vec<String> {
fn walk(v: serde_json::Value, out: &mut Vec<String>) {
match v {
serde_json::Value::String(s) => out.push(s),
serde_json::Value::Array(a) => a.into_iter().for_each(|v| walk(v, out)),
serde_json::Value::Object(o) => o.into_iter().for_each(|(_, v)| walk(v, out)),
_ => {}
}
}
let mut out = vec![];
if secret.trim_start().starts_with(['{', '[']) {
if let Ok(v) = serde_json::from_str(secret) {
walk(v, &mut out);
}
}
out
}
/// Register a secret value for a specific job.
/// Used for the job's own token and for `$encrypted:` args, where we know the job ID.
pub fn register_secret_for_job(job_id: Uuid, secret: &str) {
@@ -251,9 +305,7 @@ pub fn register_secret_for_job(job_id: Uuid, secret: &str) {
}
let mut masks = SENSITIVE_MASKS.write().unwrap_or_else(|e| e.into_inner());
if let Some(job) = masks.get_mut(&job_id) {
if job.secrets.insert(secret.to_string()) {
job.compiled = None;
}
insert_masks(job, secret);
}
}
@@ -281,6 +333,32 @@ mod tests {
unregister_running_job(job_id);
}
/// Logs are masked line by line, so a multi-line secret must be masked one line at a time.
#[test]
fn masks_each_line_of_a_multi_line_secret() {
let job_id = Uuid::new_v4();
register_running_job(job_id);
register_secret_for_job(
job_id,
"-----BEGIN KEY-----\nfirstlineofkey\nsecondlineofkey\n",
);
let snap = snapshot(&job_id).expect("secret registered");
for line in ["firstlineofkey", "secondlineofkey"] {
assert!(!snap.mask(line).contains(line), "{line}");
}
assert_eq!(snap.mask("-----BEGIN KEY-----"), "-----BEGIN KEY-----");
unregister_running_job(job_id);
}
#[test]
fn finds_each_string_of_a_whole_secret() {
assert_eq!(
json_strings(r#"{"a":"secretvalue1","b":["secretvalue2"],"n":12345678}"#),
["secretvalue1", "secretvalue2"]
);
assert!(json_strings("not json").is_empty());
}
/// A line-by-line sink must not repeat the notice on every line, and must still
/// pick up a secret registered after the masker was built.
#[test]
+5 -5
View File
@@ -9,7 +9,7 @@
use crate::db::{Authable, UserDB};
use crate::error::{self, Error};
use crate::scripts::ScriptHash;
use crate::secret_backend::{get_secret_value, is_external_stored_value};
use crate::secret_backend::{get_secret_value, resolves_externally};
use crate::utils::{strip_json_nul, WarnAfterExt};
use crate::worker::{to_raw_value, Connection};
use crate::workspace_crypt::CipherFormat;
@@ -490,7 +490,7 @@ pub async fn decrypt_secret_value(
) -> crate::error::Result<String> {
if value.is_empty() {
Ok(value)
} else if is_external_stored_value(&value) {
} else if resolves_externally(&value) {
get_secret_value(db, w_id, path, &value).await
} else {
let mc = build_crypt(db, w_id).await?;
@@ -687,7 +687,7 @@ async fn upgrade_workspace_secret_variables(db: &DB, w_id: &str) -> error::Resul
.await?;
let last_page = (rows.len() as i64) < LEGACY_SECRET_UPGRADE_BATCH;
for (path, value) in rows {
if value.is_empty() || is_external_stored_value(&value) || mc.is_v2(&value) {
if value.is_empty() || resolves_externally(&value) || mc.is_v2(&value) {
after = path;
continue;
}
@@ -1029,7 +1029,7 @@ pub async fn get_variable_or_self(
if let Some(record) = record {
let mut value = record.value;
if record.is_secret {
if is_external_stored_value(&value) {
if resolves_externally(&value) {
value = get_secret_value(db, w_id, &path, &value).await?;
} else {
let mc = build_crypt(db, w_id).await?;
@@ -1078,7 +1078,7 @@ pub async fn get_variable_or_self_as<T: Authable + Sync>(
if let Some(record) = record {
let mut value = record.value;
if record.is_secret {
if is_external_stored_value(&value) {
if resolves_externally(&value) {
value = get_secret_value(db, w_id, &var_path, &value).await?;
} else {
let mc = build_crypt(db, w_id).await?;
+2 -2
View File
@@ -11,7 +11,7 @@ use crate::{
error::{self, to_anyhow, Error, Result},
get_database_url,
roles::{caller_capabilities, AuthorLevel, RoleCapabilities},
secret_backend::{get_secret_value, is_external_stored_value},
secret_backend::{get_secret_value, resolves_externally},
utils::{get_custom_pg_instance_password, get_custom_pg_instance_replication_password},
variables::{build_crypt, decrypt},
PgDatabase, DB,
@@ -3813,7 +3813,7 @@ async fn transform_json_unchecked(
.map_err(to_anyhow)?
.ok_or_else(|| Error::NotFound(format!("variable {path} does not exist")))?;
let value = if is_secret {
if is_external_stored_value(&value) {
if resolves_externally(&value) {
get_secret_value(db, w_id, &s[5..], &value).await?
} else {
let mc = build_crypt(&db, &w_id).await?;
@@ -79,6 +79,8 @@ mod tests {
kv_secret_path_prefix: None,
jwt_role: None, // Static token mode
jwt_mount_path: None,
approle_role_id: None,
approle_mount_path: None,
namespace: None,
token: Some(
std::env::var("VAULT_TOKEN").unwrap_or_else(|_| "test-root-token".to_string()),
@@ -96,6 +98,8 @@ mod tests {
kv_secret_path_prefix: None,
jwt_role: Some("windmill-secrets".to_string()), // JWT mode
jwt_mount_path: None,
approle_role_id: None,
approle_mount_path: None,
namespace: None,
token: None, // No static token - use JWT
skip_ssl_verify: None,
@@ -37,6 +37,8 @@ fn test_vault_settings() -> VaultSettings {
kv_secret_path_prefix: None,
jwt_role: Some("windmill-secrets".to_string()),
jwt_mount_path: None,
approle_role_id: None,
approle_mount_path: None,
namespace: None,
token: Some(std::env::var("VAULT_TOKEN").unwrap_or_else(|_| "test-root-token".to_string())),
skip_ssl_verify: None,
@@ -23,9 +23,10 @@ use windmill_common::{
};
pub use windmill_common::secret_backend::{
get_secret_backend, get_secret_value, is_aws_sm_stored_value, is_azure_kv_stored_value,
is_external_stored_value, is_http_store_stored_value, is_vault_backend_configured,
is_vault_stored_value,
check_external_ref, get_secret_backend, get_secret_value, is_aws_sm_stored_value,
is_azure_kv_stored_value, is_external_ref_value, is_external_stored_value,
is_http_store_stored_value, is_vault_backend_configured, is_vault_stored_value,
parse_external_ref, resolves_externally,
};
/// The HTTP store's key is derived from the variable path, so moving a reference would point it
+148 -59
View File
@@ -14,9 +14,9 @@ use windmill_common::db::{Authable, DB};
use windmill_common::workspaces::{check_deploy_rules, operator_can_draft, RuleCheckResult};
use crate::secret_backend_ext::{
delete_secret_from_backend, get_secret_value, http_store_rename_error,
is_external_stored_value, is_http_store_stored_value, is_vault_stored_value,
rename_vault_secret, store_managed_secret_value, store_secret_value,
check_external_ref, delete_secret_from_backend, get_secret_value, http_store_rename_error,
is_external_ref_value, is_http_store_stored_value, is_vault_stored_value, parse_external_ref,
rename_vault_secret, resolves_externally, store_managed_secret_value, store_secret_value,
};
use windmill_common::utils::{
check_proper_path, escape_ilike_pattern, sanitize_db_error, BulkDeleteRequest,
@@ -39,12 +39,13 @@ use windmill_common::{
db::{DbWithOptAuthed, UserDB},
error::{Error, JsonResult, Result},
scripts::ScriptHash,
secret_backend::allowed_prefixes,
user_drafts::{
decrypt_draft_secret_value, delete_all_drafts_for_path, delete_own_draft_for_path,
fetch_draft_only, fetch_draft_only_list_rows, maybe_overlay_draft, UserDraftItemKind,
WithDraftOverlay, ENCRYPTED_DRAFT_PREFIX,
},
utils::{not_found_if_none, paginate, Pagination, StripPath, WarnAfterExt},
utils::{not_found_if_none, paginate, require_admin, Pagination, StripPath, WarnAfterExt},
variables::{
build_crypt, get_reserved_variables, ContextualVariable, CreateVariable, ListableVariable,
},
@@ -76,6 +77,10 @@ pub fn workspaced_service() -> Router {
.route("/delete_bulk", delete(delete_variables_bulk))
.route("/create", post(create_variable))
.route("/encrypt", post(encrypt_value))
.route(
"/external_secret_prefixes",
get(list_external_secret_prefixes),
)
}
async fn list_contextual_variables(
@@ -139,7 +144,8 @@ async fn list_variables(
.fields(&[
"variable.workspace_id",
"variable.path",
"CASE WHEN is_secret IS TRUE THEN null ELSE variable.value::text END as value",
// An `$external:` reference names where the secret lives, not the secret.
"CASE WHEN is_secret IS TRUE AND variable.value NOT LIKE '$external:%' THEN null ELSE variable.value::text END as value",
"is_secret",
"variable.description",
"variable.extra_perms",
@@ -426,7 +432,7 @@ async fn get_variable(
} else if !value.is_empty() && decrypt_secret {
// Use secret backend for decryption (supports both DB and Vault)
Some(get_secret_value(&db, &w_id, &variable.path, &value).await?)
} else if q.include_encrypted.unwrap_or(false) {
} else if q.include_encrypted.unwrap_or(false) || is_external_ref_value(&value) {
Some(value)
} else {
None
@@ -580,7 +586,7 @@ async fn check_path_conflict(db: &DB, w_id: &str, path: &str) -> Result<()> {
/// Values stored by an external backend ($vault:/$aws_sm:/$azure_kv:/$http_store: markers)
/// are not workspace ciphertext and are passed through untouched.
fn validate_already_encrypted_secret(path: &str, value: &str) -> Result<()> {
if is_external_stored_value(value) {
if resolves_externally(value) {
return Ok(());
}
let looks_like_ciphertext = STANDARD
@@ -597,6 +603,71 @@ fn validate_already_encrypted_secret(path: &str, value: &str) -> Result<()> {
Ok(())
}
/// Creating, repointing or removing an `$external:` reference takes a workspace
/// admin. Every reference resolves with the instance's one store identity, so
/// the allow-list cannot tell users apart: any writer could point a variable at
/// a secret under an allowed prefix and read it.
fn check_external_ref_change(
is_admin: bool,
old_ref: Option<&str>,
new_ref: Option<&str>,
) -> Result<()> {
if !is_admin && old_ref != new_ref {
return Err(Error::PermissionDenied(
"Only workspace admins can create, repoint or remove an external secret reference"
.to_string(),
));
}
Ok(())
}
/// The `value` column for a secret sent to create or update, replacing
/// `old_ref` when the variable currently holds a reference. An `$external:`
/// reference is checked against the workspace's allowed prefixes and kept
/// verbatim, including when pushed as already encrypted by sync; any other
/// value goes to the configured secret backend, or with `windmill_writes`
/// (an OAuth token Windmill refreshes) to [`store_managed_secret_value`].
async fn secret_value_to_store(
db: &DB,
w_id: &str,
path: &str,
value: String,
already_encrypted: bool,
op: &'static str,
is_admin: bool,
old_ref: Option<&str>,
windmill_writes: bool,
) -> Result<String> {
if already_encrypted && !is_external_ref_value(&value) {
check_external_ref_change(is_admin, old_ref, None)?;
// Stored verbatim, so it must be ciphertext and not plaintext mislabeled as encrypted.
validate_already_encrypted_secret(path, &value)?;
return Ok(value);
}
// A restored draft sends the `$encrypted:` marker as-is; decrypt it back
// (validating against the workspace key) before storing it.
let value = if value.starts_with(ENCRYPTED_DRAFT_PREFIX) {
decrypt_draft_secret_value(db, w_id, &value).await?
} else {
value
};
check_external_ref_change(
is_admin,
old_ref,
is_external_ref_value(&value).then_some(value.as_str()),
)?;
if is_external_ref_value(&value) {
check_external_ref(db, w_id, &value).await?;
windmill_common::feature_usage::log_feature_usage("external_secret_ref", "saved", op);
return Ok(value);
}
if windmill_writes {
store_managed_secret_value(db, w_id, path, &value).await
} else {
store_secret_value(db, w_id, path, &value).await
}
}
async fn create_variable(
authed: ApiAuthed,
Extension(db): Extension<DB>,
@@ -644,7 +715,7 @@ pub async fn create_variable_with_account(
user_db: UserDB,
webhook: &WebhookShared,
w_id: String,
variable: CreateVariable,
mut variable: CreateVariable,
already_encrypted: bool,
account: Option<NewOAuthAccount>,
) -> Result<Option<i32>> {
@@ -680,25 +751,21 @@ pub async fn create_variable_with_account(
let authed = maybe_refresh_folders(&variable.path, &w_id, authed, &db).await;
check_path_conflict(&db, &w_id, &variable.path).await?;
let value = if variable.is_secret && !already_encrypted {
// A restored draft sends the `$encrypted:` marker as-is; decrypt it back
// (validating against the workspace key) before the secret backend re-stores it.
let plain = if variable.value.starts_with(ENCRYPTED_DRAFT_PREFIX) {
decrypt_draft_secret_value(&db, &w_id, &variable.value).await?
} else {
variable.value.clone()
};
if account.is_some() {
store_managed_secret_value(&db, &w_id, &variable.path, &plain).await?
} else {
store_secret_value(&db, &w_id, &variable.path, &plain).await?
}
let value = if variable.is_secret {
let value = std::mem::take(&mut variable.value);
secret_value_to_store(
&db,
&w_id,
&variable.path,
value,
already_encrypted,
"create",
authed.is_admin,
None,
account.is_some(),
)
.await?
} else {
if variable.is_secret {
// already_encrypted == true: value is stored verbatim, so it must be
// ciphertext and not plaintext mislabeled as encrypted.
validate_already_encrypted_secret(&variable.path, &variable.value)?;
}
variable.value
};
@@ -788,6 +855,15 @@ pub async fn create_variable_with_account(
Ok(account_id)
}
async fn list_external_secret_prefixes(
authed: ApiAuthed,
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
) -> JsonResult<Vec<String>> {
require_admin(authed.is_admin, &authed.username)?;
Ok(Json(allowed_prefixes(&db, &w_id).await?))
}
async fn encrypt_value(
Extension(db): Extension<DB>,
Path(w_id): Path<String>,
@@ -821,9 +897,10 @@ async fn delete_variable(
return Err(Error::PermissionDenied(msg));
}
// Check if variable is a secret before deleting (for Vault cleanup)
// Check if variable is a secret before deleting (for Vault cleanup). An external reference
// is excluded: its path may name a secret Windmill does not own.
let is_secret = sqlx::query_scalar!(
"SELECT is_secret FROM variable WHERE path = $1 AND workspace_id = $2",
r#"SELECT is_secret AND value NOT LIKE '$external:%' AS "is_secret!" FROM variable WHERE path = $1 AND workspace_id = $2"#,
path,
&w_id
)
@@ -995,9 +1072,9 @@ async fn delete_variables_bulk(
return Err(Error::PermissionDenied(msg));
}
// Query which paths are secrets before deletion (for Vault cleanup)
// Query which paths are secrets before deletion (for Vault cleanup), external references excluded
let secret_paths: Vec<String> = sqlx::query_scalar!(
"SELECT path FROM variable WHERE path = ANY($1) AND workspace_id = $2 AND is_secret = true",
"SELECT path FROM variable WHERE path = ANY($1) AND workspace_id = $2 AND is_secret = true AND value NOT LIKE '$external:%'",
&request.paths,
&w_id
)
@@ -1199,35 +1276,37 @@ async fn update_variable(
// Determine the target path for storing secrets (use new path if provided)
let target_path = ns.path.as_deref().unwrap_or(path);
if let Some(nvalue) = ns.value.clone() {
let is_secret = if ns.is_secret.is_some() {
ns.is_secret.unwrap()
} else {
sqlx::query_scalar!(
"SELECT is_secret from variable WHERE path = $1 AND workspace_id = $2",
&path,
&w_id
)
.fetch_optional(&db)
.await?
.unwrap_or(false)
};
let old = sqlx::query!(
"SELECT value::text AS \"value!\", is_secret FROM variable WHERE path = $1 AND workspace_id = $2",
path,
w_id
)
.fetch_optional(&db)
.await?;
let is_secret = ns
.is_secret
.unwrap_or_else(|| old.as_ref().is_some_and(|o| o.is_secret));
let old_ref = old
.as_ref()
.filter(|o| o.is_secret && is_external_ref_value(&o.value))
.map(|o| o.value.as_str());
let value = if is_secret && !already_encrypted.unwrap_or(false) {
// Decrypt a restored draft's `$encrypted:` marker before re-storing; see create_variable.
let plain = if nvalue.starts_with(ENCRYPTED_DRAFT_PREFIX) {
decrypt_draft_secret_value(&db, &w_id, &nvalue).await?
} else {
nvalue
};
// Use secret backend for encryption (supports both DB and Vault)
// Store at target_path (new path if renaming, otherwise current path)
store_secret_value(&db, &w_id, target_path, &plain).await?
let value = if is_secret {
// Stored at target_path (the new path when renaming)
secret_value_to_store(
&db,
&w_id,
target_path,
nvalue,
already_encrypted.unwrap_or(false),
"update",
authed.is_admin,
old_ref,
false,
)
.await?
} else {
if is_secret {
// already_encrypted == true: value is stored verbatim, so it must
// be ciphertext and not plaintext mislabeled as encrypted.
validate_already_encrypted_secret(target_path, &nvalue)?;
}
check_external_ref_change(authed.is_admin, old_ref, None)?;
nvalue
};
sqlb.set_str("value", &value);
@@ -1594,6 +1673,10 @@ pub async fn get_value_internal<'a>(
unreachable!()
};
// A reference is resolved on every read, so Vault rotation and allow-list changes apply at once.
let cacheable = !is_external_ref_value(&variable.value);
let whole_secret_ref = is_external_ref_value(&variable.value)
&& parse_external_ref(&variable.value).is_ok_and(|r| r.key.is_none());
let r = if variable.is_secret {
audit_decrypt_secret(db_with_opt_authed, &w_id, &variable.path).await?;
@@ -1623,13 +1706,19 @@ pub async fn get_value_internal<'a>(
};
if variable.is_secret && !r.is_empty() {
windmill_common::sensitive_log_masks::register_secret_for_all_running_jobs(&r);
if whole_secret_ref {
windmill_common::sensitive_log_masks::register_secret_and_fields_for_all_running_jobs(
&r,
);
} else {
windmill_common::sensitive_log_masks::register_secret_for_all_running_jobs(&r);
}
}
// Cache the result when explicitly allowed. Secrets are cached too: their per-read side
// effects (audit + running-job registration) are re-run on a hit (see the hit path above),
// and `is_secret` is stored so the hit knows to do so.
if let Some(identity) = cache_identity.as_deref() {
if let Some(identity) = cache_identity.as_deref().filter(|_| cacheable) {
cache_variable(
&w_id,
&path,
+5 -3
View File
@@ -111,15 +111,17 @@ export interface VariableFile {
* ciphertext (AES-256-GCM or legacy AES-256-CBC, base64) is standard base64
* decoding to a non-zero
* multiple of the 16-byte block size. External secret-backend markers
* ($vault:/$aws_sm:/$azure_kv:/$http_store:) are stored verbatim too, so they
* count as already-encrypted. This is a shape check only — it never decrypts.
* ($vault:/$aws_sm:/$azure_kv:/$http_store:) and `$external:` references to a secret in
* an external store are stored verbatim too, so they count as already-encrypted.
* This is a shape check only — it never decrypts.
*/
export function looksLikeWorkspaceCiphertext(value: string): boolean {
if (
value.startsWith("$vault:") ||
value.startsWith("$aws_sm:") ||
value.startsWith("$azure_kv:") ||
value.startsWith("$http_store:")
value.startsWith("$http_store:") ||
value.startsWith("$external:")
) {
return true;
}
@@ -19,6 +19,7 @@ test("treats workspace-ciphertext-shaped values as already-encrypted", () => {
"$vault:f/x/cfg",
"$aws_sm:f/x/cfg",
"$azure_kv:f/x/cfg",
"$external:vault://secret/team/app#token",
];
for (const value of ciphertextShaped) {
expect(looksLikeWorkspaceCiphertext(value)).toBe(true);
+10 -2
View File
@@ -602,11 +602,19 @@ export async function deployItem(
}
}
} else if (kind === "variable") {
const variable = await provider.getVariable({
// An external secret reference is copied as the reference, never as the value it resolves to.
let variable = await provider.getVariable({
workspace: workspaceFrom,
path,
decryptSecret: true,
decryptSecret: false,
});
if (variable.is_secret && !variable.value?.startsWith("$external:")) {
variable = await provider.getVariable({
workspace: workspaceFrom,
path,
decryptSecret: true,
});
}
if (alreadyExists) {
await provider.updateVariable({
workspace: workspaceTo,
@@ -17,6 +17,7 @@
UserPlus
} from 'lucide-svelte'
import { onDestroy, untrack } from 'svelte'
import { isExternalRef } from '$lib/externalSecretRef'
import type { CiTestResult } from '$lib/gen'
import { Alert, Badge } from './common'
import {
@@ -285,7 +286,11 @@
}
})
} else {
const v = await VariableService.getVariable({ workspace: from, path, decryptSecret: true })
// An external secret reference is copied as the reference, never as its resolved value.
let v = await VariableService.getVariable({ workspace: from, path, decryptSecret: false })
if (v.is_secret && !isExternalRef(v.value)) {
v = await VariableService.getVariable({ workspace: from, path, decryptSecret: true })
}
await VariableService.createVariable({
workspace: to,
requestBody: {
@@ -26,6 +26,7 @@
import WorkspaceRolesSetting from './instanceSettings/WorkspaceRolesSetting.svelte'
import SmtpSettings from './instanceSettings/SmtpSettings.svelte'
import SecretBackendConfig from './instanceSettings/SecretBackendConfig.svelte'
import ExternalSecretsConfig from './instanceSettings/ExternalSecretsConfig.svelte'
import ExternalInstancePgSettings from './instanceSettings/ExternalInstancePgSettings.svelte'
import InstancePgSettings from './instanceSettings/InstancePgSettings.svelte'
import GhesAppSettings from './instanceSettings/GhesAppSettings.svelte'
@@ -879,6 +880,8 @@
<SmtpSettings {values} disabled={loading} />
{:else if setting.fieldType == 'secret_backend'}
<SecretBackendConfig {values} disabled={loading} />
{:else if setting.fieldType == 'external_secrets'}
<ExternalSecretsConfig {values} disabled={loading} />
{:else if setting.fieldType == 'external_instance_pg'}
<ExternalInstancePgSettings
{values}
@@ -764,7 +764,20 @@
custom_instance_pg_databases: ['user_pwd'],
external_instance_pg: ['password'],
rsa_keys: ['private_key'],
github_enterprise_app: ['private_key']
github_enterprise_app: ['private_key'],
external_secrets: ['vault.token']
}
// A field may name one nested level, `parent.child`.
function getField(obj: any, field: string): any {
const [a, b] = field.split('.')
return b === undefined ? obj?.[a] : obj?.[a]?.[b]
}
function setField(obj: any, field: string, value: any) {
const [a, b] = field.split('.')
if (b === undefined) obj[a] = value
else if (obj[a] && typeof obj[a] === 'object') obj[a] = { ...obj[a], [b]: value }
}
/** Returns SENSITIVE_UNCHANGED if the value is non-empty and matches the initial */
@@ -792,10 +805,10 @@
const init = initialValues?.[key]
for (const field of nestedSensitiveFields[key]) {
const m = maskField(
field === 'serviceAccountKey' ? JSON.stringify(cp[field]) : cp[field],
field === 'serviceAccountKey' ? JSON.stringify(init?.[field]) : init?.[field]
field === 'serviceAccountKey' ? JSON.stringify(cp[field]) : getField(cp, field),
field === 'serviceAccountKey' ? JSON.stringify(init?.[field]) : getField(init, field)
)
if (m) cp[field] = m
if (m) setField(cp, field, m)
}
masked[key] = cp
} else if (sensitiveKeys.has(key) && value != null && value !== '') {
@@ -898,8 +911,8 @@
if (parsed[parentKey] && typeof parsed[parentKey] === 'object') {
const init = initialValues?.[parentKey]
for (const field of fields) {
if (parsed[parentKey][field] === SENSITIVE_UNCHANGED) {
parsed[parentKey][field] = init?.[field]
if (getField(parsed[parentKey], field) === SENSITIVE_UNCHANGED) {
setField(parsed[parentKey], field, getField(init, field))
}
}
}
@@ -31,6 +31,7 @@
import { setLocalDraftHint } from '$lib/localDraftHints.svelte'
import { logFeatureUsage } from '$lib/utils/featureUsage'
import { useOperatingWorkspace } from '$lib/components/operatingWorkspace.svelte'
import { isExternalRef } from '$lib/externalSecretRef'
const operatingWorkspace = useOperatingWorkspace()
@@ -431,7 +432,8 @@
if (!isSecret || !selected || !existedInitially[selected]) return
const s = states[selected]?.draft
const ini = initialStates[selected]
if (!s || !ini?.variable.is_secret) return
// A reference was never loaded, and its baseline is what tells save() to replace it.
if (!s || !ini?.variable.is_secret || isExternalRef(ini.variable.value)) return
if (s.variable.value !== '' && s.variable.value !== ini.variable.value) return
s.variable.value = ''
ini.variable.value = ''
@@ -456,7 +458,13 @@
path: ini.path,
requestBody: {
path: ini.path != s.path ? s.path : undefined,
value: s.variable.value == '' ? undefined : s.variable.value,
// An empty value keeps the stored one, except when it replaces a
// reference: the reference is then dropped for the empty value.
value:
s.variable.value == '' &&
!(isExternalRef(ini.variable.value) && !isExternalRef(s.variable.value))
? undefined
: s.variable.value,
is_secret:
ini.variable.is_secret != s.variable.is_secret ? s.variable.is_secret : undefined,
description:
@@ -599,6 +607,9 @@
deployedSecret={!!selected &&
!!existedInitially[selected] &&
!!initialStates[selected]?.variable.is_secret}
deployedRef={!!selected &&
!!existedInitially[selected] &&
isExternalRef(initialStates[selected]?.variable.value ?? '')}
workspace={selected}
actingUser={acting.in(selected) ?? null}
oauthLinked={!!selected && !!oauthLinked[selected]}
+227 -105
View File
@@ -17,6 +17,10 @@
import EncryptedDraftField from './EncryptedDraftField.svelte'
import DateTimeInput from './DateTimeInput.svelte'
import { useOperatingWorkspace } from '$lib/components/operatingWorkspace.svelte'
import TextInput from './text_input/TextInput.svelte'
import { formatVaultRef, isExternalRef, parseVaultRef } from '$lib/externalSecretRef'
import { VariableService } from '$lib/gen'
import { resource } from 'runed'
const operatingWorkspace = useOperatingWorkspace()
@@ -42,6 +46,8 @@
onSecretChange?: (isSecret: boolean) => void
/** Whether the deployed variable being edited is secret. */
deployedSecret?: boolean
/** Whether the deployed variable being edited is an external reference. */
deployedRef?: boolean
/** Workspace the path is validated against; defaults to the nav workspace. */
workspace?: string | undefined
/** The user acting in `workspace`, resolved by the editor above. `undefined` while
@@ -67,6 +73,7 @@
onLoadSecret,
onSecretChange,
deployedSecret = false,
deployedRef = false,
workspace = undefined,
actingUser,
oauthLinked = false
@@ -96,11 +103,43 @@
let hasStagedValue = $derived(variable.value !== '')
// Un-securing has to send the plaintext, and fetching it silently on the toggle would
// reveal the secret without the explicit, audited "Load secret value" action.
let unsecureNeedsLoad = $derived(edit && deployedSecret && variable.is_secret && !hasStagedValue)
// reveal the secret without the explicit, audited "Load secret value" action. A saved
// reference holds no value to load: whatever is typed, even nothing, replaces it.
let unsecureNeedsLoad = $derived(
edit && deployedSecret && !deployedRef && variable.is_secret && !hasStagedValue
)
// The API refuses anyone else creating, repointing or removing a reference.
let canEditRef = $derived(can_write && !!actingUser?.is_admin)
const MAX_VARIABLE_LENGTH = 10000
let externalRef = $derived(variable.is_secret && isExternalRef(variable.value))
let vaultRef = $derived(parseVaultRef(variable.value))
let kind = $derived(externalRef ? 'ref' : variable.is_secret ? 'secret' : 'plain')
let secretLocked = $derived(
edit && (!actingUser || !caps.code || isEncryptedDraftValue(variable.value))
)
function setKind(next: string | undefined) {
if (!next || next === kind) return
if (kind === 'ref') variable.value = ''
if (variable.is_secret !== (next !== 'plain')) {
variable.is_secret = next !== 'plain'
onSecretChange?.(variable.is_secret)
}
if (next === 'ref') variable.value = formatVaultRef('', '')
}
let allowedPrefixes = resource(
() => (canEditRef ? ws : undefined),
async (ws) =>
ws ? await VariableService.listExternalSecretPrefixes({ workspace: ws }) : undefined
)
let pathPlaceholder = $derived(
(allowedPrefixes.current?.[0]?.replace(/^vault:\/\//, '').replace(/\/*$/, '') ??
'secret/team') + '/app'
)
let editorKind: 'plain' | 'json' | 'yaml' = $state('plain')
let editor: any = $state(undefined)
@@ -123,29 +162,66 @@
/>
<LabelsInput bind:labels />
</div>
<!-- No <label> for the heading: clicking it would flip the toggle, and a wrapping one would
stretch the hit area over the whole row and the alert. The toggle carries its name instead. -->
<div class="flex flex-col gap-1">
<span class="text-xs font-semibold text-emphasis">Secret</span>
<span class="text-xs font-semibold text-emphasis">Type</span>
<div class="flex flex-row items-center gap-2">
<!-- An `$encrypted:` value is only redeemable while the variable stays secret — the
deploy endpoints decrypt the marker inside their `is_secret` branch and store it
verbatim otherwise — so un-securing one has to be unreachable until it is Reset. -->
<Toggle
ariaLabel="Secret"
class="w-fit"
on:change={(e) => onSecretChange?.(e.detail)}
bind:checked={variable.is_secret}
disabled={unsecureNeedsLoad ||
(edit && (!actingUser || !caps.code || isEncryptedDraftValue(variable.value)))}
/>
{#if unsecureNeedsLoad && can_write && caps.code}
<span class="text-2xs text-secondary">
Load the secret value or enter a new one to make it non-secret
</span>
{/if}
<ToggleButtonGroup selected={kind} onSelected={setKind}>
{#snippet children({ item })}
<ToggleButton
value="plain"
label="Non-secret"
{item}
disabled={kind === 'ref'
? !canEditRef
: kind === 'secret' && (secretLocked || unsecureNeedsLoad)}
/>
<ToggleButton
value="secret"
label="Secret"
{item}
disabled={kind === 'plain'
? secretLocked
: kind === 'ref' && (!canEditRef || isEncryptedDraftValue(variable.value))}
/>
<!-- Without an allowed prefix (the `external_secrets` setting unset, or naming no
prefix for this workspace) every reference is refused, so none can be created. -->
{#if externalRef || (canEditRef && allowedPrefixes.current?.length)}
<ToggleButton
value="ref"
label="Vault reference"
tooltip="Read the value from an existing secret in HashiCorp Vault"
{item}
disabled={kind !== 'ref' &&
(!canEditRef ||
isEncryptedDraftValue(variable.value) ||
(kind === 'plain' && secretLocked))}
/>
{/if}
{/snippet}
</ToggleButtonGroup>
</div>
{#if variable.is_secret}
{#if externalRef && can_write && !canEditRef}
<span class="text-xs text-secondary">
Only workspace admins can create or change a reference
</span>
{/if}
{#if deployedRef && kind !== 'ref' && !hasStagedValue}
<span class="text-xs text-secondary">Enter a value to replace the Vault reference</span>
{:else if unsecureNeedsLoad && can_write && caps.code}
<span class="text-xs text-secondary">
Load the secret value or enter a new one to make it non-secret
</span>
{/if}
{#if externalRef}
<Alert type="info" title="Read from Vault on each access">
Only the reference is stored in Windmill. The value is read from HashiCorp Vault each time the
variable is read, and every read generates an audit log whose operation name is:
variables.decrypt_secret
</Alert>
{:else if variable.is_secret}
<Alert type="info" title="Audit log for each access">
Every secret is encrypted at rest and in transit with a key specific to this workspace. In
addition, any read of a secret variable generates an audit log whose operation name is:
@@ -163,96 +239,142 @@ stretch the hit area over the whole row and the alert. The toggle carries its na
</Label>
{/if}
<div class="flex flex-col gap-1">
<label for="variable-value" class="flex flex-row justify-left items-center">
<span class="text-xs font-semibold text-emphasis">Variable value&nbsp;</span>
{#if !isEncryptedDraftValue(variable.value)}
<span class="text-xs text-secondary font-normal">
({variable.value.length}/{MAX_VARIABLE_LENGTH} characters)
</span>
{/if}
{#if edit && variable.is_secret}
<div class="ml-3"></div>
{#if hasStagedValue}
<!-- Clearing the staged value is the only way back to loading the deployed one;
see `hasStagedValue`. An `$encrypted:` value additionally cannot be displayed. -->
<Button
size="xs"
variant="default"
startIcon={{ icon: RotateCcw }}
disabled={!can_write}
on:click={() => (variable.value = '')}
>
Reset
</Button>
{:else if !caps.code}
<div class="p-2 border">Your role cannot load secret values</div>
{:else}
<Button size="xs" variant="default" on:click={() => onLoadSecret?.()}>
Load secret value<Tooltip>Will generate an audit log</Tooltip>
</Button>
{#if externalRef}
<div class="flex flex-col gap-2">
<div class="flex flex-col gap-1">
<label for="external-ref-path" class="text-xs font-semibold text-emphasis">Vault path</label>
{#if allowedPrefixes.current?.length}
<span class="text-xs text-secondary">
Under <code
>{allowedPrefixes.current
.map((p) => p.replace(/^vault:\/\//, '').replace(/\/*$/, '/**'))
.join(', ')}</code
>
</span>
{:else if allowedPrefixes.current}
<Alert type="warning" size="xs" title="No Vault paths allowed for this workspace">
Ask a superadmin to allow one in the instance settings.
</Alert>
{/if}
{/if}
</label>
<div>
{#if isEncryptedDraftValue(variable.value)}
<!-- The draft's secret value was encrypted server-side and can't be
<TextInput
inputProps={{
id: 'external-ref-path',
placeholder: pathPlaceholder,
disabled: !canEditRef
}}
bind:value={
() => vaultRef.path, (v) => (variable.value = formatVaultRef(v ?? '', vaultRef.key))
}
/>
</div>
<div class="flex flex-col gap-1">
<label for="external-ref-key" class="text-xs font-semibold text-emphasis">
Key (optional)
</label>
<span class="text-xs text-secondary">
The entry to read from the secret, like <code>token</code>. Leave empty to get all its
entries as a JSON object.
</span>
<TextInput
inputProps={{ id: 'external-ref-key', placeholder: 'token', disabled: !canEditRef }}
bind:value={
() => vaultRef.key, (v) => (variable.value = formatVaultRef(vaultRef.path, v ?? ''))
}
/>
</div>
</div>
{:else}
<div class="flex flex-col gap-1">
<label for="variable-value" class="flex flex-row justify-left items-center">
<span class="text-xs font-semibold text-emphasis">Variable value&nbsp;</span>
{#if !isEncryptedDraftValue(variable.value)}
<span class="text-xs text-secondary font-normal">
({variable.value.length}/{MAX_VARIABLE_LENGTH} characters)
</span>
{/if}
{#if edit && variable.is_secret && !(deployedRef && !hasStagedValue)}
<div class="ml-3"></div>
{#if hasStagedValue}
<!-- Clearing the staged value is the only way back to loading the deployed one;
see `hasStagedValue`. An `$encrypted:` value additionally cannot be displayed. -->
<Button
size="xs"
variant="default"
startIcon={{ icon: RotateCcw }}
disabled={!can_write}
on:click={() => (variable.value = '')}
>
Reset
</Button>
{:else if !caps.code}
<div class="p-2 border">Your role cannot load secret values</div>
{:else}
<Button size="xs" variant="default" on:click={() => onLoadSecret?.()}>
Load secret value<Tooltip>Will generate an audit log</Tooltip>
</Button>
{/if}
{/if}
</label>
<div>
{#if isEncryptedDraftValue(variable.value)}
<!-- The draft's secret value was encrypted server-side and can't be
loaded back. Saving deploys the last saved value as-is; Reset clears
it so a new secret can be typed. -->
<EncryptedDraftField disabled={!can_write} onReset={() => (variable.value = '')} />
{:else}
<div class="flex flex-col gap-2">
<ToggleButtonGroup bind:selected={editorKind}>
{#snippet children({ item })}
<ToggleButton value="plain" label="Plain" {item} />
<ToggleButton value="json" label="Json" {item} />
<ToggleButton value="yaml" label="YAML" {item} />
{/snippet}
</ToggleButtonGroup>
{#if editorKind == 'plain'}
<textarea
disabled={!can_write}
rows="4"
use:autosize
bind:value={variable.value}
placeholder="Update variable value"
id="variable-value"
></textarea>
{:else if editorKind == 'json'}
<div class="border rounded mb-4 w-full">
{#await import('$lib/components/SimpleEditor.svelte')}
<Loader2 class="animate-spin" />
{:then Module}
<Module.default
bind:this={editor}
autoHeight
lang="json"
bind:code={variable.value}
fixedOverflowWidgets={false}
class="bg-surface-tertiary"
/>
{/await}
</div>
{:else if editorKind == 'yaml'}
<div class="border rounded mb-4 w-full">
{#await import('$lib/components/SimpleEditor.svelte')}
<Loader2 class="animate-spin" />
{:then Module}
<Module.default
bind:this={editor}
autoHeight
lang="yaml"
bind:code={variable.value}
fixedOverflowWidgets={false}
class="bg-surface-tertiary"
/>
{/await}
</div>
{/if}
</div>
{/if}
<EncryptedDraftField disabled={!can_write} onReset={() => (variable.value = '')} />
{:else}
<div class="flex flex-col gap-2">
<ToggleButtonGroup bind:selected={editorKind}>
{#snippet children({ item })}
<ToggleButton value="plain" label="Plain" {item} />
<ToggleButton value="json" label="Json" {item} />
<ToggleButton value="yaml" label="YAML" {item} />
{/snippet}
</ToggleButtonGroup>
{#if editorKind == 'plain'}
<textarea
disabled={!can_write}
rows="4"
use:autosize
bind:value={variable.value}
placeholder="Update variable value"
id="variable-value"
></textarea>
{:else if editorKind == 'json'}
<div class="border rounded mb-4 w-full">
{#await import('$lib/components/SimpleEditor.svelte')}
<Loader2 class="animate-spin" />
{:then Module}
<Module.default
bind:this={editor}
autoHeight
lang="json"
bind:code={variable.value}
fixedOverflowWidgets={false}
class="bg-surface-tertiary"
/>
{/await}
</div>
{:else if editorKind == 'yaml'}
<div class="border rounded mb-4 w-full">
{#await import('$lib/components/SimpleEditor.svelte')}
<Loader2 class="animate-spin" />
{:then Module}
<Module.default
bind:this={editor}
autoHeight
lang="yaml"
bind:code={variable.value}
fixedOverflowWidgets={false}
class="bg-surface-tertiary"
/>
{/await}
</div>
{/if}
</div>
{/if}
</div>
</div>
</div>
{/if}
<label class="flex flex-col gap-1">
<span class="text-xs font-semibold text-emphasis">Description</span>
<textarea rows="4" use:autosize bind:value={variable.description} placeholder="Used for X"
@@ -485,7 +485,7 @@ export const emailTriggerRequestSchema = z.object({
export const variableRequestSchema = z.object({
"path": z.string().describe("The path to the variable"),
"value": z.string().describe("The value of the variable"),
"value": z.string().describe("The value of the variable. For a secret, `$external:vault://<path>#<key>` stores a reference to an existing secret in the instance's external HashiCorp Vault (path as `vault kv get` takes it) instead of a value; without `#<key>` it reads the whole secret as a JSON object of its fields. It is resolved on each read, only under the path prefixes a superadmin allowed for the workspace, and is returned as-is when the secret is not decrypted.\n"),
"is_secret": z.boolean().describe("Whether the variable is a secret"),
"description": z.string().describe("The description of the variable"),
"account": z.number().int().describe("Ignored. A variable is linked to an OAuth account only when `createAccount` creates both.").optional(),
@@ -70,6 +70,7 @@ export interface Setting {
| 'otel'
| 'otel_tracing_proxy'
| 'secret_backend'
| 'external_secrets'
| 'external_instance_pg'
| 'instance_pg'
| 'github_enterprise_app'
@@ -1270,6 +1271,15 @@ export const settings: Record<string, Setting[]> = {
storage: 'setting',
ee_only:
'External secret store integrations are Enterprise Edition features'
},
{
label: 'External secret references',
description:
'Let secret variables reference existing secrets in HashiCorp Vault, read when the variable is read, under the path prefixes allowed here.',
key: 'external_secrets',
fieldType: 'external_secrets',
storage: 'setting',
ee_only: 'External secret references are an Enterprise Edition feature'
}
],
'GitHub App': [
@@ -0,0 +1,335 @@
<script lang="ts">
import { Alert, Button } from '$lib/components/common'
import Password from '../Password.svelte'
import { SettingService, WorkspaceService } from '$lib/gen'
import { sendUserToast } from '$lib/toast'
import TextInput from '../text_input/TextInput.svelte'
import Toggle from '../Toggle.svelte'
import { Plus, Server, Trash2 } from 'lucide-svelte'
import type { Writable } from 'svelte/store'
import { enterpriseLicense } from '$lib/stores'
import ToggleButtonGroup from '../common/toggleButton-v2/ToggleButtonGroup.svelte'
import ToggleButton from '../common/toggleButton-v2/ToggleButton.svelte'
import EEOnly from '../EEOnly.svelte'
import MultiSelect from '../select/MultiSelect.svelte'
import { resource } from 'runed'
interface Props {
values: Writable<Record<string, any>>
disabled?: boolean
}
let { values, disabled = false }: Props = $props()
type AllowedPrefix = { prefix: string; workspaces: string[] }
let config = $derived(
$values['external_secrets'] as
| { vault?: Record<string, any>; allowed?: AllowedPrefix[] }
| undefined
)
let vault = $derived(config?.vault)
let backendVault = $derived(
$values['secret_backend']?.type === 'HashiCorpVault' ? $values['secret_backend'] : undefined
)
let workspaces = resource(
() => disabled,
async (disabled) =>
disabled
? []
: (await WorkspaceService.listWorkspacesAsSuperAdmin()).map((w) => ({
label: w.id,
value: w.id
}))
)
let allowed = $derived(config?.allowed ?? [])
let authMethod: 'token' | 'approle' | 'jwt' = $derived(
vault?.token != null ? 'token' : vault?.approle_role_id != null ? 'approle' : 'jwt'
)
let testing = $state(false)
function update(patch: { vault?: Record<string, any> | undefined; allowed?: AllowedPrefix[] }) {
const next = { ...(config ?? {}), ...patch }
$values['external_secrets'] = next.vault || next.allowed?.length ? next : undefined
}
function setConnection(mode: string | undefined) {
if (mode === 'same') {
update({ vault: undefined })
} else if (mode === 'separate' && backendVault && !vault) {
// The backend's storage location does not apply to references, only how to reach Vault.
const { type: _t, mount_path: _m, kv_secret_path_prefix: _p, ...conn } = backendVault
update({ vault: conn })
}
}
function setVaultField(field: string, value: any) {
update({ vault: { ...vault, [field]: value === '' ? undefined : value } })
}
function setAuthMethod(method: string | undefined) {
if (!vault || !method) return
const { token: _t, approle_role_id: _a, jwt_role: _j, ...rest } = vault
update({
vault: {
...rest,
...(method === 'token' ? { token: vault.token ?? '' } : {}),
...(method === 'approle' ? { approle_role_id: vault.approle_role_id ?? '' } : {}),
...(method === 'jwt' ? { jwt_role: vault.jwt_role ?? 'windmill-secrets' } : {})
}
})
}
function setAllowed(i: number, patch: Partial<AllowedPrefix>) {
update({ allowed: allowed.map((a, j) => (i === j ? { ...a, ...patch } : a)) })
}
async function testConnection() {
if (!vault) return
testing = true
try {
await SettingService.testSecretBackend({
requestBody: { ...vault, address: vault.address, mount_path: '' }
})
sendUserToast('Successfully connected to HashiCorp Vault')
} catch (e: any) {
sendUserToast(`Connection failed: ${e.body ?? e.message}`, true)
} finally {
testing = false
}
}
</script>
<div class="flex flex-col gap-4">
{#if !$enterpriseLicense}
<EEOnly>External secret references require Enterprise Edition</EEOnly>
{/if}
<span class="text-2xs text-secondary">
A secret variable can reference an existing secret, as
<code>$external:vault://&lt;path&gt;#&lt;key&gt;</code>, or the whole secret as JSON without
<code>#&lt;key&gt;</code>, instead of holding its value. The value is read from Vault each time
the variable is read.
</span>
<Alert type="info" size="xs" title="Prefer the Vault backend for secrets Windmill owns">
Use HashiCorp Vault as the secret backend for those. Use references for secrets that are managed
outside Windmill and must stay there.
</Alert>
{#if backendVault}
<div class="flex flex-col gap-2">
<span class="text-xs font-semibold text-emphasis">Vault connection</span>
<ToggleButtonGroup selected={vault ? 'separate' : 'same'} onSelected={setConnection}>
{#snippet children({ item })}
<ToggleButton
value="same"
label="Same as secret backend"
tooltip="Read references through the secret backend's Vault connection"
{item}
{disabled}
/>
<ToggleButton
value="separate"
label="Separate connection"
tooltip="Read references through their own Vault connection, for another Vault or a narrower identity"
{item}
{disabled}
/>
{/snippet}
</ToggleButtonGroup>
{#if !vault}
<span class="text-2xs text-secondary">
References are read from {backendVault.address || 'the Vault address above'} with the secret
backend's authentication.
</span>
{/if}
</div>
{:else}
<Toggle
{disabled}
checked={!!vault}
on:change={(e) =>
update({ vault: e.detail ? { address: '', jwt_role: 'windmill-secrets' } : undefined })}
size="xs"
options={{ right: 'HashiCorp Vault connection' }}
/>
{/if}
{#if vault}
<div class="flex flex-col gap-4 p-4 border rounded-lg">
<div class="flex flex-col gap-1">
<label for="ext_vault_address" class="text-xs font-semibold text-emphasis">
Vault address
</label>
<TextInput
inputProps={{
id: 'ext_vault_address',
placeholder: 'https://vault.company.com:8200',
disabled
}}
bind:value={() => vault?.address ?? '', (v) => setVaultField('address', v)}
/>
</div>
<div class="flex flex-col gap-2">
<span class="text-xs font-semibold text-emphasis">Authentication method</span>
<ToggleButtonGroup selected={authMethod} onSelected={(v) => setAuthMethod(v)}>
{#snippet children({ item })}
<ToggleButton
value="approle"
label="AppRole"
tooltip="Log in with an AppRole role_id and secret_id"
{item}
{disabled}
/>
<ToggleButton
value="jwt"
label="JWT auth"
tooltip="Authenticate using Windmill-signed JWTs"
{item}
{disabled}
/>
<ToggleButton
value="token"
label="Static token"
tooltip="Use a static Vault token"
{item}
{disabled}
/>
{/snippet}
</ToggleButtonGroup>
</div>
{#if authMethod === 'approle'}
<div class="flex flex-col gap-2 p-3 bg-surface-secondary rounded-lg">
<label for="ext_vault_role_id" class="text-xs font-semibold text-emphasis">Role ID</label>
<TextInput
inputProps={{ id: 'ext_vault_role_id', disabled }}
bind:value={
() => vault?.approle_role_id ?? '',
(v) => update({ vault: { ...vault, approle_role_id: v ?? '' } })
}
/>
<span class="text-2xs text-secondary">
The secret_id is read from the <code>VAULT_APPROLE_SECRET_ID</code> environment variable
of every server and worker, so it is never stored in Windmill. Leave it unset for a role
with <code>bind_secret_id=false</code>.
</span>
<label for="ext_vault_approle_mount" class="text-xs font-semibold text-emphasis">
AppRole mount path (optional)
</label>
<TextInput
inputProps={{ id: 'ext_vault_approle_mount', placeholder: 'approle', disabled }}
bind:value={
() => vault?.approle_mount_path ?? '', (v) => setVaultField('approle_mount_path', v)
}
/>
</div>
{:else if authMethod === 'jwt'}
<div class="flex flex-col gap-2 p-3 bg-surface-secondary rounded-lg">
<label for="ext_vault_jwt_role" class="text-xs font-semibold text-emphasis">
JWT auth role
</label>
<TextInput
inputProps={{ id: 'ext_vault_jwt_role', placeholder: 'windmill-secrets', disabled }}
bind:value={
() => vault?.jwt_role ?? '', (v) => update({ vault: { ...vault, jwt_role: v ?? '' } })
}
/>
<label for="ext_vault_jwt_mount" class="text-xs font-semibold text-emphasis">
JWT auth mount path (optional)
</label>
<TextInput
inputProps={{ id: 'ext_vault_jwt_mount', placeholder: 'jwt', disabled }}
bind:value={
() => vault?.jwt_mount_path ?? '', (v) => setVaultField('jwt_mount_path', v)
}
/>
</div>
{:else}
<div class="flex flex-col gap-1 p-3 bg-surface-secondary rounded-lg">
<span class="text-xs font-semibold text-emphasis">Vault token</span>
<span class="text-2xs text-secondary">Recommended only for testing.</span>
<Password
bind:password={
() => vault?.token ?? '', (v) => update({ vault: { ...vault, token: v ?? '' } })
}
small
{disabled}
/>
</div>
{/if}
<div class="flex flex-col gap-1">
<label for="ext_vault_namespace" class="text-xs font-semibold text-emphasis">
Namespace (optional)
</label>
<TextInput
inputProps={{ id: 'ext_vault_namespace', placeholder: 'admin/my-namespace', disabled }}
bind:value={() => vault?.namespace ?? '', (v) => setVaultField('namespace', v)}
/>
</div>
<Toggle
{disabled}
checked={!!vault?.skip_ssl_verify}
on:change={(e) => setVaultField('skip_ssl_verify', e.detail || undefined)}
size="xs"
options={{ right: 'Skip TLS certificate verification' }}
/>
<Button
unifiedSize="md"
variant="default"
onclick={testConnection}
disabled={disabled || !vault?.address || testing}
loading={testing}
startIcon={{ icon: Server }}>Test connection</Button
>
</div>
{/if}
<div class="flex flex-col gap-2">
<span class="text-xs font-semibold text-emphasis">Allowed prefixes</span>
<span class="text-2xs text-secondary">
A reference resolves only under one of these prefixes, matched on whole path segments, for the
listed workspaces (all workspaces when empty). Any admin of a listed workspace can read every
secret under its prefixes, within what Windmill's Vault identity can read.
</span>
{#if allowed.length === 0}
<Alert type="warning" size="xs" title="No prefixes allowed yet">
References are refused until you add a prefix and choose its workspaces.
</Alert>
{/if}
{#each allowed as a, i (i)}
<div class="flex flex-row gap-2 items-center">
<TextInput
class="flex-1"
inputProps={{ placeholder: 'vault://secret/team', disabled }}
bind:value={() => a.prefix, (v) => setAllowed(i, { prefix: v ?? '' })}
/>
<MultiSelect
class="flex-1"
items={workspaces.current ?? []}
placeholder="All workspaces"
{disabled}
bind:value={() => a.workspaces, (v) => setAllowed(i, { workspaces: v })}
/>
<Button
unifiedSize="md"
variant="subtle"
iconOnly
startIcon={{ icon: Trash2 }}
{disabled}
aria-label="Remove prefix"
onclick={() => update({ allowed: allowed.filter((_, j) => j !== i) })}
/>
</div>
{/each}
<Button
unifiedSize="sm"
variant="default"
startIcon={{ icon: Plus }}
{disabled}
wrapperClasses="self-start"
onclick={() => update({ allowed: [...allowed, { prefix: 'vault://', workspaces: [] }] })}
>Add prefix</Button
>
</div>
</div>
@@ -1,5 +1,5 @@
<script lang="ts">
import { Button } from '$lib/components/common'
import { Alert, Button } from '$lib/components/common'
import Password from '../Password.svelte'
import { SettingService } from '$lib/gen'
import { sendUserToast } from '$lib/toast'
@@ -34,9 +34,10 @@
| 'AwsSecretsManager'
| 'HttpStore' = $derived($values['secret_backend']?.type ?? 'Database')
let authMethod: 'token' | 'jwt' = $derived.by(() => {
let authMethod: 'token' | 'approle' | 'jwt' = $derived.by(() => {
const config = $values['secret_backend']
if (!config || config.type !== 'HashiCorpVault') return 'jwt'
if (config.approle_role_id != null) return 'approle'
return config.jwt_role === null ? 'token' : 'jwt'
})
@@ -60,6 +61,22 @@
let vaultDisabled = $derived(!$enterpriseLicense)
// The Vault connection being switched away from, while external secret references
// still read through it.
let droppedVault: Record<string, any> | undefined = $state()
let refsLoseConnection = $derived(
!!droppedVault &&
selectedType !== 'HashiCorpVault' &&
!$values['external_secrets']?.vault &&
($values['external_secrets']?.allowed?.length ?? 0) > 0
)
function keepConnectionForRefs() {
if (!droppedVault) return
const { type: _t, mount_path: _m, kv_secret_path_prefix: _p, ...conn } = droppedVault
$values['external_secrets'] = { ...$values['external_secrets'], vault: conn }
}
function setBackendType(type: string | undefined) {
if (!type) return
if (
@@ -70,6 +87,9 @@
vaultDisabled
)
return
if ($values['secret_backend']?.type === 'HashiCorpVault' && type !== 'HashiCorpVault') {
droppedVault = $values['secret_backend']
}
if (type === 'Database') {
$values['secret_backend'] = { type: 'Database' }
} else if (type === 'HashiCorpVault') {
@@ -124,12 +144,21 @@
$values['secret_backend'] = {
...$values['secret_backend'],
jwt_role: null,
approle_role_id: undefined,
token: $values['secret_backend'].token ?? ''
}
} else if (method === 'approle') {
$values['secret_backend'] = {
...$values['secret_backend'],
jwt_role: null,
token: null,
approle_role_id: $values['secret_backend'].approle_role_id ?? ''
}
} else if (method === 'jwt') {
$values['secret_backend'] = {
...$values['secret_backend'],
token: null,
approle_role_id: undefined,
jwt_role: $values['secret_backend'].jwt_role ?? 'windmill-secrets'
}
}
@@ -142,6 +171,8 @@
kv_secret_path_prefix: $values['secret_backend'].kv_secret_path_prefix || undefined,
jwt_role: $values['secret_backend'].jwt_role,
jwt_mount_path: $values['secret_backend'].jwt_mount_path || undefined,
approle_role_id: $values['secret_backend'].approle_role_id ?? undefined,
approle_mount_path: $values['secret_backend'].approle_mount_path || undefined,
namespace: $values['secret_backend'].namespace || undefined,
token: $values['secret_backend'].token || undefined,
skip_ssl_verify: $values['secret_backend'].skip_ssl_verify || undefined
@@ -254,7 +285,8 @@
const hasMountPath = $values['secret_backend'].mount_path?.trim() !== ''
const hasToken = $values['secret_backend'].token?.trim()
const hasJwtRole = $values['secret_backend'].jwt_role?.trim()
return hasAddress && hasMountPath && (hasToken || hasJwtRole)
const hasAppRole = $values['secret_backend'].approle_role_id?.trim()
return hasAddress && hasMountPath && (hasToken || hasJwtRole || hasAppRole)
}
function getAzureKvSettings() {
@@ -461,6 +493,16 @@
/>
{/snippet}
</ToggleButtonGroup>
{#if refsLoseConnection}
<Alert
type="warning"
size="xs"
title="External secret references use this Vault connection"
actions={[{ label: 'Keep the connection for references', onClick: keepConnectionForRefs }]}
>
Keep it as their own connection, or they can no longer be read.
</Alert>
{/if}
{#if vaultDisabled}
<div class="flex items-center gap-1">
<EEOnly>External secret store integrations require Enterprise Edition</EEOnly>
@@ -547,6 +589,13 @@
item={toggleButton}
{disabled}
/>
<ToggleButton
value="approle"
label="AppRole"
tooltip="Log in with an AppRole role_id and secret_id"
item={toggleButton}
{disabled}
/>
<ToggleButton
value="token"
label="Static Token"
@@ -557,7 +606,34 @@
{/snippet}
</ToggleButtonGroup>
</div>
{#if authMethod === 'token'}
{#if authMethod === 'approle'}
<div class="flex flex-col gap-2 p-3 bg-surface-secondary rounded-lg">
<label for="vault_approle_role_id" class="block text-xs font-semibold text-emphasis"
>Role ID</label
>
<TextInput
inputProps={{ type: 'text', id: 'vault_approle_role_id', disabled }}
bind:value={$values['secret_backend'].approle_role_id}
/>
<span class="text-2xs text-secondary"
>The secret_id is read from the <code>VAULT_APPROLE_SECRET_ID</code> environment
variable of every server and worker, so it is never stored in Windmill. Leave it unset
for a role with <code>bind_secret_id=false</code>.</span
>
<label for="vault_approle_mount_path" class="block text-xs font-semibold text-emphasis"
>AppRole Mount Path (optional)</label
>
<TextInput
inputProps={{
type: 'text',
id: 'vault_approle_mount_path',
placeholder: 'approle',
disabled
}}
bind:value={$values['secret_backend'].approle_mount_path}
/>
</div>
{:else if authMethod === 'token'}
<div class="flex flex-col gap-1 p-3 bg-surface-secondary rounded-lg">
<label for="vault_token" class="block text-xs font-semibold text-emphasis"
>Vault Token</label
@@ -59,6 +59,7 @@
} from 'lucide-svelte'
import { untrack } from 'svelte'
import { page } from '$app/stores'
import { describeExternalRef, isExternalRef } from '$lib/externalSecretRef'
import { useHostedPage } from '$lib/components/hostedPage'
import {
useOperatingWorkspace,
@@ -490,7 +491,9 @@
<Cell>
<span class="inline-flex flex-row items-center gap-2">
<div class="text-xs break-words">
{#if value}
{#if is_secret && isExternalRef(value)}
{describeExternalRef(value ?? '')}
{:else if value}
{truncate(value, 20)}
{:else}
&lowast;&lowast;&lowast;&lowast;
+26
View File
@@ -0,0 +1,26 @@
// A secret variable whose stored value is `$external:<scheme>://<path>[#<key>]` is resolved
// from an external secret store each time it is read, without a key as a JSON object of all
// the secret's fields; the backend parses and authorizes it
// (`secret_backend/external_ref.rs`). Only Vault is supported.
export const EXTERNAL_REF_PREFIX = '$external:'
const VAULT_PREFIX = `${EXTERNAL_REF_PREFIX}vault://`
export function isExternalRef(value: string | undefined | null): boolean {
return !!value?.startsWith(EXTERNAL_REF_PREFIX)
}
export function parseVaultRef(value: string): { path: string; key: string } {
const rest = value.startsWith(VAULT_PREFIX) ? value.slice(VAULT_PREFIX.length) : ''
const i = rest.indexOf('#')
return i < 0 ? { path: rest, key: '' } : { path: rest.slice(0, i), key: rest.slice(i + 1) }
}
export function formatVaultRef(path: string, key: string): string {
return `${VAULT_PREFIX}${path}${key ? `#${key}` : ''}`
}
/** "Vault · secret/team/app#token", or "Vault · secret/team/app" for the whole secret, for lists. */
export function describeExternalRef(value: string): string {
const { path, key } = parseVaultRef(value)
return `Vault · ${path}${key ? `#${key}` : ''}`
}