windmill/examples/deploy/instance-config-iac/README.md

# Windmill Instance Configuration as Code

Windmill supports managing instance configuration (global settings + worker group configs) declaratively through YAML files. This enables Infrastructure-as-Code (IaC) workflows where your Windmill instance settings are version-controlled and applied automatically.

Two deployment models are supported:

| Approach | Best for | Requires |
|---|---|---|
| **`sync-config`** CLI | Docker Compose, VMs, CI/CD pipelines | Database access |
| **Kubernetes Operator** | Kubernetes clusters | `operator` feature flag, RBAC |

Both use the same YAML schema (`InstanceConfig`) and the same secret reference mechanisms.

---

## Config File Reference

A Windmill instance config file has two top-level keys:

```yaml
global_settings:
  # Instance-wide settings (stored in the global_settings table)
  base_url: "https://windmill.example.com"
  retention_period_secs: 2592000
  # ...

worker_configs:
  # Worker group configurations (stored in the config table as worker__<name>)
  default:
    worker_tags: ["deno", "python3", "bun", "go", "bash"]
  gpu:
    dedicated_worker: "ws:f/gpu_inference"
    # ...
```

All fields are optional. Only the fields you specify are synced to the database.

### Sensitive Field References

Fields that contain secrets (license keys, OAuth secrets, SMTP passwords, etc.) support three formats:

```yaml
# 1. Plain literal (not recommended for production)
license_key: "my-license-key"

# 2. Environment variable reference (works everywhere)
license_key:
  envRef: "WM_LICENSE_KEY"

# 3. Kubernetes Secret reference (K8s only)
license_key:
  secretKeyRef:
    name: windmill-secrets    # Secret resource name
    key: license-key          # Key within the Secret
```

Fields that support `envRef` and `secretKeyRef`:

- `license_key`
- `hub_api_secret`
- `scim_token`
- `smtp_settings.smtp_password`
- `oauths.<provider>.secret` (each OAuth client secret)
- `custom_instance_pg_databases.user_pwd`

---

## Docker Compose (`sync-config`)

The `sync-config` subcommand reads a YAML config file, resolves any `envRef` references from the process environment, and syncs the result to the database.

### How it works

1. Windmill reads and parses the YAML file
2. Any `envRef` fields are resolved from the container's environment variables
3. The current database state is read
4. A diff is computed (using `Replace` mode: settings absent from the file are deleted, except protected ones like `ducklake_settings`)
5. Changes are applied to the database

### Setup

See the included [`docker-compose.yml`](docker-compose.yml) for a complete working example. The key parts:

**1. Create your config file** (`windmill-config.yaml`):

```yaml
global_settings:
  base_url: "https://windmill.example.com"
  license_key:
    envRef: "WM_LICENSE_KEY"
  retention_period_secs: 2592000
  expose_metrics: true
  smtp_settings:
    smtp_host: "smtp.example.com"
    smtp_port: 587
    smtp_from: "windmill@example.com"
    smtp_password:
      envRef: "SMTP_PASSWORD"
  oauths:
    google:
      id: "google-client-id"
      secret:
        envRef: "GOOGLE_OAUTH_SECRET"
      login_config:
        auth_url: "https://accounts.google.com/o/oauth2/v2/auth"
        token_url: "https://oauth2.googleapis.com/token"
        userinfo_url: "https://openidconnect.googleapis.com/v1/userinfo"
        scopes: ["openid", "profile", "email"]
  custom_tags:
    - gpu
    - high-mem

worker_configs:
  default:
    worker_tags: ["deno", "python3", "bun", "go", "bash", "powershell"]
    init_bash: "echo 'Worker starting'"
  native:
    worker_tags: ["nativets"]
```

**2. Add an init container to `docker-compose.yml`**:

```yaml
services:
  windmill_config_sync:
    image: ${WM_IMAGE}
    # Run once at startup then exit
    restart: "no"
    command: ["windmill", "sync-config", "/config/windmill-config.yaml"]
    environment:
      - DATABASE_URL=${DATABASE_URL}
      - WM_LICENSE_KEY=${WM_LICENSE_KEY}
      - SMTP_PASSWORD=${SMTP_PASSWORD}
      - GOOGLE_OAUTH_SECRET=${GOOGLE_OAUTH_SECRET}
    volumes:
      - ./windmill-config.yaml:/config/windmill-config.yaml:ro
    depends_on:
      db:
        condition: service_healthy
```

**3. Set secrets in your `.env` file** (not committed to version control):

```env
DATABASE_URL=postgres://postgres:changeme@db/windmill
WM_IMAGE=ghcr.io/windmill-labs/windmill-ee:main
WM_LICENSE_KEY=your-license-key-here
SMTP_PASSWORD=your-smtp-password
GOOGLE_OAUTH_SECRET=your-google-oauth-secret
```

### Re-syncing after config changes

The `sync-config` container runs once and exits. To re-apply after editing the YAML:

```bash
docker compose run --rm windmill_config_sync
```

Or, for CI/CD pipelines, run the binary directly:

```bash
windmill sync-config ./windmill-config.yaml
```

### Replace semantics

`sync-config` uses **Replace** mode: any global setting present in the database but absent from your YAML file will be **deleted** (except protected settings like `ducklake_settings` and `custom_instance_pg_databases`). This ensures the database state matches the file exactly.

If you only want to manage a subset of settings, include all settings you want to keep in the YAML file.

---

## Kubernetes (Operator)

The Windmill Kubernetes operator watches `WindmillInstance` Custom Resources and continuously reconciles the database to match the declared state. It also supports `secretKeyRef` to pull values from Kubernetes Secrets natively.

### Prerequisites

- Windmill built with the `operator` feature flag
- RBAC permissions for the operator pod (see below)
- The CRD installed in the cluster

### Setup

**1. Install the CRD**:

```bash
windmill operator crd | kubectl apply -f -
```

**2. Create a Kubernetes Secret for sensitive values**:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: windmill-secrets
  namespace: windmill
type: Opaque
stringData:
  license-key: "your-license-key-here"
  smtp-password: "your-smtp-password"
  google-oauth-secret: "your-google-oauth-secret"
```

**3. Create the WindmillInstance resource** (`windmill-instance.yaml`):

```yaml
apiVersion: windmill.dev/v1alpha1
kind: WindmillInstance
metadata:
  name: production
  namespace: windmill
spec:
  global_settings:
    base_url: "https://windmill.example.com"
    license_key:
      secretKeyRef:
        name: windmill-secrets
        key: license-key
    retention_period_secs: 2592000
    expose_metrics: true
    smtp_settings:
      smtp_host: "smtp.example.com"
      smtp_port: 587
      smtp_from: "windmill@example.com"
      smtp_password:
        secretKeyRef:
          name: windmill-secrets
          key: smtp-password
    oauths:
      google:
        id: "google-client-id"
        secret:
          secretKeyRef:
            name: windmill-secrets
            key: google-oauth-secret
        login_config:
          auth_url: "https://accounts.google.com/o/oauth2/v2/auth"
          token_url: "https://oauth2.googleapis.com/token"
          userinfo_url: "https://openidconnect.googleapis.com/v1/userinfo"
          scopes: ["openid", "profile", "email"]
    custom_tags:
      - gpu
      - high-mem

  worker_configs:
    default:
      worker_tags: ["deno", "python3", "bun", "go", "bash", "powershell"]
      init_bash: "echo 'Worker starting'"
    native:
      worker_tags: ["nativets"]
```

**4. Apply**:

```bash
kubectl apply -f windmill-instance.yaml
```

**5. Check status**:

```bash
kubectl get wmi
# NAME         SYNCED   LAST SYNCED              AGE
# production   true     2025-01-15T10:30:00Z     2d
```

### Using `envRef` in Kubernetes

`envRef` also works in the operator context. Values are resolved from the operator pod's environment. This is useful when secrets are injected via pod env vars (e.g., from a vault sidecar):

```yaml
spec:
  global_settings:
    license_key:
      envRef: "WM_LICENSE_KEY"   # Read from operator pod env
```

The operator pod's Deployment would include:

```yaml
env:
  - name: WM_LICENSE_KEY
    valueFrom:
      secretKeyRef:
        name: windmill-secrets
        key: license-key
```

This is functionally equivalent to using `secretKeyRef` directly in the CRD, but lets you use any secret injection mechanism your cluster supports (external-secrets, vault-agent, etc.).

### RBAC

The operator pod needs permissions to read Secrets and manage the CRD. Minimal ClusterRole:

```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: windmill-operator
rules:
  - apiGroups: ["windmill.dev"]
    resources: ["windmillinstances", "windmillinstances/status"]
    verbs: ["get", "list", "watch", "patch", "update"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "patch"]
```

### Running the operator

```bash
# As a standalone process (for development)
DATABASE_URL=postgres://... windmill operator

# In production, deploy as a Kubernetes Deployment
```

---

## Choosing Between `envRef` and `secretKeyRef`

| Feature | `envRef` | `secretKeyRef` |
|---|---|---|
| Works in Docker Compose | Yes | No |
| Works in Kubernetes | Yes | Yes |
| Works with vault sidecars | Yes | No (use `envRef` instead) |
| Reads from | Process environment | K8s Secrets API |
| Requires RBAC for Secrets | No | Yes |

**Recommendation**: Use `envRef` for portability across deployment targets. Use `secretKeyRef` when you want direct Kubernetes-native secret binding without intermediate env vars.

---

## Full Settings Reference

For a complete list of available settings fields, generate the CRD schema:

```bash
windmill operator crd
```

The CRD's OpenAPI schema documents every field, its type, and whether it's optional. The same schema applies to `sync-config` YAML files.