Files
windmill/frontend/src/lib/components/instanceSettings.ts
T
577ceeee86 perf(audit): re-anchor S3 audit export on enable + opt-in backfill (#9818)
* [ee] perf(audit): re-anchor S3 audit export on enable + opt-in backfill

The S3/GCS audit-log export's steady-state query filters by `age(xmin)`
(unindexable), so the only scan bound is the timestamp floor. On a fresh
enable the floor was epoch, and on a re-enable the cursor resumed from its
pre-disable position — either way the first run scanned the whole
`audit_partitioned` table. Under a `statement_timeout` (e.g. Aiven) that scan
never completes: the cursor never advances, nothing is exported, and the
repeated full scans saturate the database.

Re-anchor on enable (EE companion, windmill-ee-private#634):
- New trigger migration records a recent timestamp floor instead of the epoch
  sentinel and `DO UPDATE`s the cursor to the current snapshot xmin on
  re-enable, so the export always resumes from ~now and never rescans history.
  Includes a one-time fixup for legacy epoch-sentinel checkpoints on upgrade.

Opt-in historical backfill (new `audit_logs_s3_backfill` module + endpoints):
- Exports a chosen `[from, to)` window on demand, scanning strictly by
  `timestamp` (the partition key) in bounded keyset pages — each query is an
  index scan capped at one page (verified via EXPLAIN: later partitions
  `never executed`, ~11ms/page), so it stays well under any statement timeout
  regardless of window size. Writes alongside the steady-state objects under
  logs/audit/, without touching the xmin cursor.
- POST /settings/audit_logs_s3_backfill {from,to} (super-admin + Enterprise),
  GET /settings/audit_logs_s3_backfill_status.

Also repurposes the status endpoint's `bootstrapping` flag to mean "draining a
backlog" (the cursor is capped and catching up), and updates the setting
description to point operators at the backfill.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(audit): heartbeat backfill lease per object; bump EE ref

Address review (cubic): persist progress (refreshing the lease heartbeat) after
every object PUT in the backfill page loop, not only once per page, so the gap
between heartbeats stays well under STALE_HEARTBEAT_SECS even on slow uploads
and another replica can't re-claim mid-page and run a concurrent backfill.

Bumps ee-repo-ref.txt to pull in the EE test-race fix (folding the backlog-drain
regression into the single audit e2e test).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(audit): reject unstable backfill windows; bump EE ref

Address review (P1): the backfill keyset-pages over rows visible at scan time
and declares completion when the scan runs dry, but a row's `timestamp` is its
inserting transaction's `xact_start`. A window whose upper bound is recent or in
the future could silently omit a transaction that started inside `[from, to)`
but commits after the scan passed that timestamp. `try_start` now rejects any
`to` newer than the oldest in-flight `xact_start` (everything strictly older
than the oldest running transaction is committed and stable), using the same
trustworthy stats gating as the exporter's floor (restricted role / 2PC → a
7-day-old cutoff).

Bumps ee-repo-ref.txt for the EE monotonic-checkpoint fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(audit): re-anchor legacy epoch checkpoints instead of synthetic floor

Address review (P1): the legacy-checkpoint fixup stamped last_oldest_inflight_ts
to now()-7d while leaving the old last_xmin in place. On an instance that
enabled export on the old code >7 days ago and got stuck before the first
successful batch, the next run would filter post-enable rows older than 7 days
out via `timestamp >= ts_floor` while still advancing last_xmin over the
interval — silently dropping them (the same floor-vs-cursor loss class fixed
elsewhere in this PR), and contradicting the "nothing committed after enabling
is skipped" guarantee.

A stuck epoch-sentinel checkpoint cannot be safely resumed (its backlog can be
arbitrarily old, so any recent floor prunes rows the cursor then skips, and an
epoch floor reintroduces the full scan). Re-anchor it to the migration's current
snapshot xmin instead — exactly like a fresh enable — so the export resumes
cleanly from ~now and the never-exported pre-upgrade window is recovered via the
opt-in backfill rather than silently dropped. Reword the setting description so
it no longer implies the disabled/legacy window is covered by the cursor.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(audit): end-to-end integration tests for the object-store backfill

The backfill previously had only SQL-level/EXPLAIN validation. Add real
integration tests (in-memory object store, sqlx::test) exercising the public
path:

- backfill_exports_window_in_pages: with the page size forced to 2 rows, a
  settled 3-day window is exported across multiple keyset pages; asserts every
  in-window row lands exactly once, rows outside [from,to) are excluded, a day
  that straddles a page boundary yields more than one object, progress counts
  match, and a re-run is idempotent (deterministic keys overwritten, no dupes).
- backfill_rejects_unstable_window: a future/live `to` is rejected as unstable,
  a window safely in the past is accepted.

Adds a test-only PAGE_ROWS override so multi-page behaviour is exercised with a
handful of rows.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(audit): note backfill scope is audit_partitioned only

Make explicit that, like the steady-state export, the backfill reads only
audit_partitioned; the pre-partitioning `audit` table is intentionally out of
scope (not a missed case).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(audit): reject backfill windows before the partitioned boundary

Address review (Codex P1): the backfill reads only audit_partitioned, but
pre-partitioning history lives in the legacy `audit` table (still read by audit
list/get via UNION ALL, and retained for the configured period — 365 days by
default on EE). Since the setting text points operators at this API for
"pre-existing history", a window overlapping legacy rows would report completion
while silently omitting them.

Per the decision to not export the legacy table, reject instead of silently
omit: try_start now rejects a `from` earlier than the oldest audit_partitioned
timestamp (every legacy row predates the partition cutover, so a `from` at/after
that boundary can never overlap them). Reworded the setting text to scope the
backfill to the partitioned era. Added a regression test, plus an RAII guard
(cubic P2) so the test-only globals are restored even if an assertion panics.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(audit): backfill object keys per-window; require trustworthy settled cutoff

Address review (two P1s):

- Object-key overwrite loss: keys were `dt=<day>/audit_backfill_<min_id>.ndjson`.
  A narrower, overlapping backfill can start a day's page at the same first row
  (same min_id) but hold fewer rows, and `put` would overwrite a broader run's
  object — silently dropping the rows only that object held. Include the
  requested window in the key so different ranges write disjoint objects (same
  window re-runs stay idempotent; consumers dedupe overlapping rows by id). New
  regression test (verified red→green).

- Untrustworthy settled cutoff: when min(xact_start) isn't trustworthy (role
  lacks pg_read_all_stats/superuser, or a prepared 2PC txn exists), the old
  now()-7d fallback could still let an old transaction commit rows inside an
  accepted window after the scan, so a "complete" backfill silently missed them.
  Since a backfill asserts completeness, reject in those cases instead of
  falling back. (The continuous exporter keeps its 7-day fallback — it only
  claims bounded lag.)

Also makes the tests robust under the parallel runner: run_backfill takes the
store as a param, so tests pass a local in-memory store (no global
OBJECT_STORE_SETTINGS race) and serialize on the PAGE_ROWS override.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(audit): reject backfill overlapping legacy table; regen deref openapi; trim migration comment

Address review (1 P1 + 2 P2):

- Empty-partition backfill (P1): the min(audit_partitioned) guard no-ops when
  audit_partitioned is empty, so an upgraded instance with legacy `audit` rows
  but no partitioned rows yet would accept a window and complete with zero rows,
  silently omitting the legacy rows. Check the legacy `audit` table directly:
  reject any window that overlaps a legacy row (subsumes the boundary check and
  covers the empty-partitioned case). Test updated accordingly.

- openapi-deref (P2): regenerate openapi-deref.yaml/json (served via include_str!)
  so /openapi.{yaml,json} expose the new backfill endpoints.

- Migration comment (P2): trim the PR-history narration to the durable
  constraints (why a recent floor and a monotonic cursor are required), per
  AGENTS.md.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: update ee-repo-ref to b821fecccbcba2efed544890576bf2b84321d70d

This commit updates the EE repository reference after PR #634 was merged in windmill-ee-private.

Previous ee-repo-ref: 6b191b77aabcf77658ad4f9031576e0d7b66bf89

New ee-repo-ref: b821fecccbcba2efed544890576bf2b84321d70d

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-06-26 21:37:33 +02:00

1263 lines
39 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import type { ButtonType } from './common/button/model'
import { z } from 'zod'
// Languages that support HTTP request tracing via OTEL proxy
export const OTEL_TRACING_PROXY_LANGUAGES = [
'nativets',
'python3',
'deno',
'bun',
'go',
'bash',
'rust',
'csharp',
'nu',
'ruby'
] as const
export interface Setting {
label: string
description?: string
placeholder?: string
cloudonly?: boolean
ee_only?: string
tooltip?: string
key: string
// If value is not specified for first element, it will automatcally use undefined
select_items?: {
label: string
tooltip?: string
// If not specified, label will be used
value?: string
}[]
fieldType:
| 'text'
| 'number'
| 'boolean'
| 'password'
| 'select'
| 'select_python'
| 'textarea'
| 'codearea'
| 'seconds'
| 'email'
| 'license_key'
| 'object_store_config'
| 'critical_error_channels'
| 'critical_alerts_on_db_oversize'
| 'slack_connect'
| 'smtp_connect'
| 'indexer_rates'
| 'otel'
| 'otel_tracing_proxy'
| 'secret_backend'
| 'github_enterprise_app'
| 'ws_connectivity'
storage: SettingStorage
advancedToggle?: {
label: string
onChange: (values: Record<string, any>) => Record<string, any>
checked: (values: Record<string, any>) => boolean
}
hiddenIfNull?: boolean
hiddenIfEmpty?: boolean
hiddenInEe?: boolean
hideInQuickSetup?: boolean
requiresReloadOnChange?: boolean
triggersRestart?: boolean
isValid?: (value: any) => boolean
validate?: (value: any) => Record<string, string>
error?: string
defaultValue?: () => any
codeAreaLang?: string
actionButton?: {
label: string
onclick: (values: Record<string, any>) => Promise<void>
variant?: ButtonType.Variant
}
}
export type SettingStorage = 'setting'
const positiveNumber = z.number().positive('Must be a positive number')
const nonNegativeNumber = z.number().nonnegative('Must be zero or a positive number')
const indexerSettingsSchema = z
.object({
writer_memory_budget: positiveNumber.optional(),
commit_job_max_batch_size: positiveNumber.optional(),
refresh_index_period: positiveNumber.optional(),
max_indexed_job_log_size: positiveNumber.optional(),
commit_log_max_batch_size: positiveNumber.optional(),
refresh_log_index_period: positiveNumber.optional(),
max_index_time_window_secs: nonNegativeNumber.optional()
})
.passthrough()
function validateIndexerSettings(v: any): Record<string, string> {
if (!v) return {}
const result = indexerSettingsSchema.safeParse(v)
if (result.success) return {}
const errors: Record<string, string> = {}
for (const issue of result.error.issues) {
const field = issue.path[0]?.toString()
if (field) errors[field] = issue.message
}
return errors
}
export const scimSamlSetting: Setting[] = [
{
label: 'SCIM token',
description: 'Token used to authenticate requests from the IdP',
key: 'scim_token',
fieldType: 'password',
placeholder: 'mytoken',
storage: 'setting',
ee_only: ''
},
{
label: 'SAML metadata',
description: 'XML metadata url OR content for the SAML IdP',
key: 'saml_metadata',
fieldType: 'textarea',
placeholder: 'https://dev-2578259.okta.com/app/exkaell8gidiiUWrg5d7/sso/saml/metadata ',
storage: 'setting',
ee_only: '',
triggersRestart: true
}
]
export const settings: Record<string, Setting[]> = {
Core: [
{
label: 'Base url',
description:
'Public base url of the instance. <a href="https://www.windmill.dev/docs/advanced/instance_settings#global-users">Learn more</a>',
key: 'base_url',
fieldType: 'text',
placeholder: 'https://windmill.com',
storage: 'setting',
error: 'Base url must start with http:// or https:// and not end with / or a space',
isValid: (value: string | undefined) =>
value == undefined ||
(value?.startsWith('http') &&
value.includes('://') &&
!value?.endsWith('/') &&
!value?.endsWith(' '))
},
{
label: 'Email domain',
description: 'Domain to display in webhooks for email triggers (should match the MX record)',
key: 'email_domain',
fieldType: 'text',
placeholder: 'mail.windmill.com',
storage: 'setting',
triggersRestart: true,
error: 'Must be a valid domain',
isValid: (value: string | undefined) =>
value == undefined ||
value === '' ||
/^[a-zA-Z0-9]([a-zA-Z0-9-]*[a-zA-Z0-9])?(\.[a-zA-Z0-9]([a-zA-Z0-9-]*[a-zA-Z0-9])?)*$/.test(
value
)
},
{
label: 'Request size limit in MB',
description: 'Maximum size of HTTP requests in MB.',
cloudonly: true,
key: 'request_size_limit_mb',
fieldType: 'number',
placeholder: '50',
storage: 'setting',
triggersRestart: true
},
{
label: 'License key',
description:
'License key required to use the EE (switch image for windmill-ee). <a href="https://www.windmill.dev/docs/advanced/instance_settings#license-key">Learn more</a>',
key: 'license_key',
fieldType: 'license_key',
placeholder: 'only for EE',
storage: 'setting'
},
{
label: 'Non-prod instance',
description:
'Whether we should consider the reported usage of this instance as non-prod. <a href="https://www.windmill.dev/docs/advanced/instance_settings#non-prod-instance">Learn more</a>',
key: 'dev_instance',
fieldType: 'boolean',
storage: 'setting',
ee_only: '',
hideInQuickSetup: true
},
{
label: 'App workspace prefix',
description:
'When enabled apps will be accessible at /a/{workspace_id}/{custom_path} instead of /a/{custom_path} allowing you to define same custom path for apps in different workspace without conflict',
key: 'app_workspaced_route',
fieldType: 'boolean',
storage: 'setting',
ee_only: '',
hideInQuickSetup: true
},
{
label: 'HTTP route workspace prefix',
description:
'When enabled HTTP routes will be accessible at /api/r/{workspace_id}/{route} instead of /api/r/{route} allowing you to define same route path in different workspaces without conflict',
key: 'http_route_workspaced_route',
fieldType: 'boolean',
storage: 'setting',
ee_only: '',
hideInQuickSetup: true
},
{
label: 'Audit log retention (days)',
key: 'audit_log_retention_days',
description: 'How long to keep audit log entries in the database. Default: 365 days.',
fieldType: 'number',
placeholder: '365',
storage: 'setting',
ee_only: '',
hideInQuickSetup: true
}
],
Jobs: [
{
label: 'Job isolation',
key: 'job_isolation',
fieldType: 'select',
description:
'Isolation mode for job execution. None: no isolation. Unshare: PID namespace isolation via unshare. Nsjail: full nsjail sandboxing. <a href="https://www.windmill.dev/docs/advanced/security_isolation">Learn more</a>',
storage: 'setting',
select_items: [
{
label: 'None',
value: 'none'
},
{
label: 'Unshare',
value: 'unshare'
},
{
label: 'Nsjail',
value: 'nsjail_sandboxing'
}
]
},
{
label: 'Nsjail /tmp backing',
key: 'nsjail_tmp_backing',
fieldType: 'select',
description:
'How <code>/tmp</code> is backed inside the nsjail sandbox. <strong>RAM (tmpfs)</strong> is the default — fast, with a hard size cap from <em>Nsjail tmpfs size</em>, but consumes worker memory. <strong>Disk (bind mount)</strong> uses a per-job directory on the worker disk — no RAM cost, but the only remaining per-file ceiling is <code>rlimit_fsize</code> (~1GB for python/ansible, unbounded for most other languages because they set <code>disable_rl: true</code>); pair with host disk monitoring or quotas.',
storage: 'setting',
placeholder: 'tmpfs',
defaultValue: () => 'tmpfs',
select_items: [
{ label: 'RAM (tmpfs) — default', value: 'tmpfs' },
{ label: 'Disk (bind mount)', value: 'disk' }
]
},
{
label: 'Nsjail tmpfs size (MB)',
key: 'nsjail_tmpfs_size_mb',
description:
'Override the size of the <code>/tmp</code> tmpfs mount inside the nsjail sandbox (in MB). When left empty, defaults to 800MB. Only applies when <em>Nsjail /tmp backing</em> is RAM (tmpfs).',
fieldType: 'number',
placeholder: '800',
storage: 'setting'
},
{
label: 'Sandbox image max size (MB)',
key: 'sandbox_image_max_size_mb',
description:
'Reject a <code># sandbox &lt;image&gt;</code> whose compressed download size exceeds this many MB, before any layer is downloaded. Leave empty for no limit.',
fieldType: 'number',
placeholder: 'no limit',
storage: 'setting'
},
{
label: 'Sandbox image cache cap (MB)',
key: 'sandbox_image_cache_max_mb',
description:
"Best-effort cap on the worker's cached sandbox rootfs tars. When exceeded, the oldest (by creation time) are evicted after a run. Leave empty for unbounded.",
fieldType: 'number',
placeholder: 'unbounded',
storage: 'setting'
},
{
label: 'Sandbox image pull policy',
key: 'sandbox_image_pull_policy',
description:
'When to re-pull a <code># sandbox</code> image. <strong>newer</strong> (default) re-pulls only when the registry digest changed, so moving tags like <code>:latest</code> stay fresh without re-downloading unchanged layers. <strong>missing</strong> pulls only if absent (fastest, tags can go stale). <strong>always</strong> re-checks every job.',
fieldType: 'select',
storage: 'setting',
placeholder: 'newer',
defaultValue: () => 'newer',
select_items: [
{ label: 'Newer (default)', value: 'newer' },
{ label: 'Missing', value: 'missing' },
{ label: 'Always', value: 'always' },
{ label: 'Never', value: 'never' }
]
},
{
label: 'Sandbox image default registry',
key: 'sandbox_image_default_registry',
description:
'If set, unqualified <code># sandbox</code> images (e.g. <code>alpine</code>) are pulled from this registry instead of <code>docker.io</code>. Fully-qualified refs (e.g. <code>ghcr.io/org/img</code>) are unaffected. Example: <code>myregistry.example.com</code>.',
fieldType: 'text',
placeholder: 'docker.io',
storage: 'setting'
},
{
label: 'Sandbox registry auth',
key: 'sandbox_registry_auth',
description:
'Credentials for private registries used by <code># sandbox</code> images, in docker <code>config.json</code> / <code>auth.json</code> format. Written to a per-job <code>DOCKER_CONFIG</code> dir (removed with the job) and used by crane for the pull.',
fieldType: 'codearea',
codeAreaLang: 'json',
placeholder:
'{\n "auths": {\n "myregistry.example.com": {\n "auth": "BASE64(username:password)"\n }\n }\n}',
storage: 'setting'
},
{
label: 'SSH execution (#ssh)',
key: 'ssh_execution_enabled',
fieldType: 'boolean',
description:
'Allow bash scripts starting with a <code>#ssh &lt;resource_path&gt;</code> directive to run on the remote host described by the referenced <code>ssh_target</code> resource instead of the worker. Off by default.',
storage: 'setting',
ee_only: '',
hideInQuickSetup: true
},
{
label: 'Default timeout',
key: 'job_default_timeout',
description:
'Default timeout for individual jobs. <a href="https://www.windmill.dev/docs/core_concepts/jobs#retention-policy">Learn more</a>',
fieldType: 'seconds',
storage: 'setting',
cloudonly: false
},
{
label: 'Max timeout for sync endpoints',
description:
'Maximum amount of time (measured in seconds) that a <a href="https://www.windmill.dev/docs/core_concepts/webhooks">sync endpoint</a> is allowed to run before it is forcibly stopped or timed out.',
key: 'timeout_wait_result',
fieldType: 'seconds',
placeholder: '60',
storage: 'setting'
},
{
label: 'Keep job directories for debug',
key: 'keep_job_dir',
fieldType: 'boolean',
description: 'Keep Job directories after execution at /tmp/windmill/WORKER/JOB_ID',
storage: 'setting'
},
{
label: 'Retention period in secs',
key: 'retention_period_secs',
description:
'How long to keep the jobs data in the database (max 30 days on CE). <a href="https://www.windmill.dev/docs/advanced/instance_settings#retention-period-in-secs">Learn more</a>',
fieldType: 'seconds',
placeholder: '30',
storage: 'setting',
ee_only: 'You can only adjust this setting to above 30 days in the EE version',
cloudonly: false
},
{
label: 'Workspace fairness — enabled',
description:
'Multi-tenant safeguard against a single workspace dominating the shared worker pool. <strong>Only relevant on instances where multiple workspaces share one worker group</strong> — single-tenant deployments do not need this. When a workspace accounts for at least <em>Workspace fairness — max percent</em> of cluster activity over the last <em>Workspace fairness — duration</em> seconds, each worker pull stochastically excludes that workspace so its share converges to the cap without on/off oscillation. Idle workers always fall back to running its jobs, so capping never starves the queue.',
key: 'workspace_fairness_enabled',
fieldType: 'boolean',
storage: 'setting',
cloudonly: false,
ee_only:
'Workspace fairness is an Enterprise feature — only useful on multi-tenant clusters where one noisy workspace would otherwise degrade QoS for other workspaces sharing the same worker pool.',
hideInQuickSetup: true
},
{
label: 'Workspace fairness — max percent',
description:
'Maximum share of cluster activity any single workspace may sustain before being stochastically throttled by the pull query. The admitted probability for capped workspaces is set just above this value so the cap is statistically stable rather than oscillating. Default 50.',
key: 'workspace_fairness_max_percent',
fieldType: 'number',
placeholder: '50',
storage: 'setting',
cloudonly: false,
ee_only: 'Workspace fairness is an Enterprise feature.',
hideInQuickSetup: true
},
{
label: 'Workspace fairness — duration (seconds)',
description:
'Rolling window used to measure workspace share. Activity = currently running jobs ∪ jobs completed in the last N seconds. Default 10.',
key: 'workspace_fairness_duration_secs',
fieldType: 'seconds',
placeholder: '10',
storage: 'setting',
cloudonly: false,
ee_only: 'Workspace fairness is an Enterprise feature.',
hideInQuickSetup: true
},
{
label: 'Workspace fairness — minimum total jobs',
description:
'Cap is only applied when cluster-wide activity exceeds this floor. Prevents over-eager capping on small clusters or quiet periods. Default 4.',
key: 'workspace_fairness_min_total_jobs',
fieldType: 'number',
placeholder: '4',
storage: 'setting',
cloudonly: false,
ee_only: 'Workspace fairness is an Enterprise feature.',
hideInQuickSetup: true
}
],
'Object Storage': [
{
label: 'Instance object storage',
description:
' S3/Azure bucket to store large logs and global cache for Python and Go. <a href="https://www.windmill.dev/docs/core_concepts/object_storage_in_windmill#instance-object-storage">Learn more</a>',
key: 'object_store_cache_config',
fieldType: 'object_store_config',
storage: 'setting',
ee_only: '',
isValid: (v) => {
if (!v || v.type !== 'Gcs') return true
return v.serviceAccountKey !== undefined
}
},
{
label: 'Delete logs from s3 periodically',
description:
'Job and service logs are periodically deleted from disk when they expire. When this setting is on, they are also deleted from object storage. Defaults to on when object storage is configured; turn off to keep logs in object storage indefinitely.',
key: 'monitor_logs_on_s3',
fieldType: 'boolean',
storage: 'setting',
ee_only: ''
},
{
label: 'Store audit logs in object storage',
description:
'When enabled and instance object storage is configured, audit logs are also exported as newline-delimited JSON to the dedicated logs/audit/ folder (partitioned by day). Export is incremental and runs off the hot path. Enabling (or re-enabling) anchors the export at ~now: while it stays enabled, every audit log committed from that point on is exported (transactions in flight at the moment of enabling may include a bounded set of just-prior rows). Pre-existing history, and any window during which export was disabled, are NOT exported by this cursor — use the opt-in backfill API to export a chosen historical range, back to when audit-log partitioning was introduced (older rows in the legacy audit table are not exported, and a window overlapping them is rejected): POST /settings/audit_logs_s3_backfill {from, to} (status at GET /settings/audit_logs_s3_backfill_status).',
key: 'store_audit_logs_s3',
fieldType: 'boolean',
storage: 'setting',
ee_only: '',
hideInQuickSetup: true
}
],
'Private Hub': [
{
label: 'Private Hub base url',
description:
'Base URL of your Private Hub instance, without trailing slash. <a href="https://www.windmill.dev/docs/core_concepts/private_hub">Learn more</a>',
placeholder: 'https://hub.company.com',
key: 'hub_base_url',
fieldType: 'text',
storage: 'setting',
ee_only: '',
advancedToggle: {
label: 'I have a different URL for Hub access from end-user browsers',
onChange(values) {
if (values['hub_accessible_url']) {
values['hub_accessible_url'] = null
} else {
values['hub_accessible_url'] = values['hub_base_url'] || 'https://hub.company.com'
}
return values
},
checked: (values) => values['hub_accessible_url'] != null
},
requiresReloadOnChange: true
},
{
label: 'Private Hub accessible url',
description:
'Base URL accessible from end-user browsers, without trailing slash. <a href="https://www.windmill.dev/docs/core_concepts/private_hub">Learn more</a>',
key: 'hub_accessible_url',
fieldType: 'text',
hiddenIfNull: true,
storage: 'setting',
ee_only: '',
requiresReloadOnChange: true
},
{
label: 'Private Hub API secret',
description:
'If access to your Private Hub is restricted, you can set the hub API secret here. <a href="https://www.windmill.dev/docs/core_concepts/private_hub">Learn more</a>',
key: 'hub_api_secret',
fieldType: 'password',
storage: 'setting',
ee_only: ''
},
{
label: 'Azure OpenAI base path',
description:
'All workspaces using an OpenAI resource for Windmill AI will run on the specified deployed model. Format: https://{your-resource-name}.openai.azure.com/openai/deployments/{deployment-id}. <a href="https://www.windmill.dev/docs/core_concepts/ai_generation#azure-openai-advanced-models">Learn more</a>',
key: 'openai_azure_base_path',
fieldType: 'text',
storage: 'setting',
ee_only: '',
hiddenIfEmpty: true
},
{
label: 'Disable Hub',
description:
'Disable the Windmill Hub integration entirely. Enable this if your instance runs in a closed environment without internet access and you do not have a private hub setup.',
key: 'disable_hub',
fieldType: 'boolean',
storage: 'setting',
ee_only: '',
requiresReloadOnChange: true
}
],
SMTP: [
{
label: 'SMTP configuration',
key: 'smtp_settings',
fieldType: 'smtp_connect',
storage: 'setting',
ee_only: ''
},
{
label: 'Disable workspace invite emails',
description:
'Do not send email notifications when a user is invited or added to a workspace. Useful for automated workflows that add users programmatically.',
key: 'disable_workspace_invite_emails',
fieldType: 'boolean',
storage: 'setting'
}
],
'Auth/OAuth/SAML': [
{
label: 'Disable password login',
description:
'Hide the email/password form on the login page and reject password login requests. Use when you only want OAuth/SAML logins.',
key: 'disable_password_login',
fieldType: 'boolean',
storage: 'setting'
},
{
label: 'Auto-login SSO provider',
description:
'If set, the login page redirects automatically to this provider. Use the OAuth provider key (e.g. "okta", "google") or "saml". The provider must be configured; otherwise the setting is ignored. Visit /user/login?no_sso=1 to bypass the redirect and fall back to the normal login form.',
key: 'auto_login_provider',
fieldType: 'text',
placeholder: 'okta',
storage: 'setting'
}
],
'DB Health': [],
Registries: [
{
label: 'Instance Python Version',
description: 'Default python version for newly deployed scripts',
key: 'instance_python_version',
fieldType: 'select_python',
// To change latest stable version:
// 1. Change placeholder in instanceSettings.ts
// 2. Change LATEST_STABLE_PY in dockerfile
// 3. Change #[default] annotation for PyVersion in backend
placeholder: '3.10,3.11,3.12,3.13',
select_items: [
{
label: 'Latest Stable',
value: 'default',
tooltip: 'python-3.12'
},
{
label: '3.10'
},
{
label: '3.11'
},
{
label: '3.12'
},
{
label: '3.13'
}
],
storage: 'setting'
},
{
label: 'UV index url',
description: 'Add private Pip registry',
key: 'pip_index_url',
fieldType: 'password',
placeholder: 'https://username:password@pypi.company.com/simple',
storage: 'setting',
ee_only: ''
},
{
label: 'UV extra index url',
description: 'Add private extra Pip registry',
key: 'pip_extra_index_url',
fieldType: 'password',
placeholder: 'https://username:password@pypi.company.com/simple',
storage: 'setting',
ee_only: ''
},
{
label: 'UV Python install mirror',
description:
'Mirror URL for downloading managed Python interpreters. Wires to <code>UV_PYTHON_INSTALL_MIRROR</code>. See <a href="https://docs.astral.sh/uv/configuration/environment/#uv_python_install_mirror">uv docs</a>.',
key: 'uv_python_install_mirror',
fieldType: 'text',
placeholder: 'https://mirror.example.com/python-build-standalone',
storage: 'setting'
},
{
label: 'UV index strategy',
description:
'Strategy for resolving packages from multiple indexes. See <a href="https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes">uv docs</a>',
key: 'uv_index_strategy',
fieldType: 'select',
placeholder: 'unsafe-best-match',
defaultValue: () => 'unsafe-best-match',
select_items: [
{
label: 'first-index',
tooltip: 'Only use the first index that contains the package'
},
{
label: 'unsafe-first-match',
tooltip: 'Search for packages across all indexes, preferring the first match'
},
{
label: 'unsafe-best-match (default)',
value: 'unsafe-best-match',
tooltip: 'Search for packages across all indexes, preferring the best match'
}
],
storage: 'setting',
ee_only: ''
},
{
label: 'NPM Registry Configuration (.npmrc)',
description:
'Full .npmrc file content for private npm registries. Used by Bun, Deno, and the npm proxy. Takes precedence over the legacy fields below.',
key: 'npmrc',
fieldType: 'codearea',
codeAreaLang: 'ini',
placeholder:
'registry=https://registry.mycompany.com/\n//registry.mycompany.com/:_authToken=YOUR_TOKEN\n\n@myorg:registry=https://registry.myorg.com/\n//registry.myorg.com/:_authToken=SCOPED_TOKEN',
storage: 'setting',
ee_only: ''
},
{
label: 'Npm config registry (legacy)',
description: 'Add private npm registry. Prefer using the .npmrc field above.',
key: 'npm_config_registry',
fieldType: 'password',
placeholder: 'https://registry.npmjs.org/:_authToken=npm_FOOBAR',
storage: 'setting',
ee_only: '',
hiddenIfEmpty: true
},
{
label: 'Bunfig install scopes (legacy)',
description:
'Add private scoped registries for Bun. Prefer using the .npmrc field above. See: https://bun.sh/docs/install/registries',
key: 'bunfig_install_scopes',
fieldType: 'password',
placeholder: '"@myorg3" = { token = "mytoken", url = "https://registry.myorg.com/" }',
storage: 'setting',
ee_only: '',
hiddenIfEmpty: true
},
{
label: 'Minimum release age (uv / Python)',
description:
'Refuse to install Python packages younger than this many seconds. Protects against supply-chain attacks via freshly published versions. Wires to <code>uv pip --exclude-newer</code>.',
key: 'uv_exclude_newer',
fieldType: 'seconds',
placeholder: '604800',
storage: 'setting'
},
{
label: 'Minimum release age (bun / npm)',
description:
'Refuse to install npm packages younger than this many seconds. Protects against supply-chain attacks via freshly published versions. Sets <code>BUN_INSTALL_MINIMUM_RELEASE_AGE</code>.',
key: 'bun_install_min_release_age',
fieldType: 'seconds',
placeholder: '604800',
storage: 'setting'
},
{
label: 'Nuget Config',
description:
'Write a nuget.config file to set custom package sources and credentials. Use <clear /> inside <packageSources> to remove default sources and only use your custom ones',
key: 'nuget_config',
fieldType: 'codearea',
codeAreaLang: 'xml',
storage: 'setting',
ee_only: ''
},
{
label: 'Maven/Ivy repositories',
description: 'Add private Maven/Ivy repositories',
key: 'maven_repos',
fieldType: 'password',
placeholder: 'https://user:password@artifacts.foo.com/maven',
storage: 'setting',
ee_only: ''
},
{
label: 'Maven settings.xml',
description:
'Write a Maven settings.xml file for custom repositories, mirrors, and credentials',
key: 'maven_settings_xml',
fieldType: 'codearea',
codeAreaLang: 'xml',
storage: 'setting',
ee_only: ''
},
{
label: 'Disable default Maven repository',
description: 'Do not use default Maven repository',
key: 'no_default_maven',
fieldType: 'boolean',
storage: 'setting',
ee_only: ''
},
{
label: 'Ruby Gems repositories',
description: 'Add private Ruby repositories with credentials. Should end with /',
key: 'ruby_repos',
fieldType: 'password',
placeholder: 'https://user:password@gems.foo.com/',
storage: 'setting',
ee_only: ''
},
{
label: 'Cargo registries',
description: 'Write a .cargo/config.toml to set custom Cargo registries and credentials',
key: 'cargo_registries',
fieldType: 'codearea',
codeAreaLang: 'toml',
storage: 'setting',
ee_only: ''
},
{
label: 'PowerShell Repository URL',
description: 'Add private PowerShell repository URL',
key: 'powershell_repo_url',
placeholder:
'https://pkgs.dev.azure.com/<org>/<project>/_packaging/<feed>/nuget/v3/index.json',
fieldType: 'text',
storage: 'setting',
ee_only: ''
},
{
label: 'PowerShell Repository PAT',
description:
'Add private PowerShell repository Personal Access Token (optional, for authenticated repositories)',
key: 'powershell_repo_pat',
fieldType: 'password',
storage: 'setting',
ee_only: ''
}
],
Alerts: [
{
label: 'Critical alert channels',
description:
'Channels to send critical alerts to. <a href="https://www.windmill.dev/docs/core_concepts/critical_alerts">Learn more</a>',
key: 'critical_error_channels',
fieldType: 'critical_error_channels',
storage: 'setting',
ee_only: 'Channels other than tracing are only available in the EE version',
actionButton: {
label: 'Test all channels',
onclick: async (values) => {
const { SettingService } = await import('$lib/gen')
const { sendUserToast } = await import('$lib/toast')
try {
await SettingService.testCriticalChannels({
requestBody: values.critical_error_channels
})
sendUserToast('Test message sent successfully to critical channels', false)
} catch (error: any) {
sendUserToast('Failed to send test message: ' + error.message, true)
}
},
variant: 'accent'
}
},
{
label: 'Mute critical alerts in UI',
description: 'Enable to mute critical alerts in the UI',
key: 'critical_alert_mute_ui',
fieldType: 'boolean',
storage: 'setting',
requiresReloadOnChange: true,
ee_only: 'Critical alerts in UI are only available in the EE version'
},
{
label: 'Alert on token expiry',
description:
'Send critical alerts when API tokens are about to expire (within 7 days) or have expired',
key: 'critical_alerts_on_token_expiry',
fieldType: 'boolean',
storage: 'setting',
ee_only: ''
},
{
label: 'Slack',
key: 'slack',
fieldType: 'slack_connect',
storage: 'setting',
ee_only: ''
},
{
label: 'Alert on DB oversize',
key: 'critical_alerts_on_db_oversize',
description: 'Alert if DB grows more than specified size',
fieldType: 'critical_alerts_on_db_oversize',
placeholder: '100',
storage: 'setting',
ee_only: ''
}
],
Webhooks: [
{
label: 'Instance Events Webhook',
description:
'URL to receive POST requests for instance events (user added, OAuth signup, user invited/added/joined workspace).',
key: 'instance_events_webhook',
fieldType: 'text',
placeholder: 'https://example.com/webhook',
storage: 'setting'
}
],
'OTEL/Prom': [
{
label: 'OpenTelemetry',
key: 'otel',
fieldType: 'otel',
storage: 'setting',
ee_only: '',
triggersRestart: true
},
{
label: 'HTTP Request Tracing',
description:
'Capture HTTP/HTTPS requests from job scripts as OpenTelemetry spans. Visible in job details and exported to your OTEL collector if configured. Toggling restarts workers.',
key: 'otel_tracing_proxy',
fieldType: 'otel_tracing_proxy',
storage: 'setting',
ee_only: 'HTTP Request Tracing is an EE feature',
triggersRestart: true,
defaultValue: () => ({ enabled: false, enabled_languages: [...OTEL_TRACING_PROXY_LANGUAGES] })
},
{
label: 'Prometheus',
description:
'Expose Prometheus metrics for workers and servers on port 8001 at /metrics. <a target="_blank" href="https://www.windmill.dev/docs/advanced/instance_settings#expose-metrics">Learn more</a>',
key: 'expose_metrics',
fieldType: 'boolean',
storage: 'setting',
ee_only: '',
triggersRestart: true
}
],
Indexer: [
{
label: '',
key: 'indexer_settings',
fieldType: 'indexer_rates',
storage: 'setting',
validate: validateIndexerSettings
}
],
Telemetry: [
{
label: 'Minimal telemetry',
key: 'disable_stats',
fieldType: 'boolean',
storage: 'setting'
}
],
'Secret Storage': [
{
label: 'Backend type',
description:
'By default, secrets are encrypted and stored in the database. Enterprise Edition supports HashiCorp Vault, Azure Key Vault, and AWS Secrets Manager as external secret backends.',
key: 'secret_backend',
fieldType: 'secret_backend',
storage: 'setting',
ee_only:
'HashiCorp Vault, Azure Key Vault, and AWS Secrets Manager integrations are Enterprise Edition features'
}
],
'GitHub App': [
{
label: 'GitHub App',
description:
'Configure a self-managed GitHub App to enable git sync without stats.windmill.dev.',
key: 'github_enterprise_app',
fieldType: 'github_enterprise_app',
storage: 'setting',
ee_only: '',
error:
'When self-managed mode is enabled, Base URL, App ID, App Slug, Client ID, and Private Key are required.',
isValid: (v: any) => {
if (!v?.self_managed) return true
return !!(v?.base_url && v?.app_id && v?.app_slug && v?.client_id && v?.private_key)
}
}
],
WebSocket: [
{
label: 'WebSocket connectivity',
description:
'Test connectivity to multiplayer, LSP, and debugger WebSocket services. Enable custom URL override for deployments where WebSocket traffic routes to a different host.',
key: 'ws_base_url',
fieldType: 'ws_connectivity',
storage: 'setting',
requiresReloadOnChange: true,
isValid: (value: string | undefined) =>
!value ||
(value.startsWith('ws') &&
value.includes('://') &&
!value.endsWith('/') &&
!value.endsWith(' '))
}
],
LSP: [
{
label: 'Ruff config (ruff.toml)',
description:
'Shared ruff.toml applied to the Python editor linter across the whole instance. The LSP container fetches this every minute and writes it next to edited files. See <a href="https://docs.astral.sh/ruff/configuration/">ruff docs</a>',
key: 'ruff_config',
fieldType: 'codearea',
codeAreaLang: 'toml',
placeholder: 'line-length = 100\n\n[lint]\nselect = ["E", "F", "I"]\nignore = ["E501"]',
storage: 'setting'
}
]
}
export const settingsKeys = Object.keys(settings)
// --- Sidebar navigation for instance settings ---
export const instanceSettingsNavigationGroups = [
{
title: 'Core',
items: [
{
id: 'users',
label: 'Users',
aiId: 'instance-settings-users',
aiDescription: 'Instance users settings'
},
{
id: 'general',
label: 'General',
aiId: 'instance-settings-general',
aiDescription: 'Instance general settings'
},
{
id: 'jobs',
label: 'Jobs',
aiId: 'instance-settings-jobs',
aiDescription: 'Instance jobs settings'
}
]
},
{
title: 'Authentication',
items: [
{
id: 'sso',
label: 'SSO',
aiId: 'instance-settings-sso',
aiDescription: 'Instance SSO settings'
},
{
id: 'oauth',
label: 'OAuth',
aiId: 'instance-settings-oauth',
aiDescription: 'Instance OAuth settings'
},
{
id: 'scim_saml',
label: 'SCIM/SAML',
aiId: 'instance-settings-scim-saml',
aiDescription: 'Instance SCIM/SAML settings',
isEE: true
}
]
},
{
title: 'Infrastructure',
items: [
{
id: 'smtp',
label: 'SMTP',
aiId: 'instance-settings-smtp',
aiDescription: 'Instance SMTP settings'
},
{
id: 'registries',
label: 'Registries',
aiId: 'instance-settings-registries',
aiDescription: 'Instance registries settings'
},
{
id: 'object_storage',
label: 'Object Storage',
aiId: 'instance-settings-object-storage',
aiDescription: 'Instance object storage settings',
isEE: true
}
]
},
{
title: 'Monitoring',
items: [
{
id: 'alerts',
label: 'Alerts',
aiId: 'instance-settings-alerts',
aiDescription: 'Instance alerts settings',
isEE: true
},
{
id: 'webhooks',
label: 'Webhooks',
aiId: 'instance-settings-webhooks',
aiDescription: 'Instance events webhook settings'
},
{
id: 'otel_prom',
label: 'OTEL/Prometheus',
aiId: 'instance-settings-otel-prom',
aiDescription: 'Instance OTEL/Prometheus settings',
isEE: true
},
{
id: 'indexer',
label: 'Indexer',
aiId: 'instance-settings-indexer',
aiDescription: 'Instance indexer settings',
isEE: true
},
{
id: 'db_health',
label: 'DB Health',
aiId: 'instance-settings-db-health',
aiDescription: 'Database health diagnostics and performance insights'
}
]
},
{
title: 'AI',
items: [
{
id: 'ai',
label: 'AI',
aiId: 'instance-settings-ai',
aiDescription: 'Instance AI settings (providers, models, prompts)'
}
]
},
{
title: 'Advanced',
items: [
{
id: 'github_enterprise_app',
label: 'GitHub App',
aiId: 'instance-settings-github-enterprise-app',
aiDescription: 'Self-managed GitHub App for git sync',
isEE: true
},
{
id: 'private_hub',
label: 'Private Hub',
aiId: 'instance-settings-private-hub',
aiDescription: 'Instance private hub settings',
isEE: true
},
{
id: 'telemetry',
label: 'Telemetry',
aiId: 'instance-settings-telemetry',
aiDescription: 'Instance telemetry settings'
},
{
id: 'secret_storage',
label: 'Secret Storage',
aiId: 'instance-settings-secret-storage',
aiDescription: 'Instance secret storage settings'
},
{
id: 'websocket',
label: 'WebSocket',
aiId: 'instance-settings-websocket',
aiDescription: 'WebSocket connectivity test and URL override'
},
{
id: 'lsp',
label: 'LSP',
aiId: 'instance-settings-lsp',
aiDescription: 'Language server protocol settings (ruff config, editor linting)'
}
]
}
]
export const tabToCategoryMap: Record<string, string> = {
general: 'Core',
ai: 'AI',
sso: 'Auth/OAuth/SAML',
oauth: 'Auth/OAuth/SAML',
scim_saml: 'Auth/OAuth/SAML',
smtp: 'SMTP',
registries: 'Registries',
alerts: 'Alerts',
webhooks: 'Webhooks',
otel_prom: 'OTEL/Prom',
indexer: 'Indexer',
telemetry: 'Telemetry',
secret_storage: 'Secret Storage',
object_storage: 'Object Storage',
jobs: 'Jobs',
private_hub: 'Private Hub',
github_enterprise_app: 'GitHub App',
websocket: 'WebSocket',
db_health: 'DB Health',
lsp: 'LSP'
}
export const tabToAuthSubTab: Record<string, 'sso' | 'oauth' | 'scim'> = {
sso: 'sso',
oauth: 'oauth',
scim_saml: 'scim'
}
// Navigation groups for the initial setup flow (no Users tab)
export const setupNavigationGroups = instanceSettingsNavigationGroups
.map((group) => ({
...group,
items: group.items.filter((item) => item.id !== 'users')
}))
.filter((group) => group.items.length > 0)
export const categoryToTabMap: Record<string, string> = {
Core: 'general',
AI: 'ai',
SMTP: 'smtp',
'Auth/OAuth/SAML': 'sso',
Registries: 'registries',
Alerts: 'alerts',
Webhooks: 'webhooks',
'OTEL/Prom': 'otel_prom',
Indexer: 'indexer',
Telemetry: 'telemetry',
'Secret Storage': 'secret_storage',
'Object Storage': 'object_storage',
Jobs: 'jobs',
'Private Hub': 'private_hub',
'GitHub App': 'github_enterprise_app',
WebSocket: 'websocket',
'DB Health': 'db_health',
LSP: 'lsp'
}
export interface SearchableSettingItem {
label: string
tabId: string
settingKey?: string
category: string
/** Full description text (HTML stripped), used for search matching only — not displayed */
description?: string
}
/**
* Extract the label portion from a uFuzzy marked/highlighted string.
* Only allows `<mark>` and `</mark>` tags through (sanitizes everything else).
*/
export function extractMarkedLabel(marked: string | undefined, labelLength: number): string {
if (!marked) return ''
let plainIdx = 0
let markedIdx = 0
while (plainIdx < labelLength && markedIdx < marked.length) {
if (marked[markedIdx] === '<') {
while (markedIdx < marked.length && marked[markedIdx] !== '>') markedIdx++
markedIdx++
} else {
plainIdx++
markedIdx++
}
}
// Include any closing </mark> right after
if (marked.startsWith('</mark>', markedIdx)) {
markedIdx += '</mark>'.length
}
// Sanitize: only allow <mark> and </mark> tags from uFuzzy highlight
return marked.slice(0, markedIdx).replace(/<(?!\/?mark>)[^>]*>/g, '')
}
export function buildSearchableSettingItems(
navigationGroups: typeof instanceSettingsNavigationGroups = instanceSettingsNavigationGroups
): SearchableSettingItem[] {
const items: SearchableSettingItem[] = []
// Add sidebar navigation items (tab-level)
for (const group of navigationGroups) {
for (const navItem of group.items) {
items.push({
label: navItem.label,
tabId: navItem.id,
category: group.title
})
}
}
// Add individual settings from each category
for (const [category, categorySettings] of Object.entries(settings)) {
const tabId = categoryToTabMap[category]
if (!tabId) continue
for (const setting of categorySettings) {
if (!setting.label) continue
items.push({
label: setting.label,
tabId,
settingKey: setting.key,
category,
description: setting.description?.replace(/<[^>]*>/g, '') ?? ''
})
}
}
// Add SCIM/SAML settings
for (const setting of scimSamlSetting) {
if (!setting.label) continue
items.push({
label: setting.label,
tabId: 'scim_saml',
settingKey: setting.key,
category: 'SCIM/SAML',
description: setting.description?.replace(/<[^>]*>/g, '') ?? ''
})
}
return items
}
/** Registry settings that support per-workspace overrides. Excludes instance_python_version, uv_index_strategy, and uv_python_install_mirror which are instance-wide only. */
export const WORKSPACE_REGISTRY_SETTINGS: Setting[] = settings['Registries'].filter(
(s) =>
s.key !== 'instance_python_version' &&
s.key !== 'uv_index_strategy' &&
s.key !== 'uv_python_install_mirror'
)