Files
windmill/flake.nix
pyranota 74b662d8de feat(benchmarks): k8s sim mode + util-group dashboard + reliability fixes
Stand up a minikube-backed simulation subsystem for benching Windmill under
realistic multi-node load, with a per-bench measurement pipeline and a
dashboard renderer that consolidates throughput, queue depth, per-node CPU,
PG latency/conns, OOM events, and per-node CPU-util-vs-oversaturation into
one SVG report.

Sim infrastructure (sim/):
- k8s_provisioner: minikube up + heterogeneous node sizing from topology JSON
- helm_deploy: helm install Windmill with smoke.yaml + local.yaml overlays
- image_cache: pre-load required images so bench bringup is offline-safe
- toxiproxy_k8s: per-node toxiproxy DaemonSet for cross-node latency injection
- cpu_sampler_k8s: privileged DS reading per-cgroup cpu.stat at 10Hz, dual-
  writes to stdout AND a host-mounted log file (/var/log/wm-sim-cpu-sampler/
  sampler.tsv) so heavy benches no longer lose early samples to kubelet log
  rotation
- pg_logging: ALTER SYSTEM + SIGHUP to enable verbose PG logging without restart
- pgbadger: post-bench PG log analysis HTML report
- readiness: pre-bench cluster health check (samplers stable ≥30s, workers
  ready, PG responsive, queue empty, **deploy.status rollout-complete**) —
  the rollout-complete check catches mid-rolling-update fires that previously
  starved m04's sampler under cgroup_mutex contention

Per-bench JSONL pollers, started/finalized alongside the bench loop:
- pod_timeline: 1Hz workers-per-node Ready counts (used for the workers panel)
- oom_poller: live OOM event capture (kernel + kubelet evictions + cgroup)
- pg_latency_poller: 4Hz psql \\timing on SELECT 1 vs kubectl-exec roundtrip
- pg_conn_poller: 1Hz pg_stat_activity by state (active/idle/idle_in_xact)
- node_load_poller: 2Hz /proc/loadavg + /proc/stat procs_running per node

Dashboard renderer (sim/render_report.ts + graph.ts):
- Util group: one panel per node with translucent orange oversaturation area
  BEHIND solid blue CPU-util area, 100% reference line, phase-boundary verticals.
  cols:2 grid wraps after 2 panels per row.
- PG node tinted with [PG] flag in legend across the dashboard.
- Phase-boundary verticals + push-window shaded zones layered consistently.
- All x-axes switched from wall-clock HH:MM to relative seconds-from-bench-
  start. Shared origin sourced from meta.json's bench_start_ms so 0s on every
  panel = the same wall-clock moment (previously each chart picked its own
  earliest sample as origin, causing drift between panels).

Oversaturation metric, with explicit fallback:
- Primary: (procs_running - ncpu) / ncpu × 100 — true CPU run-queue pressure.
- Fallback to load1 when procs_running is missing (older reports).
- load1 overcounted previously because it includes uninterruptible D-state
  procs (PG backends in disk I/O, cgroup_mutex waits), inflating "saturation"
  by 5-10x under load.
- Pure helper extracted to sim/util_metrics.ts; 8 unit tests cover the
  procs_running > load1 preference, the clamp-at-zero, invalid-ncpu cases.

Sampler reliability:
- HostPath log file in addition to stdout so the bench's scp-based collector
  bypasses kubelet log rotation entirely.
- main.ts truncates the host log file on every node before pushers start
  (parallel ssh, best-effort) so it doesn't grow unbounded across runs.
- Collector falls back to kubectl-logs when scp fails for any node.

Workloads (workloads/):
- io_4phase: four-phase IO step (idle → 2.5s → 500ms → 150ms jobs)
- io_150ms_flood / io_300ms_flood / io_1s_flood / io_2s_flood: single-phase
  flood configs to isolate the worker-host CFS context-switch storm vs PG
  contention regime
- burst, ops_day, cpu_*, etc. for other scenarios

Tests:
- sim/util_metrics_test.ts — 8 cases for computeOversatPct
- sim/util_panel_snapshot_test.ts — 5 assertions guarding util-panel SVG
  invariants (orange behind blue, 100% ref line, relative-time ticks NOT
  wall-clock, phase-boundary verticals, shared-origin override)

Helm values:
- sim/values/smoke.yaml — bench-tuned: workers w/ no CPU limit & low mem
  request, PG w/ 3-core request + wm-critical priorityClass + oomImmune +
  maxConnections, app w/ wm-critical + oomImmune + no resource limits.
- sim/values/local.example.yaml — template for the gitignored local.yaml
  that carries the EE license key.
- Depends on the wm-critical PriorityClass + oomImmune + maxConnections
  knobs landing in windmill-helm-charts (separate PR).

graph.ts additions:
- areaFills param: ordered list of per-kind translucent area fills drawn
  before lines, used by the util panel for orange-behind-blue layering
- lineColorOverrides: pin per-kind line colors so oversaturation reliably
  renders orange regardless of d3 ordinal-color insertion order
- highlightKindToken: substring-match flag for the PG-node tint in Node CPU
- xRelativeOriginMs: shared bench-start origin for the relative-time x-axis
- DataPointMulti is now exported for downstream tests

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-08 11:43:47 +02:00

647 lines
25 KiB
Nix

{
inputs = {
nixpkgs.url = "nixpkgs/nixos-unstable";
flake-utils.url = "github:numtide/flake-utils";
rust-overlay.url = "github:oxalica/rust-overlay";
# Pin openapi-generator-cli to 7.10.0
nixpkgs-oapi-gen.url = "nixpkgs/2d068ae5c6516b2d04562de50a58c682540de9bf";
# Fresh nixos-unstable for the k8s sim tools (minikube/libvirt/kvm2-driver).
# The main `nixpkgs` lock can lag behind the host system, but the kvm2
# driver MUST link against system-compatible glibc/libvirt — otherwise it
# crashes loading the system libvirt's transitive libs at runtime. Verified
# at this rev: minikube 1.38.1, libvirt 12.2.0 (matches NixOS 26.05 hosts).
nixpkgs-sim.url = "nixpkgs/64c08a7ca051951c8eae34e3e3cb1e202fe36786";
};
outputs = { self, nixpkgs, flake-utils, rust-overlay, nixpkgs-oapi-gen, nixpkgs-sim }:
flake-utils.lib.eachDefaultSystem (system:
let
pkgs = import nixpkgs {
inherit system;
config.allowUnfree = true;
overlays = [ (import rust-overlay) ];
};
# Fresh nixos-unstable just for the k8s sim tools — keeps the kvm2
# driver / minikube / helm aligned with the host system's libvirtd.
# See `inputs.nixpkgs-sim` above for the why.
pkgsSim = import nixpkgs-sim {
inherit system;
config.allowUnfree = true;
};
lib = pkgs.lib;
stdenv = pkgs.stdenv;
# ---------------------------------------------------------------
# Rust toolchain
# ---------------------------------------------------------------
rustStable = pkgs.rust-bin.stable.latest.default.override {
extensions = [ "rust-src" "rust-analyzer" "rustfmt" ];
};
patchedClang = pkgs.llvmPackages_18.clang.overrideAttrs (oldAttrs: {
postFixup = ''
# Copy the original postFixup logic but skip add-hardening.sh
${oldAttrs.postFixup or ""}
# Remove the line that substitutes add-hardening.sh
sed -i 's/.*source.*add-hardening\.sh.*//' $out/bin/clang
'';
});
# ---------------------------------------------------------------
# Native C/C++ dependencies (required to compile the backend)
# ---------------------------------------------------------------
nativeBuildDeps = with pkgs; [
# Crypto / TLS
openssl
openssl.dev
# XML / SAML (enterprise_saml feature)
libxml2.dev
xmlsec.dev
libxslt.dev
# FFI / codegen
libclang.dev
libffi # deno_ffi on macOS
# Networking / compression
curl.dev
zlib.dev
# Auth (kafka-gssapi, mssql-kerberos)
cyrus_sasl
krb5
# Misc
libtool
postgresql
# Build tooling
pkg-config
llvmPackages_18.clang # linker — pinned to 18 to avoid SIGSEGV with mold + newer clang
mold
cmake # required by rdkafka cmake-build
];
# ---------------------------------------------------------------
# Prebuilt V8 binary (must match version in Cargo.toml)
# ---------------------------------------------------------------
rustyV8Archive = let
version = "137.1.0";
target = stdenv.hostPlatform.rust.rustcTarget;
sha256 = {
x86_64-linux =
"sha256-Tiscfy2bzYGR3s0T+SC1IB3xWvTVpVcSEdjq3MCRoRw=";
aarch64-linux = lib.fakeHash;
x86_64-darwin = lib.fakeHash;
aarch64-darwin = lib.fakeHash;
}.${system};
in pkgs.fetchurl {
name = "librusty_v8-${version}";
url =
"https://github.com/denoland/rusty_v8/releases/download/v${version}/librusty_v8_release_${target}.a.gz";
inherit sha256;
};
# ---------------------------------------------------------------
# pkg-config search path for native libraries
# ---------------------------------------------------------------
pkgConfigPath = lib.makeSearchPath "lib/pkgconfig" (with pkgs; [
openssl.dev
libxml2.dev
xmlsec.dev
libxslt.dev
cyrus_sasl.dev
krb5.dev
]);
# ---------------------------------------------------------------
# RPATH — embed Nix store library paths into compiled binaries
# ---------------------------------------------------------------
rpathLibs = lib.makeLibraryPath (with pkgs; [
openssl
libffi
cyrus_sasl
krb5
libxml2
xmlsec
libxslt
stdenv.cc.cc.lib
]);
# ---------------------------------------------------------------
# Bindgen configuration
# Bindgen uses libclang directly (not $CC), so we must explicitly
# provide all Nix header search paths.
# See: https://web.archive.org/web/20220523141208/https://hoverbear.org/blog/rust-bindgen-in-nix/
# ---------------------------------------------------------------
bindgenClangArgs = builtins.concatStringsSep " " ([
"-nostdinc"
(builtins.readFile "${stdenv.cc}/nix-support/libc-crt1-cflags")
(builtins.readFile "${stdenv.cc}/nix-support/libc-cflags")
(builtins.readFile "${stdenv.cc}/nix-support/cc-cflags")
(builtins.readFile "${stdenv.cc}/nix-support/libcxx-cxxflags")
"-idirafter ${pkgs.libiconv}/include"
] ++ lib.optionals stdenv.cc.isClang [
"-idirafter ${stdenv.cc.cc}/lib/clang/${
lib.getVersion stdenv.cc.cc
}/include"
] ++ lib.optionals stdenv.cc.isGNU [
"-isystem ${stdenv.cc.cc}/include/c++/${lib.getVersion stdenv.cc.cc}"
"-isystem ${stdenv.cc.cc}/include/c++/${
lib.getVersion stdenv.cc.cc
}/${stdenv.hostPlatform.config}"
"-idirafter ${stdenv.cc.cc}/lib/gcc/${stdenv.hostPlatform.config}/${
lib.getVersion stdenv.cc.cc
}/include"
]);
# ---------------------------------------------------------------
# Build environment variables (shared by all shells that compile Rust)
# ---------------------------------------------------------------
buildEnvVars = {
PKG_CONFIG_PATH = pkgConfigPath;
RUSTY_V8_ARCHIVE = rustyV8Archive;
LIBCLANG_PATH = "${pkgs.llvmPackages.libclang.lib}/lib";
BINDGEN_EXTRA_CLANG_ARGS = bindgenClangArgs;
# Force clang 18 as cargo linker (stdenv may bring a newer clang that causes SIGSEGV with mold)
CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER =
"${pkgs.llvmPackages_18.clang}/bin/clang";
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER =
"${pkgs.llvmPackages_18.clang}/bin/clang";
# Embed rpath so binaries find Nix store .so files at runtime
CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUSTFLAGS =
"-C link-arg=-fuse-ld=mold -C link-arg=-Wl,-rpath,${rpathLibs}";
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_RUSTFLAGS =
"-C link-arg=-fuse-ld=mold -C link-arg=-Wl,-rpath,${rpathLibs}";
CARGO_HOST_RUSTFLAGS = "-C link-arg=-Wl,-rpath,${rpathLibs}";
# https://github.com/NixOS/nixpkgs/issues/370494 — jemalloc build fix
CFLAGS = "-Wno-error=int-conversion";
LD_LIBRARY_PATH = lib.makeLibraryPath [ pkgs.zlib stdenv.cc.cc.lib ];
};
# ---------------------------------------------------------------
# OpenAPI generator (pinned)
# ---------------------------------------------------------------
openapi-generator-cli =
(import nixpkgs-oapi-gen { inherit system; }).openapi-generator-cli;
# ---------------------------------------------------------------
# Common worker runtimes (languages the worker executes)
# ---------------------------------------------------------------
commonRuntimes = with pkgs; [
deno
python3
python3Packages.pip
uv
go
bun
nushell
typescript
flock
];
# ---------------------------------------------------------------
# Runtime PATH env vars — tells the worker where to find interpreters
# ---------------------------------------------------------------
commonRuntimeVars = {
DENO_PATH = "${pkgs.deno}/bin/deno";
GO_PATH = "${pkgs.go}/bin/go";
BUN_PATH = "${pkgs.bun}/bin/bun";
NODE_PATH = "${pkgs.nodejs}/bin/node";
NODE_BIN_PATH = "${pkgs.nodejs}/bin/node";
UV_PATH = "${pkgs.uv}/bin/uv";
NU_PATH = "${pkgs.nushell}/bin/nu";
FLOCK_PATH = "${pkgs.flock}/bin/flock";
CARGO_PATH = "${rustStable}/bin/cargo";
BASH_PATH = "bash";
GIT_PATH = "${pkgs.git}/bin/git";
};
# ---------------------------------------------------------------
# Extra language runtimes (full shell only)
# ---------------------------------------------------------------
coursier = pkgs.fetchFromGitHub {
owner = "coursier";
repo = "launchers";
rev = "79d927f7586c09ca6d8cd01862adb0d9f9d88dff";
hash = "sha256-8E0WtDFc7RcqmftDigMyy1xXUkjgL4X4kpf7h1GdE48=";
};
rWithPackages = pkgs.rWrapper.override {
packages = with pkgs.rPackages; [ renv ];
};
extraRuntimes = with pkgs; [
dotnet-sdk_9
php
php84Packages.composer
ruby_3_4
jdk21
ansible
oracle-instantclient
];
extraRuntimeVars = {
JAVA_PATH = "${pkgs.jdk21}/bin/java";
JAVAC_PATH = "${pkgs.jdk21}/bin/javac";
COURSIER_PATH = "${coursier}/coursier";
DOTNET_PATH = "${pkgs.dotnet-sdk_9}/bin/dotnet";
DOTNET_ROOT = "${pkgs.dotnet-sdk_9}/share/dotnet";
PHP_PATH = "${pkgs.php}/bin/php";
COMPOSER_PATH = "${pkgs.php84Packages.composer}/bin/composer";
RUBY_PATH = "${pkgs.ruby_3_4}/bin/ruby";
RUBY_BUNDLE_PATH = "${pkgs.ruby_3_4}/bin/bundle";
RUBY_GEM_PATH = "${pkgs.ruby_3_4}/bin/gem";
ORACLE_LIB_DIR = "${pkgs.oracle-instantclient.lib}/lib";
ANSIBLE_PLAYBOOK_PATH = "${pkgs.ansible}/bin/ansible-playbook";
ANSIBLE_GALAXY_PATH = "${pkgs.ansible}/bin/ansible-galaxy";
CARGO_SWEEP_PATH = "${pkgs.cargo-sweep}/bin/cargo-sweep";
RSCRIPT_PATH = "${rWithPackages}/bin/Rscript";
};
# ---------------------------------------------------------------
# General dev environment variables
# ---------------------------------------------------------------
devEnvVars = {
NODE_ENV = "development";
NODE_OPTIONS = "--max-old-space-size=16384";
};
# Connection-specific defaults — set via shellHook so they respect
# pre-existing values (e.g. from webmux runtime.env / .env.local).
# Nix attrs are injected unconditionally and would override per-worktree
# values set by webmux before the interactive shell starts.
devShellHook = ''
export DATABASE_URL="''${DATABASE_URL:-postgres://postgres:changeme@127.0.0.1:5432/windmill?sslmode=disable}"
export REMOTE="''${REMOTE:-http://127.0.0.1:8000}"
export REMOTE_LSP="''${REMOTE_LSP:-http://127.0.0.1:3001}"
'';
# ---------------------------------------------------------------
# Helper scripts — base set (default + full)
# ---------------------------------------------------------------
# minikube kvm2 driver for the k8s benchmark sim (not in nixpkgs).
# Built against pkgsSim so glibc/libvirt match the host system.
kvm2Driver = import ./benchmarks/sim/nix/kvm2-driver.nix { pkgs = pkgsSim; };
# `wm_sim` — k8s sim entry, factored out so wasm/cli devShells can pull
# just it without inheriting the rest of helperScriptsBase.
wmSimWrapper = pkgs.writeShellScriptBin "wm_sim" ''
export SIM_KVM2_DRIVER_DIR="${kvm2Driver}/bin"
# minikube + helm from pkgsSim so they share the kvm2 driver's libc.
export SIM_MINIKUBE_BIN="${pkgsSim.minikube}/bin/minikube"
export SIM_HELM_BIN="${pkgsSim.kubernetes-helm}/bin/helm"
# System libvirt — matches the system virsh that minikube's kvm2
# preflight invokes. Falls back to pkgsSim's libvirt (also 12.2 in
# nixos-unstable) if the system path can't be resolved.
sysvirsh="$(command -v virsh 2>/dev/null || echo /run/current-system/sw/bin/virsh)"
syslib="$(ldd "$sysvirsh" 2>/dev/null | awk '/libvirt\.so\.0/{print $3}' | head -1)"
if [ -n "$syslib" ]; then
export SIM_LIBVIRT_LIB_DIR="$(dirname "$syslib")"
else
export SIM_LIBVIRT_LIB_DIR="${pkgsSim.libvirt}/lib"
fi
# virsh path for the provisioner's pre-start sweep of leaked domains.
export SIM_VIRSH_BIN="$sysvirsh"
root="$(git rev-parse --show-toplevel 2>/dev/null || echo .)"
exec ${pkgs.deno}/bin/deno run --no-check -A "$root/benchmarks/sim/sim.ts" "$@"
'';
helperScriptsBase = [
(pkgs.writeScriptBin "wm" ''
cd ./frontend
npm install
npm run ${
if stdenv.isDarwin then
"generate-backend-client-mac"
else
"generate-backend-client"
}
npm run dev "$@"
'')
(pkgs.writeScriptBin "wm-build" ''
cd ./frontend
npm install
npm run ${
if stdenv.isDarwin then
"generate-backend-client-mac"
else
"generate-backend-client"
}
npm run build "$@"
'')
(pkgs.writeScriptBin "wm-migrate" ''
cd ./backend
sqlx migrate run
'')
(pkgs.writeScriptBin "wm-reset" ''
sqlx database drop -f
sqlx database create
wm-migrate
'')
(pkgs.writeScriptBin "wm-minio" ''
set -e
cd ./backend
mkdir -p .minio-data/wmill
${pkgs.minio}/bin/minio server ./.minio-data --console-address ":9001"
'')
(pkgs.writeScriptBin "wm-minio-keys" ''
set -e
cd ./backend
${pkgs.minio-client}/bin/mc alias set 'wmill-minio-dev' 'http://localhost:9000' 'minioadmin' 'minioadmin'
if [[ -f .minio-data/secrets.txt ]] && [[ -s .minio-data/secrets.txt ]]; then
echo "Access keys already exist:"
cat .minio-data/secrets.txt
echo ""
echo "Keys loaded from: ./backend/.minio-data/secrets.txt"
else
echo "Creating new access keys..."
mkdir -p .minio-data
${pkgs.minio-client}/bin/mc admin accesskey create 'wmill-minio-dev' | tee .minio-data/secrets.txt
echo ""
echo 'New keys saved to: ./backend/.minio-data/secrets.txt'
fi
echo "bucket: wmill"
echo "endpoint: http://localhost:9000"
'')
# k8s sim entry — provisioning only, no bench. See `wmSimWrapper`.
wmSimWrapper
# Bench entry — same env shape as wm_sim so `--topology` (sim-driven
# provisioning) works. For plain `--host` benches the env is harmless.
(pkgs.writeShellScriptBin "wm-bench" ''
export SIM_KVM2_DRIVER_DIR="${kvm2Driver}/bin"
export SIM_MINIKUBE_BIN="${pkgsSim.minikube}/bin/minikube"
export SIM_HELM_BIN="${pkgsSim.kubernetes-helm}/bin/helm"
sysvirsh="$(command -v virsh 2>/dev/null || echo /run/current-system/sw/bin/virsh)"
syslib="$(ldd "$sysvirsh" 2>/dev/null | awk '/libvirt\.so\.0/{print $3}' | head -1)"
if [ -n "$syslib" ]; then
export SIM_LIBVIRT_LIB_DIR="$(dirname "$syslib")"
else
export SIM_LIBVIRT_LIB_DIR="${pkgsSim.libvirt}/lib"
fi
export SIM_VIRSH_BIN="$sysvirsh"
root="$(git rev-parse --show-toplevel 2>/dev/null || echo .)"
exec ${pkgs.deno}/bin/deno run --no-check -A "$root/benchmarks/main.ts" -e admin@windmill.dev -p changeme "$@"
'')
];
# ---------------------------------------------------------------
# Helper scripts — extra (full shell only)
# ---------------------------------------------------------------
helperScriptsFull = [
(pkgs.writeScriptBin "wm-caddy" ''
cd ./frontend
xcaddy build "$@" \
--with github.com/mholt/caddy-l4@145ec36251a44286f05a10d231d8bfb3a8192e09 \
--with github.com/RussellLuo/caddy-ext/layer4@ab1e18cfe426012af351a68463937ae2e934a2a1
'')
(pkgs.writeScriptBin "wm-setup" ''
sqlx database create
wm-build
wm-caddy
wm-migrate
'')
];
# ---------------------------------------------------------------
# Shared inputs and settings for default + full shells
# ---------------------------------------------------------------
coreBuildInputs = nativeBuildDeps ++ commonRuntimes
++ [ rustStable openapi-generator-cli ] ++ (with pkgs; [
nodejs
git
sqlx-cli
cargo-watch
jq
gnused
# CLI tools (for AI agents and dev workflow)
gh
asciinema
mermaid-cli
# Network shaping / topology benches
toxiproxy
pgbadger
# Local k8s for sim topologies. Single-node uses the built-in qemu2
# driver (no libvirt, no sudo). Multi-node (--nodes=N) needs the
# kvm2 driver — the docker-machine-driver-kvm2 binary isn't in
# nixpkgs and minikube's auto-download won't run on NixOS, so kvm2
# requires a custom overlay + host-level libvirtd. qemu is shared by
# both. VM-based nodes sidestep the rootless-cgroup wall (k3d/kind).
minikube
kubectl
kubernetes-helm
qemu
]);
# Playwright: use Nix-provided browsers (version-matched to playwright-driver)
# Mermaid/Puppeteer: point at Nix chromium (Puppeteer respects this env var)
browserVars = {
PLAYWRIGHT_BROWSERS_PATH = "${pkgs.playwright-driver.browsers}";
PUPPETEER_EXECUTABLE_PATH = "${pkgs.chromium}/bin/chromium";
PUPPETEER_SKIP_DOWNLOAD = "true";
};
# Wrapper for the Nix-provided playwright CLI (version-matched to its browsers)
playwrightWrapper = pkgs.writeShellScriptBin "playwright" ''
export PLAYWRIGHT_BROWSERS_PATH="${pkgs.playwright-driver.browsers}"
exec ${pkgs.nodejs}/bin/node ${pkgs.playwright-driver}/cli.js "$@"
'';
# ---------------------------------------------------------------
# sandbox-env script — outputs env vars for browser tooling
# Usage: eval "$(sandbox-env)"
# ---------------------------------------------------------------
sandboxEnvScript = pkgs.writeShellScriptBin "sandbox-env" ''
echo "export PLAYWRIGHT_BROWSERS_PATH=${pkgs.playwright-driver.browsers}"
echo "export PUPPETEER_EXECUTABLE_PATH=${pkgs.chromium}/bin/chromium"
echo "export PUPPETEER_SKIP_DOWNLOAD=true"
'';
# ---------------------------------------------------------------
# pkg-config wrapper — bakes in the Nix pkg-config search path
# so sandbox profiles (buildEnv) work without setting env vars.
# ---------------------------------------------------------------
pkgConfigWrapper = pkgs.writeShellScriptBin "pkg-config" ''
export PKG_CONFIG_PATH="${pkgConfigPath}:$PKG_CONFIG_PATH"
exec ${pkgs.pkg-config}/bin/pkg-config "$@"
'';
# ---------------------------------------------------------------
# Installable sandbox profiles (nix profile install .#sandbox)
# ---------------------------------------------------------------
sandboxEnv = pkgs.buildEnv {
name = "windmill-sandbox";
paths = coreBuildInputs ++ helperScriptsBase ++ [
playwrightWrapper
sandboxEnvScript
pkgConfigWrapper
pkgs.chromium
];
};
sandboxFullEnv = pkgs.buildEnv {
name = "windmill-sandbox-full";
paths = coreBuildInputs ++ extraRuntimes ++ helperScriptsBase
++ helperScriptsFull ++ [
playwrightWrapper
sandboxEnvScript
pkgConfigWrapper
pkgs.chromium
pkgs.cargo-sweep
pkgs.xcaddy
pkgs.nsjail
];
};
in {
# =============================================================
# Installable profiles — for Docker / nix profile install
# Usage: nix profile install .#sandbox
# =============================================================
packages.sandbox = sandboxEnv;
packages.sandbox-full = sandboxFullEnv;
packages.default = sandboxEnv;
# =============================================================
# default — daily driver for backend + frontend development
# Usage: nix develop
# =============================================================
devShells.default = pkgs.mkShell (buildEnvVars // commonRuntimeVars // devEnvVars // browserVars // {
shellHook = devShellHook;
buildInputs = coreBuildInputs;
packages = helperScriptsBase ++ [ playwrightWrapper ];
});
# =============================================================
# full — all language runtimes, k8s tooling, specialized scripts
# Usage: nix develop .#full
# =============================================================
devShells.full = pkgs.mkShell (buildEnvVars // commonRuntimeVars // extraRuntimeVars // devEnvVars // browserVars // {
shellHook = devShellHook;
buildInputs = coreBuildInputs ++ extraRuntimes ++ (with pkgs; [
# Python extras
poetry
pyright
openapi-python-client
# LSP / editor
svelte-language-server
taplo
# Extra dev tools
cargo-sweep
# Kubernetes
minikube
kubectl
kubernetes-helm
conntrack-tools
cri-tools
# Extra
xcaddy
nsjail
]);
packages = helperScriptsBase ++ helperScriptsFull
++ [ playwrightWrapper ];
});
# =============================================================
# wasm — WASM target compilation (nightly Rust)
# Usage: nix develop .#wasm
# =============================================================
devShells.wasm = pkgs.mkShell (buildEnvVars // {
hardeningDisable = [ "all" ];
# Explicitly set paths for headers and linker
# DO NOT REMOVE - if absent, breaks wasm builds on NixOS.
shellHook = ''
export CC=${patchedClang}/bin/clang
'';
buildInputs = nativeBuildDeps ++ (with pkgs; [
(rust-bin.nightly.latest.default.override {
extensions = [ "rust-src" "rust-analyzer" ];
targets =
[ "wasm32-unknown-unknown" "wasm32-unknown-emscripten" ];
})
wasm-pack
deno
emscripten
nushell
nodejs
glibc_multi
]);
packages = [ wmSimWrapper ];
});
# =============================================================
# cli — lightweight Bun-based CLI development
# Usage: nix develop .#cli
# =============================================================
devShells.cli = pkgs.mkShell {
shellHook = ''
if command -v git >/dev/null 2>&1 && git rev-parse --is-inside-work-tree >/dev/null 2>&1; then
export FLAKE_ROOT="$(git rev-parse --show-toplevel)"
else
export FLAKE_ROOT="$PWD"
fi
'';
buildInputs = with pkgs; [ bun nodejs git ];
packages = [
wmSimWrapper
(pkgs.writeScriptBin "wm-cli" ''
bun run $FLAKE_ROOT/cli/src/main.ts "$@"
'')
(pkgs.writeScriptBin "wm-cli-deps" ''
pushd $FLAKE_ROOT/cli/
${if stdenv.isDarwin then
"./gen_wm_client_mac.sh && ./windmill-utils-internal/gen_wm_client_mac.sh"
else
"./gen_wm_client.sh && ./windmill-utils-internal/gen_wm_client.sh"}
popd
'')
];
};
});
}