mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-08-18 16:02:10 +00:00
74b662d8de
Stand up a minikube-backed simulation subsystem for benching Windmill under realistic multi-node load, with a per-bench measurement pipeline and a dashboard renderer that consolidates throughput, queue depth, per-node CPU, PG latency/conns, OOM events, and per-node CPU-util-vs-oversaturation into one SVG report. Sim infrastructure (sim/): - k8s_provisioner: minikube up + heterogeneous node sizing from topology JSON - helm_deploy: helm install Windmill with smoke.yaml + local.yaml overlays - image_cache: pre-load required images so bench bringup is offline-safe - toxiproxy_k8s: per-node toxiproxy DaemonSet for cross-node latency injection - cpu_sampler_k8s: privileged DS reading per-cgroup cpu.stat at 10Hz, dual- writes to stdout AND a host-mounted log file (/var/log/wm-sim-cpu-sampler/ sampler.tsv) so heavy benches no longer lose early samples to kubelet log rotation - pg_logging: ALTER SYSTEM + SIGHUP to enable verbose PG logging without restart - pgbadger: post-bench PG log analysis HTML report - readiness: pre-bench cluster health check (samplers stable ≥30s, workers ready, PG responsive, queue empty, **deploy.status rollout-complete**) — the rollout-complete check catches mid-rolling-update fires that previously starved m04's sampler under cgroup_mutex contention Per-bench JSONL pollers, started/finalized alongside the bench loop: - pod_timeline: 1Hz workers-per-node Ready counts (used for the workers panel) - oom_poller: live OOM event capture (kernel + kubelet evictions + cgroup) - pg_latency_poller: 4Hz psql \\timing on SELECT 1 vs kubectl-exec roundtrip - pg_conn_poller: 1Hz pg_stat_activity by state (active/idle/idle_in_xact) - node_load_poller: 2Hz /proc/loadavg + /proc/stat procs_running per node Dashboard renderer (sim/render_report.ts + graph.ts): - Util group: one panel per node with translucent orange oversaturation area BEHIND solid blue CPU-util area, 100% reference line, phase-boundary verticals. cols:2 grid wraps after 2 panels per row. - PG node tinted with [PG] flag in legend across the dashboard. - Phase-boundary verticals + push-window shaded zones layered consistently. - All x-axes switched from wall-clock HH:MM to relative seconds-from-bench- start. Shared origin sourced from meta.json's bench_start_ms so 0s on every panel = the same wall-clock moment (previously each chart picked its own earliest sample as origin, causing drift between panels). Oversaturation metric, with explicit fallback: - Primary: (procs_running - ncpu) / ncpu × 100 — true CPU run-queue pressure. - Fallback to load1 when procs_running is missing (older reports). - load1 overcounted previously because it includes uninterruptible D-state procs (PG backends in disk I/O, cgroup_mutex waits), inflating "saturation" by 5-10x under load. - Pure helper extracted to sim/util_metrics.ts; 8 unit tests cover the procs_running > load1 preference, the clamp-at-zero, invalid-ncpu cases. Sampler reliability: - HostPath log file in addition to stdout so the bench's scp-based collector bypasses kubelet log rotation entirely. - main.ts truncates the host log file on every node before pushers start (parallel ssh, best-effort) so it doesn't grow unbounded across runs. - Collector falls back to kubectl-logs when scp fails for any node. Workloads (workloads/): - io_4phase: four-phase IO step (idle → 2.5s → 500ms → 150ms jobs) - io_150ms_flood / io_300ms_flood / io_1s_flood / io_2s_flood: single-phase flood configs to isolate the worker-host CFS context-switch storm vs PG contention regime - burst, ops_day, cpu_*, etc. for other scenarios Tests: - sim/util_metrics_test.ts — 8 cases for computeOversatPct - sim/util_panel_snapshot_test.ts — 5 assertions guarding util-panel SVG invariants (orange behind blue, 100% ref line, relative-time ticks NOT wall-clock, phase-boundary verticals, shared-origin override) Helm values: - sim/values/smoke.yaml — bench-tuned: workers w/ no CPU limit & low mem request, PG w/ 3-core request + wm-critical priorityClass + oomImmune + maxConnections, app w/ wm-critical + oomImmune + no resource limits. - sim/values/local.example.yaml — template for the gitignored local.yaml that carries the EE license key. - Depends on the wm-critical PriorityClass + oomImmune + maxConnections knobs landing in windmill-helm-charts (separate PR). graph.ts additions: - areaFills param: ordered list of per-kind translucent area fills drawn before lines, used by the util panel for orange-behind-blue layering - lineColorOverrides: pin per-kind line colors so oversaturation reliably renders orange regardless of d3 ordinal-color insertion order - highlightKindToken: substring-match flag for the PG-node tint in Node CPU - xRelativeOriginMs: shared bench-start origin for the relative-time x-axis - DataPointMulti is now exported for downstream tests Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
647 lines
25 KiB
Nix
647 lines
25 KiB
Nix
{
|
|
inputs = {
|
|
nixpkgs.url = "nixpkgs/nixos-unstable";
|
|
flake-utils.url = "github:numtide/flake-utils";
|
|
rust-overlay.url = "github:oxalica/rust-overlay";
|
|
# Pin openapi-generator-cli to 7.10.0
|
|
nixpkgs-oapi-gen.url = "nixpkgs/2d068ae5c6516b2d04562de50a58c682540de9bf";
|
|
# Fresh nixos-unstable for the k8s sim tools (minikube/libvirt/kvm2-driver).
|
|
# The main `nixpkgs` lock can lag behind the host system, but the kvm2
|
|
# driver MUST link against system-compatible glibc/libvirt — otherwise it
|
|
# crashes loading the system libvirt's transitive libs at runtime. Verified
|
|
# at this rev: minikube 1.38.1, libvirt 12.2.0 (matches NixOS 26.05 hosts).
|
|
nixpkgs-sim.url = "nixpkgs/64c08a7ca051951c8eae34e3e3cb1e202fe36786";
|
|
};
|
|
|
|
outputs = { self, nixpkgs, flake-utils, rust-overlay, nixpkgs-oapi-gen, nixpkgs-sim }:
|
|
flake-utils.lib.eachDefaultSystem (system:
|
|
let
|
|
pkgs = import nixpkgs {
|
|
inherit system;
|
|
config.allowUnfree = true;
|
|
overlays = [ (import rust-overlay) ];
|
|
};
|
|
|
|
# Fresh nixos-unstable just for the k8s sim tools — keeps the kvm2
|
|
# driver / minikube / helm aligned with the host system's libvirtd.
|
|
# See `inputs.nixpkgs-sim` above for the why.
|
|
pkgsSim = import nixpkgs-sim {
|
|
inherit system;
|
|
config.allowUnfree = true;
|
|
};
|
|
|
|
lib = pkgs.lib;
|
|
stdenv = pkgs.stdenv;
|
|
|
|
# ---------------------------------------------------------------
|
|
# Rust toolchain
|
|
# ---------------------------------------------------------------
|
|
|
|
rustStable = pkgs.rust-bin.stable.latest.default.override {
|
|
extensions = [ "rust-src" "rust-analyzer" "rustfmt" ];
|
|
};
|
|
|
|
patchedClang = pkgs.llvmPackages_18.clang.overrideAttrs (oldAttrs: {
|
|
postFixup = ''
|
|
# Copy the original postFixup logic but skip add-hardening.sh
|
|
${oldAttrs.postFixup or ""}
|
|
|
|
# Remove the line that substitutes add-hardening.sh
|
|
sed -i 's/.*source.*add-hardening\.sh.*//' $out/bin/clang
|
|
'';
|
|
});
|
|
|
|
# ---------------------------------------------------------------
|
|
# Native C/C++ dependencies (required to compile the backend)
|
|
# ---------------------------------------------------------------
|
|
|
|
nativeBuildDeps = with pkgs; [
|
|
# Crypto / TLS
|
|
openssl
|
|
openssl.dev
|
|
|
|
# XML / SAML (enterprise_saml feature)
|
|
libxml2.dev
|
|
xmlsec.dev
|
|
libxslt.dev
|
|
|
|
# FFI / codegen
|
|
libclang.dev
|
|
libffi # deno_ffi on macOS
|
|
|
|
# Networking / compression
|
|
curl.dev
|
|
zlib.dev
|
|
|
|
# Auth (kafka-gssapi, mssql-kerberos)
|
|
cyrus_sasl
|
|
krb5
|
|
|
|
# Misc
|
|
libtool
|
|
postgresql
|
|
|
|
# Build tooling
|
|
pkg-config
|
|
llvmPackages_18.clang # linker — pinned to 18 to avoid SIGSEGV with mold + newer clang
|
|
mold
|
|
cmake # required by rdkafka cmake-build
|
|
];
|
|
|
|
# ---------------------------------------------------------------
|
|
# Prebuilt V8 binary (must match version in Cargo.toml)
|
|
# ---------------------------------------------------------------
|
|
|
|
rustyV8Archive = let
|
|
version = "137.1.0";
|
|
target = stdenv.hostPlatform.rust.rustcTarget;
|
|
sha256 = {
|
|
x86_64-linux =
|
|
"sha256-Tiscfy2bzYGR3s0T+SC1IB3xWvTVpVcSEdjq3MCRoRw=";
|
|
aarch64-linux = lib.fakeHash;
|
|
x86_64-darwin = lib.fakeHash;
|
|
aarch64-darwin = lib.fakeHash;
|
|
}.${system};
|
|
in pkgs.fetchurl {
|
|
name = "librusty_v8-${version}";
|
|
url =
|
|
"https://github.com/denoland/rusty_v8/releases/download/v${version}/librusty_v8_release_${target}.a.gz";
|
|
inherit sha256;
|
|
};
|
|
|
|
# ---------------------------------------------------------------
|
|
# pkg-config search path for native libraries
|
|
# ---------------------------------------------------------------
|
|
|
|
pkgConfigPath = lib.makeSearchPath "lib/pkgconfig" (with pkgs; [
|
|
openssl.dev
|
|
libxml2.dev
|
|
xmlsec.dev
|
|
libxslt.dev
|
|
cyrus_sasl.dev
|
|
krb5.dev
|
|
]);
|
|
|
|
# ---------------------------------------------------------------
|
|
# RPATH — embed Nix store library paths into compiled binaries
|
|
# ---------------------------------------------------------------
|
|
|
|
rpathLibs = lib.makeLibraryPath (with pkgs; [
|
|
openssl
|
|
libffi
|
|
cyrus_sasl
|
|
krb5
|
|
libxml2
|
|
xmlsec
|
|
libxslt
|
|
stdenv.cc.cc.lib
|
|
]);
|
|
|
|
# ---------------------------------------------------------------
|
|
# Bindgen configuration
|
|
# Bindgen uses libclang directly (not $CC), so we must explicitly
|
|
# provide all Nix header search paths.
|
|
# See: https://web.archive.org/web/20220523141208/https://hoverbear.org/blog/rust-bindgen-in-nix/
|
|
# ---------------------------------------------------------------
|
|
|
|
bindgenClangArgs = builtins.concatStringsSep " " ([
|
|
"-nostdinc"
|
|
(builtins.readFile "${stdenv.cc}/nix-support/libc-crt1-cflags")
|
|
(builtins.readFile "${stdenv.cc}/nix-support/libc-cflags")
|
|
(builtins.readFile "${stdenv.cc}/nix-support/cc-cflags")
|
|
(builtins.readFile "${stdenv.cc}/nix-support/libcxx-cxxflags")
|
|
"-idirafter ${pkgs.libiconv}/include"
|
|
] ++ lib.optionals stdenv.cc.isClang [
|
|
"-idirafter ${stdenv.cc.cc}/lib/clang/${
|
|
lib.getVersion stdenv.cc.cc
|
|
}/include"
|
|
] ++ lib.optionals stdenv.cc.isGNU [
|
|
"-isystem ${stdenv.cc.cc}/include/c++/${lib.getVersion stdenv.cc.cc}"
|
|
"-isystem ${stdenv.cc.cc}/include/c++/${
|
|
lib.getVersion stdenv.cc.cc
|
|
}/${stdenv.hostPlatform.config}"
|
|
"-idirafter ${stdenv.cc.cc}/lib/gcc/${stdenv.hostPlatform.config}/${
|
|
lib.getVersion stdenv.cc.cc
|
|
}/include"
|
|
]);
|
|
|
|
# ---------------------------------------------------------------
|
|
# Build environment variables (shared by all shells that compile Rust)
|
|
# ---------------------------------------------------------------
|
|
|
|
buildEnvVars = {
|
|
PKG_CONFIG_PATH = pkgConfigPath;
|
|
RUSTY_V8_ARCHIVE = rustyV8Archive;
|
|
LIBCLANG_PATH = "${pkgs.llvmPackages.libclang.lib}/lib";
|
|
BINDGEN_EXTRA_CLANG_ARGS = bindgenClangArgs;
|
|
|
|
# Force clang 18 as cargo linker (stdenv may bring a newer clang that causes SIGSEGV with mold)
|
|
CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER =
|
|
"${pkgs.llvmPackages_18.clang}/bin/clang";
|
|
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER =
|
|
"${pkgs.llvmPackages_18.clang}/bin/clang";
|
|
|
|
# Embed rpath so binaries find Nix store .so files at runtime
|
|
CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUSTFLAGS =
|
|
"-C link-arg=-fuse-ld=mold -C link-arg=-Wl,-rpath,${rpathLibs}";
|
|
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_RUSTFLAGS =
|
|
"-C link-arg=-fuse-ld=mold -C link-arg=-Wl,-rpath,${rpathLibs}";
|
|
CARGO_HOST_RUSTFLAGS = "-C link-arg=-Wl,-rpath,${rpathLibs}";
|
|
|
|
# https://github.com/NixOS/nixpkgs/issues/370494 — jemalloc build fix
|
|
CFLAGS = "-Wno-error=int-conversion";
|
|
|
|
LD_LIBRARY_PATH = lib.makeLibraryPath [ pkgs.zlib stdenv.cc.cc.lib ];
|
|
};
|
|
|
|
# ---------------------------------------------------------------
|
|
# OpenAPI generator (pinned)
|
|
# ---------------------------------------------------------------
|
|
|
|
openapi-generator-cli =
|
|
(import nixpkgs-oapi-gen { inherit system; }).openapi-generator-cli;
|
|
|
|
# ---------------------------------------------------------------
|
|
# Common worker runtimes (languages the worker executes)
|
|
# ---------------------------------------------------------------
|
|
|
|
commonRuntimes = with pkgs; [
|
|
deno
|
|
python3
|
|
python3Packages.pip
|
|
uv
|
|
go
|
|
bun
|
|
nushell
|
|
typescript
|
|
flock
|
|
];
|
|
|
|
# ---------------------------------------------------------------
|
|
# Runtime PATH env vars — tells the worker where to find interpreters
|
|
# ---------------------------------------------------------------
|
|
|
|
commonRuntimeVars = {
|
|
DENO_PATH = "${pkgs.deno}/bin/deno";
|
|
GO_PATH = "${pkgs.go}/bin/go";
|
|
BUN_PATH = "${pkgs.bun}/bin/bun";
|
|
NODE_PATH = "${pkgs.nodejs}/bin/node";
|
|
NODE_BIN_PATH = "${pkgs.nodejs}/bin/node";
|
|
UV_PATH = "${pkgs.uv}/bin/uv";
|
|
NU_PATH = "${pkgs.nushell}/bin/nu";
|
|
FLOCK_PATH = "${pkgs.flock}/bin/flock";
|
|
CARGO_PATH = "${rustStable}/bin/cargo";
|
|
BASH_PATH = "bash";
|
|
GIT_PATH = "${pkgs.git}/bin/git";
|
|
};
|
|
|
|
# ---------------------------------------------------------------
|
|
# Extra language runtimes (full shell only)
|
|
# ---------------------------------------------------------------
|
|
|
|
coursier = pkgs.fetchFromGitHub {
|
|
owner = "coursier";
|
|
repo = "launchers";
|
|
rev = "79d927f7586c09ca6d8cd01862adb0d9f9d88dff";
|
|
hash = "sha256-8E0WtDFc7RcqmftDigMyy1xXUkjgL4X4kpf7h1GdE48=";
|
|
};
|
|
|
|
rWithPackages = pkgs.rWrapper.override {
|
|
packages = with pkgs.rPackages; [ renv ];
|
|
};
|
|
|
|
extraRuntimes = with pkgs; [
|
|
dotnet-sdk_9
|
|
php
|
|
php84Packages.composer
|
|
ruby_3_4
|
|
jdk21
|
|
ansible
|
|
oracle-instantclient
|
|
];
|
|
|
|
extraRuntimeVars = {
|
|
JAVA_PATH = "${pkgs.jdk21}/bin/java";
|
|
JAVAC_PATH = "${pkgs.jdk21}/bin/javac";
|
|
COURSIER_PATH = "${coursier}/coursier";
|
|
DOTNET_PATH = "${pkgs.dotnet-sdk_9}/bin/dotnet";
|
|
DOTNET_ROOT = "${pkgs.dotnet-sdk_9}/share/dotnet";
|
|
PHP_PATH = "${pkgs.php}/bin/php";
|
|
COMPOSER_PATH = "${pkgs.php84Packages.composer}/bin/composer";
|
|
RUBY_PATH = "${pkgs.ruby_3_4}/bin/ruby";
|
|
RUBY_BUNDLE_PATH = "${pkgs.ruby_3_4}/bin/bundle";
|
|
RUBY_GEM_PATH = "${pkgs.ruby_3_4}/bin/gem";
|
|
ORACLE_LIB_DIR = "${pkgs.oracle-instantclient.lib}/lib";
|
|
ANSIBLE_PLAYBOOK_PATH = "${pkgs.ansible}/bin/ansible-playbook";
|
|
ANSIBLE_GALAXY_PATH = "${pkgs.ansible}/bin/ansible-galaxy";
|
|
CARGO_SWEEP_PATH = "${pkgs.cargo-sweep}/bin/cargo-sweep";
|
|
RSCRIPT_PATH = "${rWithPackages}/bin/Rscript";
|
|
};
|
|
|
|
# ---------------------------------------------------------------
|
|
# General dev environment variables
|
|
# ---------------------------------------------------------------
|
|
|
|
devEnvVars = {
|
|
NODE_ENV = "development";
|
|
NODE_OPTIONS = "--max-old-space-size=16384";
|
|
};
|
|
|
|
# Connection-specific defaults — set via shellHook so they respect
|
|
# pre-existing values (e.g. from webmux runtime.env / .env.local).
|
|
# Nix attrs are injected unconditionally and would override per-worktree
|
|
# values set by webmux before the interactive shell starts.
|
|
devShellHook = ''
|
|
export DATABASE_URL="''${DATABASE_URL:-postgres://postgres:changeme@127.0.0.1:5432/windmill?sslmode=disable}"
|
|
export REMOTE="''${REMOTE:-http://127.0.0.1:8000}"
|
|
export REMOTE_LSP="''${REMOTE_LSP:-http://127.0.0.1:3001}"
|
|
'';
|
|
|
|
# ---------------------------------------------------------------
|
|
# Helper scripts — base set (default + full)
|
|
# ---------------------------------------------------------------
|
|
|
|
# minikube kvm2 driver for the k8s benchmark sim (not in nixpkgs).
|
|
# Built against pkgsSim so glibc/libvirt match the host system.
|
|
kvm2Driver = import ./benchmarks/sim/nix/kvm2-driver.nix { pkgs = pkgsSim; };
|
|
|
|
# `wm_sim` — k8s sim entry, factored out so wasm/cli devShells can pull
|
|
# just it without inheriting the rest of helperScriptsBase.
|
|
wmSimWrapper = pkgs.writeShellScriptBin "wm_sim" ''
|
|
export SIM_KVM2_DRIVER_DIR="${kvm2Driver}/bin"
|
|
# minikube + helm from pkgsSim so they share the kvm2 driver's libc.
|
|
export SIM_MINIKUBE_BIN="${pkgsSim.minikube}/bin/minikube"
|
|
export SIM_HELM_BIN="${pkgsSim.kubernetes-helm}/bin/helm"
|
|
# System libvirt — matches the system virsh that minikube's kvm2
|
|
# preflight invokes. Falls back to pkgsSim's libvirt (also 12.2 in
|
|
# nixos-unstable) if the system path can't be resolved.
|
|
sysvirsh="$(command -v virsh 2>/dev/null || echo /run/current-system/sw/bin/virsh)"
|
|
syslib="$(ldd "$sysvirsh" 2>/dev/null | awk '/libvirt\.so\.0/{print $3}' | head -1)"
|
|
if [ -n "$syslib" ]; then
|
|
export SIM_LIBVIRT_LIB_DIR="$(dirname "$syslib")"
|
|
else
|
|
export SIM_LIBVIRT_LIB_DIR="${pkgsSim.libvirt}/lib"
|
|
fi
|
|
# virsh path for the provisioner's pre-start sweep of leaked domains.
|
|
export SIM_VIRSH_BIN="$sysvirsh"
|
|
root="$(git rev-parse --show-toplevel 2>/dev/null || echo .)"
|
|
exec ${pkgs.deno}/bin/deno run --no-check -A "$root/benchmarks/sim/sim.ts" "$@"
|
|
'';
|
|
|
|
helperScriptsBase = [
|
|
(pkgs.writeScriptBin "wm" ''
|
|
cd ./frontend
|
|
npm install
|
|
npm run ${
|
|
if stdenv.isDarwin then
|
|
"generate-backend-client-mac"
|
|
else
|
|
"generate-backend-client"
|
|
}
|
|
npm run dev "$@"
|
|
'')
|
|
(pkgs.writeScriptBin "wm-build" ''
|
|
cd ./frontend
|
|
npm install
|
|
npm run ${
|
|
if stdenv.isDarwin then
|
|
"generate-backend-client-mac"
|
|
else
|
|
"generate-backend-client"
|
|
}
|
|
npm run build "$@"
|
|
'')
|
|
(pkgs.writeScriptBin "wm-migrate" ''
|
|
cd ./backend
|
|
sqlx migrate run
|
|
'')
|
|
(pkgs.writeScriptBin "wm-reset" ''
|
|
sqlx database drop -f
|
|
sqlx database create
|
|
wm-migrate
|
|
'')
|
|
(pkgs.writeScriptBin "wm-minio" ''
|
|
set -e
|
|
cd ./backend
|
|
mkdir -p .minio-data/wmill
|
|
${pkgs.minio}/bin/minio server ./.minio-data --console-address ":9001"
|
|
'')
|
|
(pkgs.writeScriptBin "wm-minio-keys" ''
|
|
set -e
|
|
cd ./backend
|
|
${pkgs.minio-client}/bin/mc alias set 'wmill-minio-dev' 'http://localhost:9000' 'minioadmin' 'minioadmin'
|
|
if [[ -f .minio-data/secrets.txt ]] && [[ -s .minio-data/secrets.txt ]]; then
|
|
echo "Access keys already exist:"
|
|
cat .minio-data/secrets.txt
|
|
echo ""
|
|
echo "Keys loaded from: ./backend/.minio-data/secrets.txt"
|
|
else
|
|
echo "Creating new access keys..."
|
|
mkdir -p .minio-data
|
|
${pkgs.minio-client}/bin/mc admin accesskey create 'wmill-minio-dev' | tee .minio-data/secrets.txt
|
|
echo ""
|
|
echo 'New keys saved to: ./backend/.minio-data/secrets.txt'
|
|
fi
|
|
echo "bucket: wmill"
|
|
echo "endpoint: http://localhost:9000"
|
|
'')
|
|
# k8s sim entry — provisioning only, no bench. See `wmSimWrapper`.
|
|
wmSimWrapper
|
|
# Bench entry — same env shape as wm_sim so `--topology` (sim-driven
|
|
# provisioning) works. For plain `--host` benches the env is harmless.
|
|
(pkgs.writeShellScriptBin "wm-bench" ''
|
|
export SIM_KVM2_DRIVER_DIR="${kvm2Driver}/bin"
|
|
export SIM_MINIKUBE_BIN="${pkgsSim.minikube}/bin/minikube"
|
|
export SIM_HELM_BIN="${pkgsSim.kubernetes-helm}/bin/helm"
|
|
sysvirsh="$(command -v virsh 2>/dev/null || echo /run/current-system/sw/bin/virsh)"
|
|
syslib="$(ldd "$sysvirsh" 2>/dev/null | awk '/libvirt\.so\.0/{print $3}' | head -1)"
|
|
if [ -n "$syslib" ]; then
|
|
export SIM_LIBVIRT_LIB_DIR="$(dirname "$syslib")"
|
|
else
|
|
export SIM_LIBVIRT_LIB_DIR="${pkgsSim.libvirt}/lib"
|
|
fi
|
|
export SIM_VIRSH_BIN="$sysvirsh"
|
|
root="$(git rev-parse --show-toplevel 2>/dev/null || echo .)"
|
|
exec ${pkgs.deno}/bin/deno run --no-check -A "$root/benchmarks/main.ts" -e admin@windmill.dev -p changeme "$@"
|
|
'')
|
|
];
|
|
|
|
# ---------------------------------------------------------------
|
|
# Helper scripts — extra (full shell only)
|
|
# ---------------------------------------------------------------
|
|
|
|
helperScriptsFull = [
|
|
(pkgs.writeScriptBin "wm-caddy" ''
|
|
cd ./frontend
|
|
xcaddy build "$@" \
|
|
--with github.com/mholt/caddy-l4@145ec36251a44286f05a10d231d8bfb3a8192e09 \
|
|
--with github.com/RussellLuo/caddy-ext/layer4@ab1e18cfe426012af351a68463937ae2e934a2a1
|
|
'')
|
|
(pkgs.writeScriptBin "wm-setup" ''
|
|
sqlx database create
|
|
wm-build
|
|
wm-caddy
|
|
wm-migrate
|
|
'')
|
|
];
|
|
|
|
# ---------------------------------------------------------------
|
|
# Shared inputs and settings for default + full shells
|
|
# ---------------------------------------------------------------
|
|
|
|
coreBuildInputs = nativeBuildDeps ++ commonRuntimes
|
|
++ [ rustStable openapi-generator-cli ] ++ (with pkgs; [
|
|
nodejs
|
|
git
|
|
sqlx-cli
|
|
cargo-watch
|
|
jq
|
|
gnused
|
|
|
|
# CLI tools (for AI agents and dev workflow)
|
|
gh
|
|
asciinema
|
|
mermaid-cli
|
|
|
|
# Network shaping / topology benches
|
|
toxiproxy
|
|
pgbadger
|
|
|
|
# Local k8s for sim topologies. Single-node uses the built-in qemu2
|
|
# driver (no libvirt, no sudo). Multi-node (--nodes=N) needs the
|
|
# kvm2 driver — the docker-machine-driver-kvm2 binary isn't in
|
|
# nixpkgs and minikube's auto-download won't run on NixOS, so kvm2
|
|
# requires a custom overlay + host-level libvirtd. qemu is shared by
|
|
# both. VM-based nodes sidestep the rootless-cgroup wall (k3d/kind).
|
|
minikube
|
|
kubectl
|
|
kubernetes-helm
|
|
qemu
|
|
]);
|
|
|
|
# Playwright: use Nix-provided browsers (version-matched to playwright-driver)
|
|
# Mermaid/Puppeteer: point at Nix chromium (Puppeteer respects this env var)
|
|
browserVars = {
|
|
PLAYWRIGHT_BROWSERS_PATH = "${pkgs.playwright-driver.browsers}";
|
|
PUPPETEER_EXECUTABLE_PATH = "${pkgs.chromium}/bin/chromium";
|
|
PUPPETEER_SKIP_DOWNLOAD = "true";
|
|
};
|
|
|
|
# Wrapper for the Nix-provided playwright CLI (version-matched to its browsers)
|
|
playwrightWrapper = pkgs.writeShellScriptBin "playwright" ''
|
|
export PLAYWRIGHT_BROWSERS_PATH="${pkgs.playwright-driver.browsers}"
|
|
exec ${pkgs.nodejs}/bin/node ${pkgs.playwright-driver}/cli.js "$@"
|
|
'';
|
|
|
|
# ---------------------------------------------------------------
|
|
# sandbox-env script — outputs env vars for browser tooling
|
|
# Usage: eval "$(sandbox-env)"
|
|
# ---------------------------------------------------------------
|
|
|
|
sandboxEnvScript = pkgs.writeShellScriptBin "sandbox-env" ''
|
|
echo "export PLAYWRIGHT_BROWSERS_PATH=${pkgs.playwright-driver.browsers}"
|
|
echo "export PUPPETEER_EXECUTABLE_PATH=${pkgs.chromium}/bin/chromium"
|
|
echo "export PUPPETEER_SKIP_DOWNLOAD=true"
|
|
'';
|
|
|
|
# ---------------------------------------------------------------
|
|
# pkg-config wrapper — bakes in the Nix pkg-config search path
|
|
# so sandbox profiles (buildEnv) work without setting env vars.
|
|
# ---------------------------------------------------------------
|
|
|
|
pkgConfigWrapper = pkgs.writeShellScriptBin "pkg-config" ''
|
|
export PKG_CONFIG_PATH="${pkgConfigPath}:$PKG_CONFIG_PATH"
|
|
exec ${pkgs.pkg-config}/bin/pkg-config "$@"
|
|
'';
|
|
|
|
# ---------------------------------------------------------------
|
|
# Installable sandbox profiles (nix profile install .#sandbox)
|
|
# ---------------------------------------------------------------
|
|
|
|
sandboxEnv = pkgs.buildEnv {
|
|
name = "windmill-sandbox";
|
|
paths = coreBuildInputs ++ helperScriptsBase ++ [
|
|
playwrightWrapper
|
|
sandboxEnvScript
|
|
pkgConfigWrapper
|
|
pkgs.chromium
|
|
];
|
|
};
|
|
|
|
sandboxFullEnv = pkgs.buildEnv {
|
|
name = "windmill-sandbox-full";
|
|
paths = coreBuildInputs ++ extraRuntimes ++ helperScriptsBase
|
|
++ helperScriptsFull ++ [
|
|
playwrightWrapper
|
|
sandboxEnvScript
|
|
pkgConfigWrapper
|
|
pkgs.chromium
|
|
pkgs.cargo-sweep
|
|
pkgs.xcaddy
|
|
pkgs.nsjail
|
|
];
|
|
};
|
|
|
|
in {
|
|
|
|
# =============================================================
|
|
# Installable profiles — for Docker / nix profile install
|
|
# Usage: nix profile install .#sandbox
|
|
# =============================================================
|
|
|
|
packages.sandbox = sandboxEnv;
|
|
packages.sandbox-full = sandboxFullEnv;
|
|
packages.default = sandboxEnv;
|
|
|
|
# =============================================================
|
|
# default — daily driver for backend + frontend development
|
|
# Usage: nix develop
|
|
# =============================================================
|
|
|
|
devShells.default = pkgs.mkShell (buildEnvVars // commonRuntimeVars // devEnvVars // browserVars // {
|
|
shellHook = devShellHook;
|
|
buildInputs = coreBuildInputs;
|
|
|
|
packages = helperScriptsBase ++ [ playwrightWrapper ];
|
|
});
|
|
|
|
# =============================================================
|
|
# full — all language runtimes, k8s tooling, specialized scripts
|
|
# Usage: nix develop .#full
|
|
# =============================================================
|
|
|
|
devShells.full = pkgs.mkShell (buildEnvVars // commonRuntimeVars // extraRuntimeVars // devEnvVars // browserVars // {
|
|
shellHook = devShellHook;
|
|
buildInputs = coreBuildInputs ++ extraRuntimes ++ (with pkgs; [
|
|
# Python extras
|
|
poetry
|
|
pyright
|
|
openapi-python-client
|
|
|
|
# LSP / editor
|
|
svelte-language-server
|
|
taplo
|
|
|
|
# Extra dev tools
|
|
cargo-sweep
|
|
|
|
# Kubernetes
|
|
minikube
|
|
kubectl
|
|
kubernetes-helm
|
|
conntrack-tools
|
|
cri-tools
|
|
|
|
# Extra
|
|
xcaddy
|
|
nsjail
|
|
]);
|
|
|
|
packages = helperScriptsBase ++ helperScriptsFull
|
|
++ [ playwrightWrapper ];
|
|
});
|
|
|
|
# =============================================================
|
|
# wasm — WASM target compilation (nightly Rust)
|
|
# Usage: nix develop .#wasm
|
|
# =============================================================
|
|
|
|
devShells.wasm = pkgs.mkShell (buildEnvVars // {
|
|
hardeningDisable = [ "all" ];
|
|
|
|
# Explicitly set paths for headers and linker
|
|
# DO NOT REMOVE - if absent, breaks wasm builds on NixOS.
|
|
shellHook = ''
|
|
export CC=${patchedClang}/bin/clang
|
|
'';
|
|
|
|
buildInputs = nativeBuildDeps ++ (with pkgs; [
|
|
(rust-bin.nightly.latest.default.override {
|
|
extensions = [ "rust-src" "rust-analyzer" ];
|
|
targets =
|
|
[ "wasm32-unknown-unknown" "wasm32-unknown-emscripten" ];
|
|
})
|
|
wasm-pack
|
|
deno
|
|
emscripten
|
|
nushell
|
|
nodejs
|
|
glibc_multi
|
|
]);
|
|
packages = [ wmSimWrapper ];
|
|
});
|
|
|
|
# =============================================================
|
|
# cli — lightweight Bun-based CLI development
|
|
# Usage: nix develop .#cli
|
|
# =============================================================
|
|
|
|
devShells.cli = pkgs.mkShell {
|
|
shellHook = ''
|
|
if command -v git >/dev/null 2>&1 && git rev-parse --is-inside-work-tree >/dev/null 2>&1; then
|
|
export FLAKE_ROOT="$(git rev-parse --show-toplevel)"
|
|
else
|
|
export FLAKE_ROOT="$PWD"
|
|
fi
|
|
'';
|
|
|
|
buildInputs = with pkgs; [ bun nodejs git ];
|
|
|
|
packages = [
|
|
wmSimWrapper
|
|
(pkgs.writeScriptBin "wm-cli" ''
|
|
bun run $FLAKE_ROOT/cli/src/main.ts "$@"
|
|
'')
|
|
(pkgs.writeScriptBin "wm-cli-deps" ''
|
|
pushd $FLAKE_ROOT/cli/
|
|
${if stdenv.isDarwin then
|
|
"./gen_wm_client_mac.sh && ./windmill-utils-internal/gen_wm_client_mac.sh"
|
|
else
|
|
"./gen_wm_client.sh && ./windmill-utils-internal/gen_wm_client.sh"}
|
|
popd
|
|
'')
|
|
];
|
|
};
|
|
});
|
|
}
|