docs: add seccomp and apparmor profiles for non-privileged nsjail

This commit is contained in:
Ruben Fiszel
2026-10-06 09:36:04 +02:00
parent 0abc0ac0c4
commit b3a1693d2f
6 changed files with 1130 additions and 0 deletions
@@ -0,0 +1,63 @@
Security profiles for nsjail without a privileged container
============================================================
[nsjail sandboxing](https://www.windmill.dev/docs/advanced/security_isolation#nsjail-sandboxing) builds each job its own PID and mount namespaces. A container runtime's default seccomp and AppArmor profiles block the system calls and mounts this takes, which is why nsjail is usually run with `privileged: true` or with both profiles set to `Unconfined`.
The two profiles here are the runtime defaults with only what nsjail needs added, so a worker can run nsjail in a non-privileged container that keeps a seccomp filter and an AppArmor profile. See [Running nsjail without privileged: true](https://www.windmill.dev/docs/advanced/security_isolation#running-nsjail-without-privileged-true) for the full guide.
## Files
| File | Purpose |
| --- | --- |
| `windmill-nsjail.seccomp.json` | Docker's default seccomp profile plus `clone` with namespace flags, `mount`, `umount2`, `pivot_root` and `sethostname` |
| `windmill-nsjail.apparmor` | The runtime default AppArmor profile with `deny mount` replaced by `mount`, `umount` and `pivot_root` rules |
| `kubernetes-user-namespaces.yaml` | Worker pod settings with no added capability (`hostUsers: false`, `procMount: Unmasked`) |
| `kubernetes-sys-admin.yaml` | Worker pod settings for clusters without user namespaces (`SYS_ADMIN`) |
| `generate-seccomp.py` | Rebuilds the seccomp profile from a newer Docker default |
## Installing the profiles on Kubernetes nodes
Both profiles are read from the node, so they have to be present on every node that runs workers.
```bash
# seccomp: relative to the kubelet's seccomp directory
sudo install -D -m 0644 windmill-nsjail.seccomp.json /var/lib/kubelet/seccomp/profiles/windmill-nsjail.json
# AppArmor: only on nodes where AppArmor is enabled
sudo apparmor_parser -r windmill-nsjail.apparmor
```
On nodes without AppArmor, remove the `appArmorProfile` block from the pod settings.
## Using the profiles with Docker
```bash
sudo apparmor_parser -r windmill-nsjail.apparmor
docker run \
--user 1000:1000 --cap-drop ALL \
--security-opt no-new-privileges \
--security-opt systempaths=unconfined \
--security-opt seccomp=windmill-nsjail.seccomp.json \
--security-opt apparmor=windmill-nsjail \
-e DISABLE_NSJAIL=false \
...
```
## Which pod settings to use
| | `kubernetes-user-namespaces.yaml` | `kubernetes-sys-admin.yaml` |
| --- | --- | --- |
| Needs | Kubernetes 1.33+ with [user namespaces](https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/) | Any cluster |
| Worker user | uid 1000 | root |
| Added capabilities | none | `SYS_ADMIN`, `SETPCAP` |
| Pod Security Standards | `baseline` on 1.35+; 1.33 and 1.34 reject `procMount: Unmasked` unless the `UserNamespacesPodSecurityStandards` feature gate is on | needs an exemption for the added capabilities |
## Regenerating the seccomp profile
`windmill-nsjail.seccomp.json` is derived from the [default profile of the Moby project](https://github.com/moby/profiles/blob/main/seccomp/default.json) (Apache License 2.0). To rebase it on a newer default:
```bash
curl -fsSLO https://raw.githubusercontent.com/moby/profiles/main/seccomp/default.json
python3 generate-seccomp.py default.json windmill-nsjail.seccomp.json
```
@@ -0,0 +1,37 @@
#!/usr/bin/env python3
"""Regenerate windmill-nsjail.seccomp.json from Docker's default seccomp profile.
usage: generate-seccomp.py <moby default.json> <output.json>
Docker's default.json carries `includes`/`excludes` conditions that only Docker
evaluates; a Kubernetes Localhost profile is handed to the runtime as is, where
those conditions are ignored and every rule applies. The output is therefore the
profile as Docker resolves it for a container with no added capability, with the
conditions removed, plus the syscalls nsjail needs.
"""
import json
import sys
# clone is allowed with namespace flags; clone3 keeps returning ENOSYS, which
# makes nsjail fall back to clone.
NSJAIL_SYSCALLS = ["clone", "mount", "pivot_root", "sethostname", "umount2"]
ARCHES = {"amd64", "x32", "x86", "arm", "arm64"}
src, out = sys.argv[1], sys.argv[2]
profile = json.load(open(src))
rules = []
for rule in profile["syscalls"]:
includes = rule.get("includes", {})
if includes.get("caps"):
continue
if includes.get("arches") and not ARCHES & set(includes["arches"]):
continue
rule = {k: v for k, v in rule.items() if k not in ("includes", "excludes", "comment")}
rule["names"] = [n for n in rule["names"] if n not in NSJAIL_SYSCALLS]
if rule["names"]:
rules.append(rule)
rules.append({"names": NSJAIL_SYSCALLS, "action": "SCMP_ACT_ALLOW"})
profile["syscalls"] = rules
with open(out, "w") as f:
json.dump(profile, f, indent=1)
f.write("\n")
@@ -0,0 +1,29 @@
# Worker pod settings for nsjail on clusters without user namespace support.
# Needs a Pod Security exemption for the added capabilities.
apiVersion: v1
kind: Pod
metadata:
name: windmill-worker
spec:
containers:
- name: windmill-worker
image: ghcr.io/windmill-labs/windmill:main
env:
- { name: MODE, value: worker }
- { name: DISABLE_NSJAIL, value: "false" }
- { name: DISABLE_NUSER, value: "true" }
securityContext:
runAsUser: 0
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
add:
- SYS_ADMIN # nsjail: namespaces, mounts, pivot_root
- SETPCAP # nsjail: dropping capabilities before running the job
- SYS_RESOURCE # optional, worker: lowering its own oom_score_adj
seccompProfile:
type: Localhost
localhostProfile: profiles/windmill-nsjail.json
appArmorProfile:
type: Localhost
localhostProfile: windmill-nsjail
@@ -0,0 +1,29 @@
# Worker pod settings for nsjail with no added capability.
# Admitted by the Pod Security Standards "baseline" level on Kubernetes 1.35+.
apiVersion: v1
kind: Pod
metadata:
name: windmill-worker
spec:
hostUsers: false
containers:
- name: windmill-worker
image: ghcr.io/windmill-labs/windmill:main
env:
- { name: MODE, value: worker }
- { name: DISABLE_NSJAIL, value: "false" }
# DISABLE_NUSER stays unset: nsjail creates its own user namespace.
securityContext:
runAsUser: 1000
runAsGroup: 1000
runAsNonRoot: true
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
procMount: Unmasked
seccompProfile:
type: Localhost
localhostProfile: profiles/windmill-nsjail.json
appArmorProfile:
type: Localhost
localhostProfile: windmill-nsjail
@@ -0,0 +1,40 @@
#include <tunables/global>
# The container runtimes' default profile (docker-default,
# cri-containerd.apparmor.d) with its blanket "deny mount" replaced by the
# mount, umount and pivot_root rules nsjail needs to build a job's filesystem.
profile windmill-nsjail flags=(attach_disconnected,mediate_deleted) {
#include <abstractions/base>
network,
capability,
file,
mount,
umount,
pivot_root,
signal (receive) peer=unconfined,
signal (receive) peer=runc,
signal (receive) peer=crun,
signal (receive) peer=cri-containerd.apparmor.d,
signal (send,receive) peer=windmill-nsjail,
deny @{PROC}/* w,
deny @{PROC}/{[^1-9],[^1-9][^0-9],[^1-9s][^0-9y][^0-9s],[^1-9][^0-9][^0-9][^0-9/]*}/** w,
deny @{PROC}/sys/[^k]** w,
deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w,
deny @{PROC}/sysrq-trigger rwklx,
deny @{PROC}/kcore rwklx,
deny /sys/[^f]*/** wklx,
deny /sys/f[^s]*/** wklx,
deny /sys/fs/[^c]*/** wklx,
deny /sys/fs/c[^g]*/** wklx,
deny /sys/fs/cg[^r]*/** wklx,
deny /sys/firmware/** rwklx,
deny /sys/devices/virtual/powercap/** rwklx,
deny /sys/kernel/security/** rwklx,
ptrace (trace,read,tracedby,readby) peer=windmill-nsjail,
}
@@ -0,0 +1,932 @@
{
"defaultAction": "SCMP_ACT_ERRNO",
"defaultErrnoRet": 1,
"archMap": [
{
"architecture": "SCMP_ARCH_X86_64",
"subArchitectures": [
"SCMP_ARCH_X86",
"SCMP_ARCH_X32"
]
},
{
"architecture": "SCMP_ARCH_AARCH64",
"subArchitectures": [
"SCMP_ARCH_ARM"
]
},
{
"architecture": "SCMP_ARCH_MIPS64",
"subArchitectures": [
"SCMP_ARCH_MIPS",
"SCMP_ARCH_MIPS64N32"
]
},
{
"architecture": "SCMP_ARCH_MIPS64N32",
"subArchitectures": [
"SCMP_ARCH_MIPS",
"SCMP_ARCH_MIPS64"
]
},
{
"architecture": "SCMP_ARCH_MIPSEL64",
"subArchitectures": [
"SCMP_ARCH_MIPSEL",
"SCMP_ARCH_MIPSEL64N32"
]
},
{
"architecture": "SCMP_ARCH_MIPSEL64N32",
"subArchitectures": [
"SCMP_ARCH_MIPSEL",
"SCMP_ARCH_MIPSEL64"
]
},
{
"architecture": "SCMP_ARCH_S390X",
"subArchitectures": [
"SCMP_ARCH_S390"
]
},
{
"architecture": "SCMP_ARCH_RISCV64",
"subArchitectures": null
},
{
"architecture": "SCMP_ARCH_LOONGARCH64",
"subArchitectures": null
}
],
"syscalls": [
{
"names": [
"accept",
"accept4",
"access",
"adjtimex",
"alarm",
"bind",
"brk",
"cachestat",
"capget",
"capset",
"chdir",
"chmod",
"chown",
"chown32",
"clock_adjtime",
"clock_adjtime64",
"clock_getres",
"clock_getres_time64",
"clock_gettime",
"clock_gettime64",
"clock_nanosleep",
"clock_nanosleep_time64",
"close",
"close_range",
"connect",
"copy_file_range",
"creat",
"dup",
"dup2",
"dup3",
"epoll_create",
"epoll_create1",
"epoll_ctl",
"epoll_ctl_old",
"epoll_pwait",
"epoll_pwait2",
"epoll_wait",
"epoll_wait_old",
"eventfd",
"eventfd2",
"execve",
"execveat",
"exit",
"exit_group",
"faccessat",
"faccessat2",
"fadvise64",
"fadvise64_64",
"fallocate",
"fanotify_mark",
"fchdir",
"fchmod",
"fchmodat",
"fchmodat2",
"fchown",
"fchown32",
"fchownat",
"fcntl",
"fcntl64",
"fdatasync",
"fgetxattr",
"flistxattr",
"flock",
"fork",
"fremovexattr",
"fsetxattr",
"fstat",
"fstat64",
"fstatat64",
"fstatfs",
"fstatfs64",
"fsync",
"ftruncate",
"ftruncate64",
"futex",
"futex_requeue",
"futex_time64",
"futex_wait",
"futex_waitv",
"futex_wake",
"futimesat",
"getcpu",
"getcwd",
"getdents",
"getdents64",
"getegid",
"getegid32",
"geteuid",
"geteuid32",
"getgid",
"getgid32",
"getgroups",
"getgroups32",
"getitimer",
"getpeername",
"getpgid",
"getpgrp",
"getpid",
"getppid",
"getpriority",
"getrandom",
"getresgid",
"getresgid32",
"getresuid",
"getresuid32",
"getrlimit",
"get_robust_list",
"getrusage",
"getsid",
"getsockname",
"getsockopt",
"get_thread_area",
"gettid",
"gettimeofday",
"getuid",
"getuid32",
"getxattr",
"getxattrat",
"inotify_add_watch",
"inotify_init",
"inotify_init1",
"inotify_rm_watch",
"io_cancel",
"ioctl",
"io_destroy",
"io_getevents",
"io_pgetevents",
"io_pgetevents_time64",
"ioprio_get",
"ioprio_set",
"io_setup",
"io_submit",
"ipc",
"kill",
"landlock_add_rule",
"landlock_create_ruleset",
"landlock_restrict_self",
"lchown",
"lchown32",
"lgetxattr",
"link",
"linkat",
"listen",
"listmount",
"listxattr",
"listxattrat",
"llistxattr",
"_llseek",
"lremovexattr",
"lseek",
"lsetxattr",
"lstat",
"lstat64",
"madvise",
"map_shadow_stack",
"membarrier",
"memfd_create",
"memfd_secret",
"mincore",
"mkdir",
"mkdirat",
"mknod",
"mknodat",
"mlock",
"mlock2",
"mlockall",
"mmap",
"mmap2",
"mprotect",
"mq_getsetattr",
"mq_notify",
"mq_open",
"mq_timedreceive",
"mq_timedreceive_time64",
"mq_timedsend",
"mq_timedsend_time64",
"mq_unlink",
"mremap",
"mseal",
"msgctl",
"msgget",
"msgrcv",
"msgsnd",
"msync",
"munlock",
"munlockall",
"munmap",
"name_to_handle_at",
"nanosleep",
"newfstatat",
"_newselect",
"open",
"openat",
"openat2",
"pause",
"pidfd_open",
"pidfd_send_signal",
"pipe",
"pipe2",
"pkey_alloc",
"pkey_free",
"pkey_mprotect",
"poll",
"ppoll",
"ppoll_time64",
"prctl",
"pread64",
"preadv",
"preadv2",
"prlimit64",
"process_mrelease",
"pselect6",
"pselect6_time64",
"pwrite64",
"pwritev",
"pwritev2",
"read",
"readahead",
"readlink",
"readlinkat",
"readv",
"recv",
"recvfrom",
"recvmmsg",
"recvmmsg_time64",
"recvmsg",
"remap_file_pages",
"removexattr",
"removexattrat",
"rename",
"renameat",
"renameat2",
"restart_syscall",
"riscv_hwprobe",
"rmdir",
"rseq",
"rt_sigaction",
"rt_sigpending",
"rt_sigprocmask",
"rt_sigqueueinfo",
"rt_sigreturn",
"rt_sigsuspend",
"rt_sigtimedwait",
"rt_sigtimedwait_time64",
"rt_tgsigqueueinfo",
"sched_getaffinity",
"sched_getattr",
"sched_getparam",
"sched_get_priority_max",
"sched_get_priority_min",
"sched_getscheduler",
"sched_rr_get_interval",
"sched_rr_get_interval_time64",
"sched_setaffinity",
"sched_setattr",
"sched_setparam",
"sched_setscheduler",
"sched_yield",
"seccomp",
"select",
"semctl",
"semget",
"semop",
"semtimedop",
"semtimedop_time64",
"send",
"sendfile",
"sendfile64",
"sendmmsg",
"sendmsg",
"sendto",
"setfsgid",
"setfsgid32",
"setfsuid",
"setfsuid32",
"setgid",
"setgid32",
"setgroups",
"setgroups32",
"setitimer",
"setpgid",
"setpriority",
"setregid",
"setregid32",
"setresgid",
"setresgid32",
"setresuid",
"setresuid32",
"setreuid",
"setreuid32",
"setrlimit",
"set_robust_list",
"setsid",
"setsockopt",
"set_thread_area",
"set_tid_address",
"setuid",
"setuid32",
"setxattr",
"setxattrat",
"shmat",
"shmctl",
"shmdt",
"shmget",
"shutdown",
"sigaltstack",
"signalfd",
"signalfd4",
"sigprocmask",
"sigreturn",
"socketcall",
"socketpair",
"splice",
"stat",
"stat64",
"statfs",
"statfs64",
"statmount",
"statx",
"symlink",
"symlinkat",
"sync",
"sync_file_range",
"syncfs",
"sysinfo",
"tee",
"tgkill",
"time",
"timer_create",
"timer_delete",
"timer_getoverrun",
"timer_gettime",
"timer_gettime64",
"timer_settime",
"timer_settime64",
"timerfd_create",
"timerfd_gettime",
"timerfd_gettime64",
"timerfd_settime",
"timerfd_settime64",
"times",
"tkill",
"truncate",
"truncate64",
"ugetrlimit",
"umask",
"uname",
"unlink",
"unlinkat",
"uretprobe",
"utime",
"utimensat",
"utimensat_time64",
"utimes",
"vfork",
"vmsplice",
"wait4",
"waitid",
"waitpid",
"write",
"writev"
],
"action": "SCMP_ACT_ALLOW"
},
{
"names": [
"process_vm_readv",
"process_vm_writev",
"ptrace"
],
"action": "SCMP_ACT_ALLOW"
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 3,
"op": "SCMP_CMP_LT"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 7,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 8,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 9,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 10,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 13,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 14,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 15,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 16,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 17,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 18,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 21,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 22,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 24,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 26,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 27,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 28,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 29,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 30,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 31,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 32,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 33,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 35,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 36,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 39,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 41,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 42,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 43,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 44,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"socket"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 45,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"personality"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 0,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"personality"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 8,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"personality"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 131072,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"personality"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 131080,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"personality"
],
"action": "SCMP_ACT_ALLOW",
"args": [
{
"index": 0,
"value": 4294967295,
"op": "SCMP_CMP_EQ"
}
]
},
{
"names": [
"arm_fadvise64_64",
"arm_sync_file_range",
"sync_file_range2",
"breakpoint",
"cacheflush",
"set_tls"
],
"action": "SCMP_ACT_ALLOW"
},
{
"names": [
"arch_prctl"
],
"action": "SCMP_ACT_ALLOW"
},
{
"names": [
"modify_ldt"
],
"action": "SCMP_ACT_ALLOW"
},
{
"names": [
"clone3"
],
"action": "SCMP_ACT_ERRNO",
"errnoRet": 38
},
{
"names": [
"clone",
"mount",
"pivot_root",
"sethostname",
"umount2"
],
"action": "SCMP_ACT_ALLOW"
}
]
}