feat: always create and mount BLOB_FS_ROOT for a node because its storage layer exits when the directory is not writable and an unmounted path is root-owned by docker, and say plainly in the script and the docs that filesystem blobs cannot serve a fleet at all since the worker has to read the body the backend wrote

This commit is contained in:
Matthew Meszaros
2026-09-09 06:06:04 -07:00
parent 4fc6d1db77
commit 75a068cd07
3 changed files with 52 additions and 15 deletions
+2 -2
View File
@@ -559,9 +559,9 @@ tls {
}
```
Then set `NATS_URL=tls://<token>@nats.yourdomain.com:4222` in `warmbly.env` and `worker.env`; the token in the URL is what every service authenticates with. Redis gets `requirepass` and `REDIS=redis://:<password>@redis.yourdomain.com:6379`, ideally over a private network or a tunnel. Restrict both ports at the firewall to the worker's address as well. Use `BLOB_PROVIDER=s3` with a bucket both sides can reach; a remote worker on the filesystem provider writes to its own disk. The [self-hosting guide](/development/deployment-guide/#remote-workers) has the same caveats in more detail.
Then set `NATS_URL=tls://<token>@nats.yourdomain.com:4222` in `warmbly.env` and `worker.env`; the token in the URL is what every service authenticates with. Redis gets `requirepass` and `REDIS=redis://:<password>@redis.yourdomain.com:6379`, ideally over a private network or a tunnel. Restrict both ports at the firewall to the worker's address as well. Use `BLOB_PROVIDER=s3` with a bucket both sides can reach: a worker reads the message body the backend wrote, so the filesystem provider only works when the two share storage and the permissions line up. The [self-hosting guide](/development/deployment-guide/#adding-a-machine) has the same caveats in more detail.
Because the worker is not in a container, the admin panel's SSH-driven day-two actions (pull image, restart container) do not apply to it. Manage it with `systemctl` and the update steps below.
Because the worker is not in a container, the join script's systemd service and update timer do not apply to it. Manage it with `systemctl` and the update steps below, and set its version by hand rather than through `warmblyctl fleet version`.
## Upgrading
@@ -725,7 +725,7 @@ The config handed to a node is generated from the backend's own environment. On
That config carries the decryption material the node needs: the internal API token, `KMS_LOCAL_MASTER_KEY`, and `CREDENTIALS_ENCRYPTION_KEY`. Serve the API over HTTPS before adding a node across a network you do not control. It deliberately does not include `PRIMARY_DB`: a worker reaches relational data through the internal API and nothing else.
On `BLOB_PROVIDER=filesystem`, a remote node writes blobs to its own local disk rather than a volume the backend shares. Use `BLOB_PROVIDER=s3` with a bucket both sides can reach when you run nodes off-host.
`BLOB_PROVIDER=filesystem` does not survive a fleet. A worker reads the message body the backend wrote, so the two need the same storage with permissions that let both reach it, and a node on another machine has neither. The join script creates and mounts `BLOB_FS_ROOT` so the node starts, and warns you, but sends will fail when the worker cannot read the body. Use `BLOB_PROVIDER=s3` with a bucket both sides can reach before running nodes off-host.
<Mermaid
chart={`
+49 -12
View File
@@ -222,26 +222,62 @@ write_config() {
}
# docker_mounts is every -v argument, on ONE line. Command substitution strips
# trailing newlines, so a multi-line value here would collapse the unit's
# trailing newlines, so a multi-line value would collapse the unit's
# continuations and hand docker a stray token as the image name.
#
# BLOB_FS_ROOT is mounted only when it already exists: it is copied from the
# backend's environment and may name a path that belongs to a co-located
# bare-metal install, so this neither creates it nor changes its ownership. A
# node on another host writes blobs to its own disk regardless, which is why
# the docs tell you to use object storage for a distributed fleet.
# BLOB_FS_ROOT needs a mount whatever happens: the node's storage layer does
# MkdirAll on it at boot and the process exits if that fails, so leaving the
# path unmounted means docker creates it root-owned and the container, running
# as uid 1000, restart-loops. Creating and owning it here is the only branch
# that reliably starts.
#
# Sharing it with a co-located backend is a different problem and not one this
# script can solve: the backend writes objects 0600 as its own system user, so
# a node cannot read them without matching ids. warn_shared_blobs says so
# rather than pretending otherwise; object storage is the supported shape for a
# fleet.
docker_mounts() {
mounts="-v $AGENT_DIR:$AGENT_DIR"
provider=$(sed -n 's/^BLOB_PROVIDER=//p' "$CONFIG_DIR/node.env" | head -n 1)
if [ "$provider" = "filesystem" ]; then
root=$(sed -n 's/^BLOB_FS_ROOT=//p' "$CONFIG_DIR/node.env" | head -n 1)
if [ -n "$root" ] && [ -d "$root" ]; then
mounts="$mounts -v $root:$root"
fi
[ "$(blob_provider)" = "filesystem" ] || { printf '%s' "$mounts"; return 0; }
root=$(blob_root)
[ -n "$root" ] || { printf '%s' "$mounts"; return 0; }
if [ ! -d "$root" ]; then
mkdir -p "$root" 2>/dev/null || true
chown 1000:1000 "$root" 2>/dev/null || true
fi
[ -d "$root" ] && mounts="$mounts -v $root:$root"
printf '%s' "$mounts"
}
blob_provider() {
sed -n 's/^BLOB_PROVIDER=//p' "$CONFIG_DIR/node.env" | head -n 1
}
blob_root() {
sed -n 's/^BLOB_FS_ROOT=//p' "$CONFIG_DIR/node.env" | head -n 1
}
# warn_shared_blobs is loud on purpose. A node on filesystem blobs either has
# its own copy (and cannot read the bodies the backend asked it to send) or
# shares a directory it has no permission on. Both fail at send time, long
# after this script has printed "Done".
warn_shared_blobs() {
[ "$(blob_provider)" = "filesystem" ] || return 0
root=$(blob_root)
warn ""
warn "WARNING: this instance stores blobs on local disk (BLOB_PROVIDER=filesystem,"
warn " BLOB_FS_ROOT=$root)."
warn ""
warn " A node needs the SAME storage the backend writes to, with"
warn " permissions it can read. That only holds when the node shares a"
warn " filesystem with the backend and the ids line up. Otherwise sends"
warn " fail when the worker cannot read the message body."
warn ""
warn " Set BLOB_PROVIDER=s3 on the backend before running nodes off-host,"
warn " then re-run this command."
warn ""
}
install_units() {
[ "$DRY_RUN" = "false" ] || return 0
@@ -372,6 +408,7 @@ main() {
write_config
install_units
start_node
warn_shared_blobs
return 0
}