diff --git a/docs/content/docs/development/bare-metal.mdx b/docs/content/docs/development/bare-metal.mdx index 188119d0..4a9a7c02 100644 --- a/docs/content/docs/development/bare-metal.mdx +++ b/docs/content/docs/development/bare-metal.mdx @@ -559,9 +559,9 @@ tls { } ``` -Then set `NATS_URL=tls://@nats.yourdomain.com:4222` in `warmbly.env` and `worker.env`; the token in the URL is what every service authenticates with. Redis gets `requirepass` and `REDIS=redis://:@redis.yourdomain.com:6379`, ideally over a private network or a tunnel. Restrict both ports at the firewall to the worker's address as well. Use `BLOB_PROVIDER=s3` with a bucket both sides can reach; a remote worker on the filesystem provider writes to its own disk. The [self-hosting guide](/development/deployment-guide/#remote-workers) has the same caveats in more detail. +Then set `NATS_URL=tls://@nats.yourdomain.com:4222` in `warmbly.env` and `worker.env`; the token in the URL is what every service authenticates with. Redis gets `requirepass` and `REDIS=redis://:@redis.yourdomain.com:6379`, ideally over a private network or a tunnel. Restrict both ports at the firewall to the worker's address as well. Use `BLOB_PROVIDER=s3` with a bucket both sides can reach: a worker reads the message body the backend wrote, so the filesystem provider only works when the two share storage and the permissions line up. The [self-hosting guide](/development/deployment-guide/#adding-a-machine) has the same caveats in more detail. -Because the worker is not in a container, the admin panel's SSH-driven day-two actions (pull image, restart container) do not apply to it. Manage it with `systemctl` and the update steps below. +Because the worker is not in a container, the join script's systemd service and update timer do not apply to it. Manage it with `systemctl` and the update steps below, and set its version by hand rather than through `warmblyctl fleet version`. ## Upgrading diff --git a/docs/content/docs/development/deployment-guide.mdx b/docs/content/docs/development/deployment-guide.mdx index e019b43d..7ec1cab1 100644 --- a/docs/content/docs/development/deployment-guide.mdx +++ b/docs/content/docs/development/deployment-guide.mdx @@ -725,7 +725,7 @@ The config handed to a node is generated from the backend's own environment. On That config carries the decryption material the node needs: the internal API token, `KMS_LOCAL_MASTER_KEY`, and `CREDENTIALS_ENCRYPTION_KEY`. Serve the API over HTTPS before adding a node across a network you do not control. It deliberately does not include `PRIMARY_DB`: a worker reaches relational data through the internal API and nothing else. -On `BLOB_PROVIDER=filesystem`, a remote node writes blobs to its own local disk rather than a volume the backend shares. Use `BLOB_PROVIDER=s3` with a bucket both sides can reach when you run nodes off-host. +`BLOB_PROVIDER=filesystem` does not survive a fleet. A worker reads the message body the backend wrote, so the two need the same storage with permissions that let both reach it, and a node on another machine has neither. The join script creates and mounts `BLOB_FS_ROOT` so the node starts, and warns you, but sends will fail when the worker cannot read the body. Use `BLOB_PROVIDER=s3` with a bucket both sides can reach before running nodes off-host. /dev/null || true + chown 1000:1000 "$root" 2>/dev/null || true fi + [ -d "$root" ] && mounts="$mounts -v $root:$root" printf '%s' "$mounts" } +blob_provider() { + sed -n 's/^BLOB_PROVIDER=//p' "$CONFIG_DIR/node.env" | head -n 1 +} + +blob_root() { + sed -n 's/^BLOB_FS_ROOT=//p' "$CONFIG_DIR/node.env" | head -n 1 +} + +# warn_shared_blobs is loud on purpose. A node on filesystem blobs either has +# its own copy (and cannot read the bodies the backend asked it to send) or +# shares a directory it has no permission on. Both fail at send time, long +# after this script has printed "Done". +warn_shared_blobs() { + [ "$(blob_provider)" = "filesystem" ] || return 0 + root=$(blob_root) + warn "" + warn "WARNING: this instance stores blobs on local disk (BLOB_PROVIDER=filesystem," + warn " BLOB_FS_ROOT=$root)." + warn "" + warn " A node needs the SAME storage the backend writes to, with" + warn " permissions it can read. That only holds when the node shares a" + warn " filesystem with the backend and the ids line up. Otherwise sends" + warn " fail when the worker cannot read the message body." + warn "" + warn " Set BLOB_PROVIDER=s3 on the backend before running nodes off-host," + warn " then re-run this command." + warn "" +} + install_units() { [ "$DRY_RUN" = "false" ] || return 0 @@ -372,6 +408,7 @@ main() { write_config install_units start_node + warn_shared_blobs return 0 }