mirror of
https://github.com/warmbly/warmbly.git
synced 2026-09-13 00:03:42 +00:00
feat: always create and mount BLOB_FS_ROOT for a node because its storage layer exits when the directory is not writable and an unmounted path is root-owned by docker, and say plainly in the script and the docs that filesystem blobs cannot serve a fleet at all since the worker has to read the body the backend wrote
This commit is contained in:
@@ -559,9 +559,9 @@ tls {
|
||||
}
|
||||
```
|
||||
|
||||
Then set `NATS_URL=tls://<token>@nats.yourdomain.com:4222` in `warmbly.env` and `worker.env`; the token in the URL is what every service authenticates with. Redis gets `requirepass` and `REDIS=redis://:<password>@redis.yourdomain.com:6379`, ideally over a private network or a tunnel. Restrict both ports at the firewall to the worker's address as well. Use `BLOB_PROVIDER=s3` with a bucket both sides can reach; a remote worker on the filesystem provider writes to its own disk. The [self-hosting guide](/development/deployment-guide/#remote-workers) has the same caveats in more detail.
|
||||
Then set `NATS_URL=tls://<token>@nats.yourdomain.com:4222` in `warmbly.env` and `worker.env`; the token in the URL is what every service authenticates with. Redis gets `requirepass` and `REDIS=redis://:<password>@redis.yourdomain.com:6379`, ideally over a private network or a tunnel. Restrict both ports at the firewall to the worker's address as well. Use `BLOB_PROVIDER=s3` with a bucket both sides can reach: a worker reads the message body the backend wrote, so the filesystem provider only works when the two share storage and the permissions line up. The [self-hosting guide](/development/deployment-guide/#adding-a-machine) has the same caveats in more detail.
|
||||
|
||||
Because the worker is not in a container, the admin panel's SSH-driven day-two actions (pull image, restart container) do not apply to it. Manage it with `systemctl` and the update steps below.
|
||||
Because the worker is not in a container, the join script's systemd service and update timer do not apply to it. Manage it with `systemctl` and the update steps below, and set its version by hand rather than through `warmblyctl fleet version`.
|
||||
|
||||
## Upgrading
|
||||
|
||||
|
||||
@@ -725,7 +725,7 @@ The config handed to a node is generated from the backend's own environment. On
|
||||
|
||||
That config carries the decryption material the node needs: the internal API token, `KMS_LOCAL_MASTER_KEY`, and `CREDENTIALS_ENCRYPTION_KEY`. Serve the API over HTTPS before adding a node across a network you do not control. It deliberately does not include `PRIMARY_DB`: a worker reaches relational data through the internal API and nothing else.
|
||||
|
||||
On `BLOB_PROVIDER=filesystem`, a remote node writes blobs to its own local disk rather than a volume the backend shares. Use `BLOB_PROVIDER=s3` with a bucket both sides can reach when you run nodes off-host.
|
||||
`BLOB_PROVIDER=filesystem` does not survive a fleet. A worker reads the message body the backend wrote, so the two need the same storage with permissions that let both reach it, and a node on another machine has neither. The join script creates and mounts `BLOB_FS_ROOT` so the node starts, and warns you, but sends will fail when the worker cannot read the body. Use `BLOB_PROVIDER=s3` with a bucket both sides can reach before running nodes off-host.
|
||||
|
||||
<Mermaid
|
||||
chart={`
|
||||
|
||||
@@ -222,26 +222,62 @@ write_config() {
|
||||
}
|
||||
|
||||
# docker_mounts is every -v argument, on ONE line. Command substitution strips
|
||||
# trailing newlines, so a multi-line value here would collapse the unit's
|
||||
# trailing newlines, so a multi-line value would collapse the unit's
|
||||
# continuations and hand docker a stray token as the image name.
|
||||
#
|
||||
# BLOB_FS_ROOT is mounted only when it already exists: it is copied from the
|
||||
# backend's environment and may name a path that belongs to a co-located
|
||||
# bare-metal install, so this neither creates it nor changes its ownership. A
|
||||
# node on another host writes blobs to its own disk regardless, which is why
|
||||
# the docs tell you to use object storage for a distributed fleet.
|
||||
# BLOB_FS_ROOT needs a mount whatever happens: the node's storage layer does
|
||||
# MkdirAll on it at boot and the process exits if that fails, so leaving the
|
||||
# path unmounted means docker creates it root-owned and the container, running
|
||||
# as uid 1000, restart-loops. Creating and owning it here is the only branch
|
||||
# that reliably starts.
|
||||
#
|
||||
# Sharing it with a co-located backend is a different problem and not one this
|
||||
# script can solve: the backend writes objects 0600 as its own system user, so
|
||||
# a node cannot read them without matching ids. warn_shared_blobs says so
|
||||
# rather than pretending otherwise; object storage is the supported shape for a
|
||||
# fleet.
|
||||
docker_mounts() {
|
||||
mounts="-v $AGENT_DIR:$AGENT_DIR"
|
||||
provider=$(sed -n 's/^BLOB_PROVIDER=//p' "$CONFIG_DIR/node.env" | head -n 1)
|
||||
if [ "$provider" = "filesystem" ]; then
|
||||
root=$(sed -n 's/^BLOB_FS_ROOT=//p' "$CONFIG_DIR/node.env" | head -n 1)
|
||||
if [ -n "$root" ] && [ -d "$root" ]; then
|
||||
mounts="$mounts -v $root:$root"
|
||||
fi
|
||||
[ "$(blob_provider)" = "filesystem" ] || { printf '%s' "$mounts"; return 0; }
|
||||
root=$(blob_root)
|
||||
[ -n "$root" ] || { printf '%s' "$mounts"; return 0; }
|
||||
if [ ! -d "$root" ]; then
|
||||
mkdir -p "$root" 2>/dev/null || true
|
||||
chown 1000:1000 "$root" 2>/dev/null || true
|
||||
fi
|
||||
[ -d "$root" ] && mounts="$mounts -v $root:$root"
|
||||
printf '%s' "$mounts"
|
||||
}
|
||||
|
||||
blob_provider() {
|
||||
sed -n 's/^BLOB_PROVIDER=//p' "$CONFIG_DIR/node.env" | head -n 1
|
||||
}
|
||||
|
||||
blob_root() {
|
||||
sed -n 's/^BLOB_FS_ROOT=//p' "$CONFIG_DIR/node.env" | head -n 1
|
||||
}
|
||||
|
||||
# warn_shared_blobs is loud on purpose. A node on filesystem blobs either has
|
||||
# its own copy (and cannot read the bodies the backend asked it to send) or
|
||||
# shares a directory it has no permission on. Both fail at send time, long
|
||||
# after this script has printed "Done".
|
||||
warn_shared_blobs() {
|
||||
[ "$(blob_provider)" = "filesystem" ] || return 0
|
||||
root=$(blob_root)
|
||||
warn ""
|
||||
warn "WARNING: this instance stores blobs on local disk (BLOB_PROVIDER=filesystem,"
|
||||
warn " BLOB_FS_ROOT=$root)."
|
||||
warn ""
|
||||
warn " A node needs the SAME storage the backend writes to, with"
|
||||
warn " permissions it can read. That only holds when the node shares a"
|
||||
warn " filesystem with the backend and the ids line up. Otherwise sends"
|
||||
warn " fail when the worker cannot read the message body."
|
||||
warn ""
|
||||
warn " Set BLOB_PROVIDER=s3 on the backend before running nodes off-host,"
|
||||
warn " then re-run this command."
|
||||
warn ""
|
||||
}
|
||||
|
||||
install_units() {
|
||||
[ "$DRY_RUN" = "false" ] || return 0
|
||||
|
||||
@@ -372,6 +408,7 @@ main() {
|
||||
write_config
|
||||
install_units
|
||||
start_node
|
||||
warn_shared_blobs
|
||||
return 0
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user