starred/warmbly
1
0
Fork 0
mirror of https://github.com/warmbly/warmbly.git synced 2026-09-12 16:04:25 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,465 Commits 4 Branches 14 Tags
fix/issue-436
Commit Graph
38 Commits
Author SHA1 Message Date
Matthew Meszaros eb21d978a8 feat: stop aws-bootstrap from treating a bucket that exists in another region as done, since head-bucket answers globally and a second region silently kept its blobs in the first, stop it printing the database master password to stdout where a terminal, a CI log or an agent transcript keeps it forever, and add an instance check that reports a tracking domain sharing a registered domain with the product's own URLs 2026-09-11 05:24:57 +02:00
Matthew Meszaros bbe9d9055a feat: let web and admin be served from a static host by teaching each app's own entrypoint to render config.js wherever WARMBLY_CONFIG_OUT points, so one definition of the runtime key set serves both the container that renders it at start and a build:pages script that renders it into dist, ship a _redirects in each so a deep link stops 404ing without nginx try_files, and add scripts/check-pages-build.sh to make lint because a malformed config.js reads fine in a diff and leaves the app blank at runtime 2026-09-10 18:03:00 +02:00
Matthew Meszaros 5ef8b608c6 feat: record the split-cloud scripts and aws-bootstrap.sh as executable, which a checkout on a filesystem that does not preserve the mode had silently dropped so make lint died with Permission denied in CI only, and assert the recorded git mode in check-split-cloud so it cannot ship non-executable again 2026-09-10 17:30:16 +02:00
Matthew Meszaros 9e689e6d91 feat: fix the three defects that stopped the split-deployment bus bundle from starting at all (NATS given store_dir twice on the command line and in nats.conf, which it refuses rather than reconciling; Redis unable to read its own private key because the image's entrypoint drops from root with gosu and discards the added group; and a healthcheck on localhost when the monitor binds IPv4 loopback, leaving the service unhealthy while serving fine), add bus/setup.sh so standing the box up is one idempotent command that verifies with real clients before printing the control plane's URLs, and add scripts/check-split-cloud.sh to make lint so a bundle that cannot start fails CI 2026-09-10 17:22:14 +02:00
Matthew Meszaros 510ee692ba feat: address the review on the split-deployment branch by moving the two broker routes onto their own NODE_BROKER_TOKEN so the internet-facing tracking and forms services no longer hold a credential that can open any organization's data key, refusing to presign any key outside the prefixes a node reaches, fixing IAM policies that named an alias ARN KMS never resolves in a Resource element, bounding both brokered HTTP clients because the sync loop's context never expires, no longer reporting a 403 from the object store as a missing body, and redacting the DSN and URL credentials the dry-run listing printed in clear 2026-09-10 14:19:53 +02:00
Matthew Meszaros 47ba13083e feat: make a split deployment work end to end by fixing the three defects that made an off-host node impossible to configure (nodeEnvKeys shipped S3_BUCKET and KMS_KEY_ID, which nothing reads, so an AWS-backed node silently used the default bucket and key alias; a joined consumer never received PRIMARY_DB and died at boot; and node.env was rewritten on every join with no file an operator could add to), then removing the need for cloud credentials on a node at all with brokered KMS and blob providers that renderNodeEnv hands out automatically, plus deploy/split-cloud, scripts/aws-bootstrap.sh, two fleet instance checks and the docs 2026-09-10 13:58:59 +02:00
Matthew Meszaros 92b811773e feat: fail the build when a Dockerfile COPY names a path that is not in the repository, because build-push.yml runs only on push to main and nothing on a pull request builds an image, so a COPY of a file the same change deleted is green everywhere until it breaks main 2026-09-09 09:03:54 -07:00
Matthew Meszaros 18a8c7b009 feat: give a self-hosted instance on-demand TLS for customer tracking and forms domains, gating Caddy's ask on a /tls/authorize endpoint that answers only for domains this instance has verified, so a workspace CNAME stops serving every tracked link and opt-out link with no certificate 2026-09-09 08:56:08 -07:00
Matthew Meszaros 50545a0c39 feat: name the join-script check's fs-blob environment for what it is instead of implying a list it cannot iterate, and force NODE_ENV empty for the default render so a value exported elsewhere in the repo cannot decide which unit the check asserts on 2026-09-09 07:18:32 -07:00
Matthew Meszaros 0fd64db51d feat: define the join-script render variants once so a new one extends every per-render assertion instead of only the first, and move the checker's own parse check out of the shellcheck branch where it was skipped exactly when shellcheck was missing and ran under whatever shell happened to be sh 2026-09-09 07:15:08 -07:00
Matthew Meszaros 031c1dcd0a feat: check every rendered unit variant for a node-writable EnvironmentFile rather than only the default one, because dropping it as redundant is the reasoning that already lost this guard once and the assertion exists for changes nobody has made yet, and parse-check the checker itself alongside shellchecking it 2026-09-09 07:09:07 -07:00
Matthew Meszaros a5993a1ed0 feat: put the standalone-statement note where the docs said it was, shellcheck the join-script checker as well since its own disable directives are load-bearing, and stop claiming the EnvironmentFile assertion covers two render variants when only the mount list varies with the environment 2026-09-09 07:04:58 -07:00
Matthew Meszaros 9203695a41 feat: harden the join-script guard against being fooled rather than against being reformatted, matching the ensure_blob_root call on its first field after a looser regex proved satisfiable by the name appearing inside a warn string, capturing the function body without a pipeline so a renamed function reports that instead of the assertion it happened to fail, and rendering both unit variants so the no-blob one is covered too 2026-09-09 06:59:14 -07:00
Matthew Meszaros 9f804c8662 feat: restore the assertion that no EnvironmentFile points into the node-writable mount, which the previous commit deleted as redundant when it is not since the surviving check only proves the correct line exists and not that a second one is absent, and match the ensure_blob_root call in command position after stripping comments so a one-liner reformat no longer fails the build 2026-09-09 06:54:17 -07:00
Matthew Meszaros eb4e1a8bb8 feat: make the join-script checker survive its own mutation tests, matching the ensure_blob_root call on the command field so a commented-out call no longer passes, tolerating a space before the parentheses in a function definition, and capturing the function body before asserting so a renamed function reports that rather than the assertion it happened to fail 2026-09-09 06:47:56 -07:00
Matthew Meszaros 6183e8b238 feat: close the join-script checker's blind spots by pinning the image reference outside the node-writable mount, asserting at their call sites the two invariants that leave no trace in the rendered unit, and matching call lines rather than any line mentioning the word so a comment containing enrolment cannot satisfy the ordering check 2026-09-09 06:42:56 -07:00
Matthew Meszaros 6947aca9b3 feat: make the join-script check assert on what the script renders via a new --print-unit mode, after the previous version compared a heredoc copied into the checker and stayed green when the systemd command-substitution bug was put back, and make docker_mounts pure so the unit can be rendered without creating directories 2026-09-09 06:37:56 -07:00
Matthew Meszaros 29bf9b4320 feat: cover the join script with make join-check, wired into make lint, since nothing tested the highest-consequence non-Go file in the repo and three defects reached the branch through it; and fix the four the review found in the last round, matching the fs provider alias, creating blob parents under a 0022 umask rather than the 0700 one write_config leaves set, warning instead of silently mounting a blob root the node cannot write, and validating the path right after enrolment rather than halfway through the install 2026-09-09 06:28:30 -07:00
Matthew Meszaros 435dbb522f feat: replace the worker tier/type/risk-pool/egress categories with a scored placement model and make the fleet pull-based, so a machine joins with one command, workers and consumers share one node registry with usage and liveness, nodes self-update to the version the control plane resolves, and the Hetzner provisioning, worker profiles and SSH orchestrator are removed 2026-09-09 04:54:01 -07:00
Matthew Meszaros c4bfbaf4a9 feat: address the review on the image publicity gate by passing build-push only the seven services that workflow actually publishes, since web, admin and cli have no :dev tag and would have warned falsely on every push to main, by taking the release tag through the step env instead of interpolating github.ref_name into the shell, and by no longer asserting the tag is fine when a pull is refused, because GHCR denies an unknown namespace exactly as it denies a private one, so a mistyped --registry now gets its own reading in the installer message, the troubleshooting table and the fork note, which also splits the personal and organization paths to the visibility setting 2026-09-07 09:02:23 -07:00
Matthew Meszaros 719a81866c feat: gate the release on every published image being pullable with no credentials, because GHCR creates each package private and does not inherit the repository's visibility, so the authenticated imagetools inspect in create-release passed for four releases while ghcr.io/warmbly/warmbly/* returned unauthorized to everyone outside the org and no curl | sh self-host install could pull a byte (#371); adds scripts/check-images-public.sh as the anonymous pull test, makes it the gate before create-release and the source of the digests in images.json, warns from build-push on main where a new service image first appears, and teaches install.sh to report a registry refusal as one instead of as a missing tag 2026-09-07 08:51:57 -07:00
Matthew Meszaros c7da25d126 fix: never leave a per-IP rate-limit counter without a TTL, which would block that address forever once it passed the limit, by dropping the key and failing open when EXPIRE fails and repairing a missing expiry on the reject path, and assert the exit status as well as the message when the CLI installer rejects a flag 2026-09-04 21:12:27 -07:00
Matthew Meszaros b35e111705 fix: address the review on the CLI PR: give the sign-in handshake its own per-IP budget so a 200-poll login cannot lock the address out of the browser login, keep https for a remote host that names a port instead of sending a bearer token in cleartext, report truncation when a paginated walk stops at max-pages, accept a piped secret with no trailing newline, normalise WEBSOCKET_URL on its suffix so a bare /socket becomes a real endpoint, destroy a minted secret the moment its code expires, gate cli-installer-ci on the required status check, print a rejected flag instead of an unbound-variable error, and use a portable sha256 so the packaging runs on macOS 2026-09-04 21:03:58 -07:00
Matthew Meszaros 4e94d2b47b feat: install the CLI in one command from warmbly.com/cli.sh or cli.ps1, with checksum-verified archives, a Homebrew formula and Scoop manifest generated alongside them, a ghcr cli image, and a make cli-check that installs from a local mirror and proves a tampered checksum stops the install 2026-09-04 20:14:38 -07:00
Matthew Meszaros f042e307e5 feat: rewrite the installer's three A && B || C constructs as guard clauses so shellcheck's SC2015 is satisfied on the CI runner's version, and print the shellcheck version in check-installer.sh 2026-09-04 05:53:16 -07:00
Matthew Meszaros d68bbcd2ab feat: add a one-command self-host installer at warmbly.com/install.sh with an interactive data-control wizard, give docker-compose.yml image keys and per-store volume variables, add an image-mode updater, move engagement/form/audit retention into instance settings, and add warmblyctl backup/restore 2026-09-04 05:49:54 -07:00
Matthew Meszaros bb35de0455 feat: address review on self-hosted updates: the updater no longer re-locks its mutex when a job finishes (every job used to deadlock at completion and freeze the status API), the backend caches the updater view so the member version pill, the health checks and the admin poll share one read and an absent updater is reported as not running rather than broken, the bare-metal upgrade builds unprivileged and hands off to a root-owned fixed-path installer that refuses symlinks so sudoers allows one command instead of install/cp/rm/chown/chmod/systemctl/ln, the installer fails when the backend does not come back, the seed image gets the version build args, the dashboard gates the update action on manage_settings and stops polling a backend that answers 404, and revived timestamps are typed as Date 2026-09-03 05:04:30 -07:00
Matthew Meszaros b2ea1f1961 feat: add self-hosted update awareness and one-click updates: every binary is stamped with its version and commit, the backend polls GitHub Releases and a new host-side updater (cmd/updater, compose profile or systemd unit) reports the checkout's commit distance, the admin panel's top bar shows a version pill that turns into an update indicator and opens a dialog with confirmation, live step progress and log, restart tracking and result, the dashboard header shows the same pill to every member of a self-hosted instance with the full update flow for platform admins, Setup and health gains update_available and updater_unreachable checks, warmblyctl status prints the version, make upgrade and scripts/upgrade-bare-metal.sh cover the by-hand paths, and docs gain an Updates page plus configuration, health, deployment and API reference updates 2026-09-03 05:04:30 -07:00
Matthew Meszaros 05d74dbe9f feat: forms v2, a full redesign of the hosted form and its builder: eight one-click themes, card/wide/split layouts with a brand cover panel, classic paged and Typeform-style focus modes split on a new page_break block, logo/cover/background uploads with size, fit and veil controls, an optional header bar that can span the page or sit with the form, a real HSV colour picker and font preview, per-contact personalized links that prefill and attribute without email verification, a render-token gate so the form JSON cannot be scraped without loading the page, funnel analytics with per-page drop-off and identified visitors, a leads-style forms list and responses table, and an organization custom forms domain verified by CNAME with hourly re-checks 2026-09-01 09:27:28 -07:00
Matthew Meszaros 1247c712f3 feat: add make poollink-dev, a two-instance dev environment for the Warmbly Cloud link (a prod-like cloud with DEPLOYMENT_MODE=cloud, Stripe gates on with placeholder keys, Mailpit email verification and the Sunrise Labs pool seeded, next to a fresh unclaimed self-hosted instance whose WARMBLY_CLOUD_URL points at it), with status/setup-link/down/reset subcommands and a local-development docs section 2026-08-29 07:14:04 -07:00
Matthew Meszaros 7c2d08f204 feat: renumber the campaign_send_outcome migration to 000091 so it stops colliding with 000089_email_auth_enforcement, which made golang-migrate refuse to init its source driver and left the backend restart-looping at boot on main, and guard the class of bug with scripts/check-migrations.sh (duplicate versions, gaps, unpaired up/down, unparseable filenames) wired into make check-migrations, make lint, a dedicated Migrations CI job and a TestEmbeddedMigrationsLoad that boots the real iofs source over the embedded FS 2026-08-23 21:26:14 -07:00
Matthew Meszaros 8bd2c2b57a feat: make self-hosting work end to end and rewrite the guide around what was tested (#97) 2026-08-13 09:47:46 +02:00
Matthew Meszaros 2430095197 feat: move infrastructure state off dynamodb 
Remove DynamoDB-backed storage paths, add Postgres/HTTP repositories for mailbox state maps, wire the internal message-map API, and add provisioning runner/migration plumbing.
 2026-06-02 15:54:12 +02:00
Matthew Meszaros b168f4d466 feat: simplify worker installer route 2026-05-30 09:37:26 +00:00
Matthew Meszaros d8a546eca5 feat: add worker enrollment install 2026-05-30 05:10:27 +00:00
Matthew Meszaros 5463e3986e worker: multi-IP support (one systemd unit per Primary IP) 
Hetzner CX32 + 16 Primary IPs becomes 16 sending identities with one
install command, without expanding ops complexity.

cmd/worker/main.go: WORKER_ID now resolves via 4-tier precedence:
  1. WORKER_ID env (explicit UUID)
  2. WORKER_BIND_IP env (derive UUIDv5 from the bound IP)
  3. hostname-as-UUID (legacy single-IP VPS)
  4. generated UUID (local dev fallback)

Boot also constructs the chosen Codec + EventBus + EncryptedKeyStore
via the FromEnv factories from earlier commits, so a worker process is
fully configured by its envelope env file plus the runtime config it
pulls from the backend on first boot.

scripts/install-worker.sh gains --ips <ipv4,ipv4,...> which:
  - writes a warmbly-worker@.service systemd template
  - drops a per-instance env file at /etc/warmbly/instances/<dashed-ip>.env
    with WORKER_BIND_IP and WORKER_ID
  - shares one /etc/warmbly/worker.env for the common config
  - --status, --update, --uninstall now multi-IP aware
  - single-IP mode preserved when --ips is absent

5 worker tests pin the UUIDv5 derivation against the installer's
uuidgen --sha1 output so the two never drift.

docs/MULTI_IP_WORKERS.md is the operator runbook with the Hetzner
recipe, OS-level IP attachment, rDNS automation, day-2 ops, and the
25%-of-fleet blast-radius rule.
 2026-05-27 14:43:20 +00:00
Matthew Meszaros d25eed3eb6 feat(dev): root docker-compose with profiles, LocalStack, richer seed 
Hoist the dev/sim stack to a single docker-compose.yml at the repo root.
Adds profiles (default / sim / seed / tools) so you can opt into heavier
setups, and bundles dependencies that were previously missing:

- LocalStack (KMS + DynamoDB + S3) with a localstack-init one-shot that
  idempotently creates alias/master-key-dev, the UserEncryptedKeys and
  EmailMessageData tables, and the main S3 bucket. Backend and workers
  wait on it via service_completed_successfully.
- stripe-mock for billing flows
- kafka-ui under the tools profile

Three workers with deterministic UUIDv5 hostnames (shared / premium /
dedicated) so assignment, rebalancing, and per-pool routing all have
real targets to exercise.

Richer seed (cmd/seed/main.go) loads 3 orgs across tiers, 6 mailboxes
joined to free/premium warmup pools, a Beta campaign with a 2-step
sequence, and 10 contacts (2 unsubscribed) so suppression behaviour is
visible in the UI. Idempotent — safe to re-run.

Makefile targets:
  make dev    — infra + app + one worker
  make sim    — adds premium + dedicated workers
  make seed   — rich fixtures
  make tools  — kafka-ui at :18090
  make reset  — nuke volumes
 2026-05-18 13:08:34 +00:00
Matthew Meszaros 89e5533dd0 feat(worker): add one-command VPS installer with IP-derived identity 
scripts/install-worker.sh is a single bash script any Debian/Ubuntu/RHEL/
Fedora/Arch/Alpine VPS can curl|sh to add a worker to the fleet.

Identity is bound to the VPS's public IPv4 via UUIDv5 (URL namespace):

  same IP  → same worker  (reputation persists across reinstalls)
  new IP   → new worker   (fresh identity, no inherited reputation)

The installer detects the public IP via api.ipify.org / ifconfig.me /
checkip.amazonaws.com, derives the deterministic UUID, installs Docker if
missing, writes /etc/warmbly/worker.env (0600) and /etc/warmbly/worker.id,
installs a systemd unit that runs the worker container with --hostname
<uuid>, and starts the service.

Supports --install/--update/--uninstall/--purge/--status, --env-file for
non-interactive config, --ip override, --image override, and a full set
of per-credential flags.

Worker reads its UUID from os.Hostname() at startup, so the systemd
hostname value becomes the worker identity — no separate registration
step needed.
 2026-05-18 13:08:16 +00:00