Commit Graph
27 Commits
Author SHA1 Message Date
Matthew Meszaros cbf4190f42 feat: make a custom tracking domain verifiable instead of permanently "Pending DNS": the CNAME target is now this install's TRACKING_DOMAIN rather than a hardcoded t.warmbly.com that resolves nowhere, matching is exact on the label boundary (or on shared addresses, so a provider-flattened CNAME stops reading as no record at all) instead of strings.Contains, and every outcome carries the reason plus what DNS actually returned, including when the tracking host the customer is told to point at has no record of its own; a pasted URL is normalized to its host and a malformed one is rejected up front instead of saved and left pending forever; only a VERIFIED mailbox domain is used at send time with the shared host as the fallback and a campaign-feed entry saying why; pixels and click tickets are built from the configured host, and with none configured mail ships untracked rather than carrying links to another deployment's tracking service; adds GET /emails/:id/track and POST /emails/:id/track/verify plus an hourly re-resolution sweep so a record that propagates later starts being used and one that breaks stops routing links; and scopes the tracking-domain write by organization like the read, which also fixes GET /emails/:id passing a user id to an org-scoped query and 404ing for every caller 2026-08-24 09:02:29 -07:00
Matthew Meszaros f0846eb034 feat: stop a freshly connected mailbox being silently excluded from every campaign send, by making an unset mailbox timezone representable as the empty string the campaign scheduler already checks for, since email_accounts.timezone defaulted to 'UTC' while campaigns.timezone defaults to 'Europe/London' and nothing in the OAuth or SMTP onboarding paths ever set either, so a brand new mailbox looked deliberately placed in UTC, was compared against the differing campaign zone and dropped by the hardcoded 8am-8pm business-hours gate whenever the current UTC hour fell outside it, emptying the candidate pool and failing the campaign start, adding a migration that changes the column default and converts existing 'UTC' rows because until now no API field, dashboard control or onboarding path could set that column at all so every such row is the old default rather than a choice, adding the missing Timezone field to UpdateEmail with IANA validation so the setting the sending-behaviour UI already tells people to change is finally reachable and an unloadable zone is rejected instead of being silently coerced to UTC by the scheduler, and replacing the misleading 'no active email accounts found for campaign's email tags' response for a pool that exists but is entirely gated out with a distinct message naming the real cause, via an ErrNoEligibleMailbox that wraps ErrNoEmailAccounts so the three callers that pause a campaign on it are unaffected (#126) (#125) 2026-08-16 07:54:45 +02:00
Matthew Meszaros 734cb5fe08 feat: make self-hosted onboarding survivable by fixing invite_only, which could not onboard anyone (the accept route is JWT-only, so redeeming the invitation that would create your account required already having one, making the self-host default silently identical to fully closed), threading the invitation token through registration so an invited person lands in the inviting organization instead of a stray workspace, gating SSO just-in-time provisioning behind DISABLE_REGISTRATION (it bypassed the gate entirely, so an instance set to true was still open to anyone the IdP would assert) with SSO_AUTO_PROVISION as the opt-out, correcting the OIDC redirect URL that pointed at /api/v1 against a route at /v1 and 404'd every SSO login, scoping the first-launch exemption so it no longer overrides an explicit lockdown, preserving the remaining TTL when restoring a losing setup token so a public endpoint cannot hold the claim window open forever, replacing a generic 403 with typed registration_invite_only, registration_closed, invitation_invalid, setup_token_invalid and setup_already_complete codes that name the next step, logging why no claim link was issued on an already-claimed instance instead of staying silent, adding a warmblyctl operator CLI (status with health checks and a non-zero exit, reissuable setup-link, user create/list/reset-password/grant-admin/revoke-admin/disable-2fa, hash-password) so a locked-out operator no longer needs hand-written psql, adding read-only instance configuration over 104 environment variables with structural secret redaction and fingerprints, 35 health checks, a database-backed settings tier for the three keys no environment variable owns, hiding the signup form when the config already says invite_only rather than failing the whole form with a toast, and documenting first run, accounts and access, configuration, instance health and troubleshooting alongside the root .env.example the README told operators to write but never shipped (#114) 2026-08-16 05:58:11 +02:00
Matthew Meszaros 0ae4db2c41 feat: make self-hosted auth work without a mail relay by rewriting the platform SMTP transport with real AUTH and TLS (it did neither, so SMTP_USERNAME/SMTP_PASSWORD were dead and every documented relay was unreachable), adding MAIL_TRANSPORT=smtp|log|ses with a log transport that prints codes so a fresh install can sign in with no relay, demoting the emailed login code to AUTH_LOGIN_CODE=always|new_device|off (off on self-host, per NIST SP 800-63B and OWASP ASVS), claiming the first owner through a single-use setup link or WARMBLY_BOOTSTRAP_* instead of register-then-psql, deriving every emailed URL from APP_URL rather than a hardcoded app.warmbly.com that leaked live reset tokens to the vendor, fixing the confirm hooks that read path params against paramless routes and broke login, register and reset confirmation in the dashboard everywhere, adding generic OIDC with PKCE, one-time state, verified nonce and (issuer,subject) identity binding, enforcing 2FA on the social paths that skipped it, adding a per-IP limiter and trusted-proxy handling to the unthrottled auth group, refusing boot on the published default secrets, and dropping mailpit from the default stack (#99) 2026-08-14 14:57:09 +02:00
Matthew Meszaros 8bd2c2b57a feat: make self-hosting work end to end and rewrite the guide around what was tested (#97) 2026-08-13 09:47:46 +02:00
Matthew Meszaros a7518a8558 docs: refresh the documentation site, fix inaccurate claims and contact addresses, add SEO primitives (#90)
* feat: rewrite the self-hosting docs against repo ground truth: turn the deployment guide into a full self-host guide (quick start with first-admin bootstrap via make grant-admin, .env secrets with exact key formats, PUBLIC_HOST derivation and HTTPS reverse-proxy vars, provider switches with build-tag caveats, mailbox OAuth, remote worker enrollment via SSH or wmenroll tokens, real CI image tags, upgrades and backups), rewrite the events page around the real NATS/Kafka bus topics and {type,body} envelopes, fix Kafka-era and make-target claims in architecture/local-development/deploy README, add API_PUBLIC_URL and drop the dead LOG_DISCORD_WEBHOOK_URL in env.example, and remove the docker-compose.kafka.yml comment pointing at a file that does not exist

* feat: make the self-hosting docs visual and skimmable by adding a Mermaid MDX component (client-rendered, theme-aware) to the docs site, condensing the self-host guide around a control-plane topology diagram, a worker enrollment sequence diagram, a dashboard screenshot, and symptom/check troubleshooting + optional-subsystem tables, and adding an execution-plane flowchart to the architecture page

* feat: stop the docs root flashing a 'Continue to the Warmbly docs' link before redirecting by navigating with an inline location.replace that runs during HTML parse, and demoting the visible link and meta refresh to no-JS fallbacks inside noscript

* feat: cut docs bulk and duplication by deleting three orphaned API pages that were stale forks of the reference section and were unreachable from the sidebar (porting their unique social sign-in, promo-code, and referral endpoints into api/reference/account-org.mdx as compact tables), condensing the deliverability and warmup guides to roughly half their length around tables instead of prose, replacing prose em dashes across the guides and MCP pages, and adding the required trailing slashes to internal links in 24 files

* feat: condense the sequences guide by about 40 percent, folding the switch-step deciders and branch conditions into tables and cutting restated prose while keeping every rule about threading, instant branches, reply matching, and stop on reply

* feat: condense the automations, unibox, advisor, and expressions guides by roughly 40 percent each, folding trigger lists, action catalogs, sending controls, and advisor checks into tables, adding a trigger-condition-action flow diagram to automations, and cutting restated prose while preserving every threshold, permission boundary, and rule

* feat: condense the mailboxes, campaigns, analytics, and team-roles guides by roughly 45 percent each, replacing prose walks through providers, rotation modes, lead statuses, counting rules, A/B confidence, and the permission matrix with compact tables and collapsing the four-way role grid into one capability table plus a one-line mapping

* feat: condense the AI-steps, security, and contacts-CRM guides by roughly 40 percent, turning sign-in methods, AI step modes, switch deciders, credit and failure behavior, import field mappings, and deal views into tables while keeping every safety boundary and dedupe rule

* feat: condense the meetings, notifications, AI-credits, and AI-assistant guides by roughly 40 percent, merging notification categories and their defaults into one table, collapsing credit costs, spend controls, and plan allowances into tables, and tightening the assistant page around its approval and permission boundaries

* feat: condense the integrations, collaboration, zapier, and make guides by roughly 35 percent, grouping the thirty-row Zapier and Make action lists into eight labelled areas, folding CRM default field mappings and presence indicators into tables, and promoting the destructive-action and unattended-delete warnings into callouts

* fix: correct three factual errors in the development docs: NOTIFICATION_EMAIL_DAILY_CAP=0 means uncapped rather than disabled (overEmailBudget returns false at limit<=0, so documenting it as a kill switch inverted the behavior), and the worker-SSH and warmup-pool migration citations in architecture.mdx pointed at pre-squash filenames that no longer exist or now belong to unrelated migrations, so both now cite the tables in 000001_baseline.up.sql

* feat: add the missing docs SEO primitives: a build-time sitemap.xml covering all 64 pages, a robots.txt that points at it and keeps the llms.mdx and og mirrors out of the index as duplicate content, and per-page canonical plus richer OpenGraph URL/title/description metadata

* fix: use the single real team@warmbly.com address everywhere a human is told to write in, replacing the invented hello/sales/legal/support inboxes across the marketing site, the transactional email footer, and the admin outreach composer default Reply-To (which pointed replies at a mailbox that does not exist), and collapse the contact page's two-inbox framing into one inbox with one published response time
2026-08-05 10:37:27 +02:00
Matthew Meszaros 79efbd4b5b feat: publish the web and admin images to GHCR in the release workflow and serve those production builds from the prod docker-compose instead of the Vite dev servers 2026-07-22 18:40:06 +02:00
Matthew Meszaros aeaa26d2d2 fix: use a valid local kms dev key and warn when sandbox port 8080 is taken 2026-07-21 17:34:59 +02:00
Matthew Meszaros af048b9d75 feat: restore the dovecot imap host behind a sandbox profile so seeded mailboxes really sync 2026-07-20 11:12:36 +02:00
Matthew Meszaros 9590bd63fb feat: add a no-cloud docker compose stack 2026-07-20 09:56:47 +02:00
Matthew Meszaros 7cfd9b26e9 feat: register the tracking-events Avro schema via a one-shot schema-registry-init infra service - the Rust tracking service encodes fetch-only against Schema Registry, so with no registered schema every open/click event failed serialization and was silently dropped in dev 2026-07-11 19:48:05 +02:00
Matthew Meszaros c8ed667aeb feat: add a dovecot IMAP service to the infra stack and accept any-credential SMTP auth on mailpit so local mailboxes can really send and sync (sandbox mail hosting), plus CODEC_PROVIDER/CREDENTIALS_ENCRYPTION_KEY env for backend, consumer, and workers and a host-gateway alias so the cloud-tasks emulator can call a natively run backend 2026-07-11 17:30:03 +02:00
Matthew Meszaros 956edabca3 feat: kafka no longer fails its first boot after an unclean stop - the container now waits for the stale ephemeral /brokers/ids/1 registration to expire in zookeeper before starting (with a 6s session timeout) instead of crashing on NodeExists and relying on the restart policy 2026-07-11 10:13:44 +02:00
Matthew Meszaros 2430095197 feat: move infrastructure state off dynamodb
Remove DynamoDB-backed storage paths, add Postgres/HTTP repositories for mailbox state maps, wire the internal message-map API, and add provisioning runner/migration plumbing.
2026-06-02 15:54:12 +02:00
Matthew Meszaros 559d05a458 feat: use dot-separated event-bus topic names for Kafka compatibility 2026-06-01 12:49:24 +02:00
Matthew Meszaros adedcc337e feat: support remote dev infra
Allow native dev services to target infrastructure running on another host by parameterizing local endpoints with INFRA_HOST and SELF_HOST.

Document the workflow and make Kafka advertise a configurable host for off-box clients.
2026-05-31 08:53:21 +02:00
Matthew Meszaros bf40834557 feat: update unibox experience 2026-05-30 16:37:50 +00:00
Matthew Meszaros 0740a8bbbe feat: fix local migrations and kafka topics 2026-05-30 14:11:47 +00:00
Matthew Meszaros e463df748f fix: add zookeeper healthcheck so kafka waits for sessions to be ready 2026-05-25 04:32:41 +00:00
Matthew Meszaros 6eb3a3bdfd merge: bring origin/main into branch, layer full seed on top of seedRich 2026-05-24 11:54:49 +00:00
Matthew Meszaros e5c0c8a448 chore(dev): standardize ports where they don't conflict
Most container ports go back to their natural defaults — the offsets
that existed weren't justified, they just made URLs harder to remember.
Now standard:

  backend         8080   (was always 8080)
  tracking        3000   (was 13000)
  realtime        4000   (was 14000)
  web             5173   (was 15173 — already changed)
  kafka           9092   (was 19092)
  schema-registry 8081   (was 18081)
  localstack      4566   (was 14566)
  cloud-tasks     8123   (was 18123)
  stripe-mock     12111  (always was)

Kept offset (the defaults conflict too often on real dev machines):

  postgres        15432  (system postgres / sibling project)
  redis           16379  (sibling docker projects with redis)
  mailpit ui      18025  (sibling docker projects with mailpit)
  mailpit smtp    11025  (same)
  kafka-ui        18090  (8080 already used by backend)

Touched: docker-compose.yml, Makefile (test-seed SEED_TEST_DB), READMEs
(root + deploy), local-development.md + deployment-guide.md. Internal
docker-network refs (kafka:29092, mailpit:1025, etc.) unchanged — only
host-port mappings moved. Compose validated, all default-profile
services come up healthy on the new ports.
2026-05-22 14:51:16 +00:00
Matthew Meszaros 33ea340660 fix(dev): set APP_URL on backend so CORS allows the web origin
The backend's CORS allow-list resolution falls back through:
  1. CORS_ALLOW_ORIGINS
  2. APP_URL
  3. origin derived from WEBSOCKET_URL

In compose only WEBSOCKET_URL was set, so allow-list landed at
http://localhost:14000 — the realtime port, not where the browser is
loaded from. The dev fallback that includes localhost:5173 only fires
when the list is empty, not when it's wrong-by-derivation. Result:
every POST to /auth/login etc. failed preflight with 403, which axios
surfaces as a generic "NetworkError" — login appeared to silently fail
after entering the password.

Setting APP_URL=http://localhost:5173 on the backend service makes the
allow-list match where the Vite dev server actually serves from. Sanity
checked with curl -X OPTIONS — preflight now returns 204.
2026-05-22 04:04:42 +00:00
Matthew Meszaros 34ab7ad266 fix(web): correct Request import depth + move web back to port 5173
The new admin API clients (audit, credentials, workers) imported Request
with four '..' segments instead of three. Vite's import-analysis failed
with "Failed to resolve import ../../../../Request" because that path
resolves to api/Request, not client/Request. tsc didn't catch it because
the resolver was permissive enough to keep going, but the runtime is
strict. Matched the existing pattern from roles/getRoles.ts (three dots
for Request, four for models).

Separately: web was on host port 15173, offset from the canonical 5173
to avoid colliding with a locally-running Vite outside Docker. Nobody
actually runs Vite locally in this setup, and the offset makes the URL
non-obvious. Moved back to 5173:5173 and updated VITE_APP_URL plus the
docs.

If a developer one day wants to run a host-side Vite alongside the
container, change the mapping back to "15173:5173" — the offset is the
escape hatch, not the default.
2026-05-22 03:55:34 +00:00
Matthew Meszaros 5a7fa6c71b fix(web): unbreak pnpm install in docker — drop rolldown-vite, fix retry loop
Four interlocking problems were causing the web container to spin in
its retry loop forever:

1. web/package.json aliased vite to "npm:rolldown-vite@7.1.14".
   rolldown-vite is being deprecated (its own warning told us to use
   7.3.1 for migration, or move to vite 8). vitest 4.x has a transitive
   `vite` dep that pnpm tried to resolve against the public registry,
   where vite@7.1.14 doesn't exist as a release (only 7.3.3 and 8.x).
   Result: ERR_PNPM_NO_MATCHING_VERSION on every retry.

2. The "resolutions" block was meant to force the alias on transitive
   deps. resolutions is Yarn syntax; pnpm doesn't read it. So the alias
   wasn't propagating, which is exactly why (1) blew up.

3. pnpm 11 stopped reading the "pnpm" field in package.json. Settings
   moved to pnpm-workspace.yaml. New file added with allowBuilds.esbuild
   set so pnpm doesn't refuse to compile esbuild's native binary at
   install time (it's transitively pulled in by vite + vitest).

4. The docker-compose web service had `until pnpm install; do echo
   "pnpm install retry..."; sleep 3; done` which spins forever on
   permanent dep-resolution errors and buries the actual message under
   thousands of retries. Replaced with fail-fast that prints a hint
   directing the admin to fix package.json and `make restart web`.

Also deleted the stale pnpm-lock.yaml so pnpm regenerates against the
fresh dep tree. Verified vite v7.3.3 boots, esbuild postinstall runs,
and Vite serves on http://localhost:15173 cleanly.

vite.config.ts has no rolldown-specific config, so the move from
rolldown-vite to plain vite is a no-op behaviourally.
2026-05-22 03:52:49 +00:00
Matthew Meszaros f95a12f2a1 Revert "feat(dev): hot reload for Go services in docker-compose"
This reverts commit 3a84e33155.
2026-05-22 03:37:34 +00:00
Matthew Meszaros 3a84e33155 feat(dev): hot reload for Go services in docker-compose
Until now, only the web service hot-reloaded (Vite HMR via the
node:22-alpine container + ./web mount). The Go services (backend,
consumer, worker) used their production multi-stage Dockerfiles, so
every code change meant `docker compose build <svc> && docker compose
up -d <svc>` — ~30s per service.

Switched all Go services to a shared dev image (go.dev.Dockerfile)
that ships:
  - full Go 1.25 toolchain on alpine
  - CGO deps for librdkafka (gcc, musl-dev, librdkafka-dev, pkgconf)
  - air v1.61.7 (the source watcher / hot-recompile tool)

docker-compose mounts the repo at /app and runs `air -c <config>`.
Each Go service has its own air.SERVICE.toml (build target +
exclusions). Named volumes for the Go module cache and build cache
so the first build is slow (~60s for module download) but subsequent
rebuilds after a save are ~2s.

Per-service compose changes:
  - backend, consumer: dockerfile, volumes, and command updated
  - worker-base (the YAML anchor used by all 3 workers): same

Production Dockerfiles in deploy/docker/{backend,consumer,worker}.
Dockerfile are unchanged and still used by release CI. The seed
one-shot in compose continues to use backend.Dockerfile (it's a
short-lived job, no benefit from the dev image).

Rust (tracking) and Elixir (realtime) still build-on-change. They
change far less often; documenting the workaround in
resources/local-development.md for now.
2026-05-22 03:37:05 +00:00
Matthew Meszaros d25eed3eb6 feat(dev): root docker-compose with profiles, LocalStack, richer seed
Hoist the dev/sim stack to a single docker-compose.yml at the repo root.
Adds profiles (default / sim / seed / tools) so you can opt into heavier
setups, and bundles dependencies that were previously missing:

- LocalStack (KMS + DynamoDB + S3) with a localstack-init one-shot that
  idempotently creates alias/master-key-dev, the UserEncryptedKeys and
  EmailMessageData tables, and the main S3 bucket. Backend and workers
  wait on it via service_completed_successfully.
- stripe-mock for billing flows
- kafka-ui under the tools profile

Three workers with deterministic UUIDv5 hostnames (shared / premium /
dedicated) so assignment, rebalancing, and per-pool routing all have
real targets to exercise.

Richer seed (cmd/seed/main.go) loads 3 orgs across tiers, 6 mailboxes
joined to free/premium warmup pools, a Beta campaign with a 2-step
sequence, and 10 contacts (2 unsubscribed) so suppression behaviour is
visible in the UI. Idempotent — safe to re-run.

Makefile targets:
  make dev    — infra + app + one worker
  make sim    — adds premium + dedicated workers
  make seed   — rich fixtures
  make tools  — kafka-ui at :18090
  make reset  — nuke volumes
2026-05-18 13:08:34 +00:00