Commit Graph
50 Commits
Author SHA1 Message Date
Matthew Meszaros 1247c712f3 feat: add make poollink-dev, a two-instance dev environment for the Warmbly Cloud link (a prod-like cloud with DEPLOYMENT_MODE=cloud, Stripe gates on with placeholder keys, Mailpit email verification and the Sunrise Labs pool seeded, next to a fresh unclaimed self-hosted instance whose WARMBLY_CLOUD_URL points at it), with status/setup-link/down/reset subcommands and a local-development docs section 2026-08-29 07:14:04 -07:00
Matthew Meszaros cbf4190f42 feat: make a custom tracking domain verifiable instead of permanently "Pending DNS": the CNAME target is now this install's TRACKING_DOMAIN rather than a hardcoded t.warmbly.com that resolves nowhere, matching is exact on the label boundary (or on shared addresses, so a provider-flattened CNAME stops reading as no record at all) instead of strings.Contains, and every outcome carries the reason plus what DNS actually returned, including when the tracking host the customer is told to point at has no record of its own; a pasted URL is normalized to its host and a malformed one is rejected up front instead of saved and left pending forever; only a VERIFIED mailbox domain is used at send time with the shared host as the fallback and a campaign-feed entry saying why; pixels and click tickets are built from the configured host, and with none configured mail ships untracked rather than carrying links to another deployment's tracking service; adds GET /emails/:id/track and POST /emails/:id/track/verify plus an hourly re-resolution sweep so a record that propagates later starts being used and one that breaks stops routing links; and scopes the tracking-domain write by organization like the read, which also fixes GET /emails/:id passing a user id to an org-scoped query and 404ing for every caller 2026-08-24 09:02:29 -07:00
Matthew Meszaros 7c2d08f204 feat: renumber the campaign_send_outcome migration to 000091 so it stops colliding with 000089_email_auth_enforcement, which made golang-migrate refuse to init its source driver and left the backend restart-looping at boot on main, and guard the class of bug with scripts/check-migrations.sh (duplicate versions, gaps, unpaired up/down, unparseable filenames) wired into make check-migrations, make lint, a dedicated Migrations CI job and a TestEmbeddedMigrationsLoad that boots the real iofs source over the embedded FS 2026-08-23 21:26:14 -07:00
Matthew Meszaros 734cb5fe08 feat: make self-hosted onboarding survivable by fixing invite_only, which could not onboard anyone (the accept route is JWT-only, so redeeming the invitation that would create your account required already having one, making the self-host default silently identical to fully closed), threading the invitation token through registration so an invited person lands in the inviting organization instead of a stray workspace, gating SSO just-in-time provisioning behind DISABLE_REGISTRATION (it bypassed the gate entirely, so an instance set to true was still open to anyone the IdP would assert) with SSO_AUTO_PROVISION as the opt-out, correcting the OIDC redirect URL that pointed at /api/v1 against a route at /v1 and 404'd every SSO login, scoping the first-launch exemption so it no longer overrides an explicit lockdown, preserving the remaining TTL when restoring a losing setup token so a public endpoint cannot hold the claim window open forever, replacing a generic 403 with typed registration_invite_only, registration_closed, invitation_invalid, setup_token_invalid and setup_already_complete codes that name the next step, logging why no claim link was issued on an already-claimed instance instead of staying silent, adding a warmblyctl operator CLI (status with health checks and a non-zero exit, reissuable setup-link, user create/list/reset-password/grant-admin/revoke-admin/disable-2fa, hash-password) so a locked-out operator no longer needs hand-written psql, adding read-only instance configuration over 104 environment variables with structural secret redaction and fingerprints, 35 health checks, a database-backed settings tier for the three keys no environment variable owns, hiding the signup form when the config already says invite_only rather than failing the whole form with a toast, and documenting first run, accounts and access, configuration, instance health and troubleshooting alongside the root .env.example the README told operators to write but never shipped (#114) 2026-08-16 05:58:11 +02:00
Matthew Meszaros 0ae4db2c41 feat: make self-hosted auth work without a mail relay by rewriting the platform SMTP transport with real AUTH and TLS (it did neither, so SMTP_USERNAME/SMTP_PASSWORD were dead and every documented relay was unreachable), adding MAIL_TRANSPORT=smtp|log|ses with a log transport that prints codes so a fresh install can sign in with no relay, demoting the emailed login code to AUTH_LOGIN_CODE=always|new_device|off (off on self-host, per NIST SP 800-63B and OWASP ASVS), claiming the first owner through a single-use setup link or WARMBLY_BOOTSTRAP_* instead of register-then-psql, deriving every emailed URL from APP_URL rather than a hardcoded app.warmbly.com that leaked live reset tokens to the vendor, fixing the confirm hooks that read path params against paramless routes and broke login, register and reset confirmation in the dashboard everywhere, adding generic OIDC with PKCE, one-time state, verified nonce and (issuer,subject) identity binding, enforcing 2FA on the social paths that skipped it, adding a per-IP limiter and trusted-proxy handling to the unthrottled auth group, refusing boot on the published default secrets, and dropping mailpit from the default stack (#99) 2026-08-14 14:57:09 +02:00
Matthew Meszaros 8bd2c2b57a feat: make self-hosting work end to end and rewrite the guide around what was tested (#97) 2026-08-13 09:47:46 +02:00
Matthew Meszaros aeaa26d2d2 fix: use a valid local kms dev key and warn when sandbox port 8080 is taken 2026-07-21 17:34:59 +02:00
Matthew Meszaros 638c40c629 feat: make make sandbox a one-command demo stack and drop the stale kafka/localstack/worker-premium targets 2026-07-20 11:12:37 +02:00
Matthew Meszaros 49ffaedc9f feat: run make dev on nats and make make up the one-command self-host 2026-07-20 09:56:47 +02:00
Matthew Meszaros fa3672464a feat: add one-command make dev target - brings up docker infra and polls real readiness (pg_isready plus kafka/schema-registry/localstack init one-shot exit codes, failing loudly with a make logs hint), applies migrations, loads seed fixtures (SEED=false to skip), installs web deps on first run, then runs backend + consumer + both shared workers + dashboard in one terminal with Ctrl-C teardown 2026-07-16 09:51:20 +02:00
Matthew Meszaros 647a1965f2 feat: replace the AI_LOCAL boolean with an AI_PROVIDER/AI_KEY/AI_MODEL/AI_BASE_URL/AI_FREE passthrough on the backend, consumer, and run targets so make run picks the provider too 2026-07-16 06:14:19 +02:00
Matthew Meszaros 0b1037843d feat: add a make AI_LOCAL=1 toggle that runs the backend and consumer against a local Ollama model (OLLAMA_HOST/OLLAMA_MODEL overridable), off by default 2026-07-15 18:52:07 +02:00
Matthew Meszaros 7cfd9b26e9 feat: register the tracking-events Avro schema via a one-shot schema-registry-init infra service - the Rust tracking service encodes fetch-only against Schema Registry, so with no registered schema every open/click event failed serialization and was silently dropped in dev 2026-07-11 19:48:05 +02:00
Matthew Meszaros f8cf7e930e feat: native dev targets for the sandbox - make sandbox / sandbox-seed, a second native worker (worker-premium, required because paid orgs place strictly onto premium-tier workers) joined into make run, CODEC_PROVIDER=json + CREDENTIALS_ENCRYPTION_KEY + MAIL_TLS_INSECURE in the dev env, dovecot in INFRA_SVCS, and SELF_HOST defaulting to host.docker.internal so the dockerized cloud-tasks emulator can reach the host-run backend 2026-07-11 17:31:42 +02:00
Matthew Meszaros 3473d09bcc feat: fix review findings in the roles redesign — GetMembers role_id column (members endpoint 500), TransferOwnership role_id hygiene, accept-time role re-resolution, race-free in-use delete guard covering invitations, assignment anti-escalation with self-role-change block, canManage-gated members UI, colored RolePills, fresh currentOrganization on refetch, dev JWT_SECRET wiring for make realtime, docs corrections 2026-06-11 10:45:21 +02:00
Matthew Meszaros 8732805934 feat: replace signed click redirects with server-side link tickets (tracked_links store, internal resolver API, opaque /c/<id> URLs, layered anti-probe caches with miss budget and circuit breaker) removing TRACKING_LINK_SECRET entirely 2026-06-11 09:30:21 +02:00
Matthew Meszaros 2c5e8b2cbd feat: make TRACKING_LINK_SECRET a required boot-time secret on backend and tracking service with no unsigned mode and no rotation grace, so rotating the key revokes old links immediately 2026-06-11 09:00:04 +02:00
Matthew Meszaros 3236ffc459 feat: expand product docs and automation references
Restructure the docs navigation into product-focused pages, add automation expression reference content, and refresh related template and personalization surfaces.
2026-06-09 09:07:56 +02:00
Matthew Meszaros 6109601e87 feat: pin lint command to installed tool
Restore the golangci-lint config format expected by the pinned v1 CI tool and make the lint target invoke the installed binary directly.
2026-06-02 16:50:42 +02:00
Matthew Meszaros 3943b8a2e7 feat: update docs for postgres-backed secrets
Refresh developer docs, deployment notes, and public-site copy to describe the Postgres-backed encrypted key and message-map model after DynamoDB removal.
2026-06-02 15:55:01 +02:00
Matthew Meszaros f2913249b4 feat: implement the two-step email-code admin login flow 2026-06-01 13:58:06 +02:00
Matthew Meszaros 76c2cb7d49 feat: run make seed natively so it no longer needs a docker backend 2026-06-01 13:19:30 +02:00
Matthew Meszaros 83eae84d49 feat: merge main into warmup branch
Resolve the main-branch conflict in realtime event publishing by keeping both warmup account-health and audit-created events. Renumber the warmup migrations after the current main migration tail.
2026-06-01 11:36:04 +02:00
Matthew Meszaros a04d7450bc feat: guard warmup reconciliation by access
Make the warmup task reconciler re-check account state and org warmup access before scheduling replacement tasks. Remove pool membership for accounts whose org can no longer use warmup to avoid repeated no-access task churn.
2026-06-01 11:19:44 +02:00
Matthew Meszaros 8b28b86b7f feat: support exposed local dev hosts
Add PUBLIC_HOST wiring for local web, admin, site, realtime, and backend CORS so the native dev stack can be reached over Tailscale or LAN without changing localhost defaults.
2026-06-01 05:51:02 +02:00
Matthew Meszaros adedcc337e feat: support remote dev infra
Allow native dev services to target infrastructure running on another host by parameterizing local endpoints with INFRA_HOST and SELF_HOST.

Document the workflow and make Kafka advertise a configurable host for off-box clients.
2026-05-31 08:53:21 +02:00
Matthew Meszaros 0740a8bbbe feat: fix local migrations and kafka topics 2026-05-30 14:11:47 +00:00
Matthew Meszaros a227938778 feat: seed unibox trial fixtures 2026-05-30 14:06:36 +00:00
Matthew Meszaros d08c984c42 feat: snapshot current dashboard changes 2026-05-30 13:56:27 +00:00
Matthew Meszaros 7f830b976f feat: sync org session, harden campaign loading, add mailbox bulk-remove 2026-05-30 03:42:39 +00:00
Matthew Meszaros 090cc1c832 feat: rework dev workflow with native go services and dockerized infra 2026-05-30 02:57:27 +00:00
Matthew Meszaros 515ffcf099 feat: rework dev workflow with native go services and dockerized infra 2026-05-30 02:16:32 +00:00
Matt 11f1563e08 feat(admin): organization permission bits + limit overrides migration
Add two new admin permission bits — view_organizations (bit 20) and
manage_organizations (bit 21) — and switch the existing read-only
/admin/organizations routes off the borrowed AdminPermViewUsers bit they
were using as a placeholder. Backfill the three predefined roles
(support, ops, analyst) with view_organizations so existing role
mappings still resolve cleanly; super continues to pick up everything
via AllAdminPermissions.

Migration 000044 adds organization_limit_overrides, the table the next
commit's write path will target. Schema follows the "0 = inherit from
plan" convention from the design discussion: each numeric column
defaults to 0 and a CHECK constraint enforces non-negative values, so
reverting an override is a write of 0 (preserving the granted_by audit
trail) rather than a DELETE.

Bumping the permission count changes every role's numeric bitmask, so
update the make grant-admin role table to match — super is now
4194303, support 1086401, ops 1062960, analyst 1055233.
2026-05-28 08:47:57 +02:00
Matt 9d0679d0e7 chore(make): add admin/site dev shortcuts and grant-admin tooling
The admin and marketing site sit outside the compose stack, so `make app`
never started them. Add `make admin` and `make site` to launch each
workspace's dev server (Vite on 5174, Astro on 4321), and update the root
README and admin/README to point at them instead of the old "open
localhost:5174" line that implied `make app` was enough.

Also add `make grant-admin EMAIL=... [ROLE=super|support|ops|analyst]`
plus `make revoke-admin` so the first super-admin can be seeded without
hand-writing SQL. Role bitmasks mirror AdminRolePermissions in
internal/models/admin_permission.go.
2026-05-28 08:38:58 +02:00
Matthew Meszaros ef9ce6a6e2 refactor: split make dev into make infra and make app
Every docker compose invocation in the Makefile now pins
-p warmbly so all git worktrees target the same compose
project. This means infra (postgres, redis, kafka, mailpit,
localstack, stripe-mock, cloud-tasks-emulator, zookeeper,
schema-registry) is brought up once and stays running across
worktree switches. App services (backend, consumer, worker,
tracking, realtime, web) recreate in place per worktree against
the bind-mounted source.

Removed targets:
- dev, dev-down, dev-logs (and the DEV_SVCS / SVCS vars)

Added targets:
- infra, infra-down
- app, app-down, app-logs

Daily flow becomes:

    make infra                     # once, from any worktree
    cd /path/to/worktree-a
    make app                       # bring up app code for branch A
    cd /path/to/worktree-b
    make app                       # recreates app against branch B;
                                   # infra untouched, caches warm

The named cache volumes already shared their content across
worktrees (warmbly_gomodcache, warmbly_gocache, warmbly_cargo_home,
warmbly_cargo_target, warmbly_mix_deps, warmbly_mix_build); pinning
the project name additionally makes container ownership shared,
which is what eliminates the per-worktree cold start.

README.md, resources/local-development.md, resources/deployment-guide.md,
deploy/README.md, and docker-compose.dev.yml all updated to reflect
the new targets.
2026-05-25 11:43:33 +00:00
Matthew Meszaros d9dba1fd49 refactor: drop buildkit cache-mount optimization, dockerfiles work on legacy builds 2026-05-25 04:14:46 +00:00
Matthew Meszaros 1aaffd25cc fix: make seed build image upfront and disable TTY so it doesnt look frozen 2026-05-25 04:09:52 +00:00
Matthew Meszaros c5e9523924 fix: make seed reconcile backend spec before running to avoid stale healthcheck 2026-05-25 04:02:20 +00:00
Matthew Meszaros 5c64addb1a fix: make dev brings up full stack (mailpit, web) not just lang services 2026-05-25 03:43:37 +00:00
Matthew Meszaros 99d2bff0cb feat: unify dev-mode hot-reload for go, rust, elixir under make dev 2026-05-25 03:21:42 +00:00
Matthew Meszaros bfa856318d feat: add make watch hot-reload mode for go services via air 2026-05-25 03:07:22 +00:00
Matthew Meszaros becb8f2d10 build: share go build cache across worktrees, parallelize restart-go 2026-05-25 02:55:36 +00:00
Matthew Meszaros f13b549928 feat(make): make logs takes positional service names
make logs                  # everything, --tail=200 + follow
  make logs backend          # one service
  make logs backend consumer # several

Same positional-args trick as `make restart`, reused. Ctrl-C to exit
the follow.
2026-05-22 03:45:03 +00:00
Matthew Meszaros 51dc2a54d9 chore(make): drop rebuild aliases — restart is the only name
Two names for the same action was just clutter. `restart` is enough.

If you ever need to genuinely restart without rebuilding (container
restart that preserves the binary), `docker compose restart <svc>`
works directly — that's a rare enough case to not need a wrapper.
2026-05-22 03:42:15 +00:00
Matthew Meszaros 02c01d8946 fix(dev): make restart/rebuild positional and actually do the rebuild
Previous attempt distinguished restart (no rebuild) from rebuild
(rebuild + restart). That distinction was useless in practice because
'docker compose restart' alone keeps the old binary — your code
change never appears. So every iteration was actually 'make rebuild',
and 'make restart' was a trap.

Collapsed both names into one behaviour. `restart` and `rebuild` are
aliases now; both do rebuild + restart, both take the service name
positionally:

  make restart backend       # was: make rebuild SVC=backend
  make rebuild backend       # same thing
  make restart-go            # all Go services
  make restart-all           # + Rust + Elixir

Positional argument plumbing via the standard Makefile trick:
captures non-target words after `restart`/`rebuild`, turns them into
no-op rules so make doesn't error.

If anyone genuinely needs the old container-restart-without-rebuild
behaviour (env var change, re-applying a migration the backend
already has), `docker compose restart <svc>` still works directly.
Documented that escape hatch.
2026-05-22 03:40:31 +00:00
Matthew Meszaros 92ef478d2a feat(dev): make targets for easy service restart / rebuild
Simpler than full hot reload for the Go side. The web service already
runs in dev mode (Vite HMR via the node container + ./web mount), so
frontend iteration was never the problem — only Go required a manual
docker rebuild + restart, which is a sequence everyone forgets.

Three new targets:

  make restart SVC=backend       restart without rebuild (config/env
                                  changes, re-applying migrations)
  make rebuild SVC=backend       rebuild + restart one service
  make rebuild-go                rebuild + restart all Go services
                                  (backend + consumer + worker)
  make rebuild-all               same plus tracking (Rust) + realtime
                                  (Elixir) — the safe one when you've
                                  touched things across stacks

local-development.md updated with an "Iterating on code" block so
this is discoverable.
2026-05-22 03:38:33 +00:00
Matthew Meszaros aaa77eecb2 test(seed): cover seedBaseline + seedRich for shape and idempotency
The seeder is one of the few things every developer runs on every new
checkout, but it had zero tests. With three migrations added in the last
few days and the rich-fixture path now creating 30+ rows, the chance of
silently breaking a schema migration without noticing was non-trivial.

cmd/seed/main_test.go connects to SEED_TEST_DB (skips otherwise — keeps
unit tests in CI fast and prevents accidentally clobbering a dev
database), runs migrations, wipes only the fixture rows, then:

  1. Runs seedBaseline twice, verifies row count stays at 1.
  2. Runs seedRich, asserts 9 different row counts match expectations
     (users, orgs, workers, accounts, campaign, sequences, contacts,
     unsubscribed contacts, campaign leads).
  3. Re-runs seedRich, asserts every count is unchanged — the most
     important guarantee the seeder makes.
  4. Verifies warmup pool membership: 2 free, 4 premium, with the
     correct accounts in each.

Plain testing package, table-driven, matches the existing style in
internal/app/warmup/service_test.go.

`make test-seed` brings up the docker-compose Postgres and runs the
suite against it.
2026-05-18 14:56:06 +00:00
Matthew Meszaros d25eed3eb6 feat(dev): root docker-compose with profiles, LocalStack, richer seed
Hoist the dev/sim stack to a single docker-compose.yml at the repo root.
Adds profiles (default / sim / seed / tools) so you can opt into heavier
setups, and bundles dependencies that were previously missing:

- LocalStack (KMS + DynamoDB + S3) with a localstack-init one-shot that
  idempotently creates alias/master-key-dev, the UserEncryptedKeys and
  EmailMessageData tables, and the main S3 bucket. Backend and workers
  wait on it via service_completed_successfully.
- stripe-mock for billing flows
- kafka-ui under the tools profile

Three workers with deterministic UUIDv5 hostnames (shared / premium /
dedicated) so assignment, rebalancing, and per-pool routing all have
real targets to exercise.

Richer seed (cmd/seed/main.go) loads 3 orgs across tiers, 6 mailboxes
joined to free/premium warmup pools, a Beta campaign with a 2-step
sequence, and 10 contacts (2 unsubscribed) so suppression behaviour is
visible in the UI. Idempotent — safe to re-run.

Makefile targets:
  make dev    — infra + app + one worker
  make sim    — adds premium + dedicated workers
  make seed   — rich fixtures
  make tools  — kafka-ui at :18090
  make reset  — nuke volumes
2026-05-18 13:08:34 +00:00
Matthew Meszaros c564b3ac95 feat: implement unibox replies, warmup conversations, and daily email limits 2026-02-20 04:54:46 +00:00
Matthew Meszaros 772c19820d New Repository: Add Backend Code 2026-01-17 14:11:14 +00:00