Commit Graph
25 Commits
Author SHA1 Message Date
Matthew Meszaros 26c88d2426 feat: address the CodeRabbit review on the PostHog PR by dropping a referrer that is not a hostname instead of storing its query string or fragment, redacting email-shaped values out of every acquisition field before they reach the database or an analytics property, counting an invited signup which returned before the count was taken, hanging subscription_started off the persisted trial-to-paid transition so a redelivered webhook cannot report a second start, capturing the validated provider in both OAuth mailbox paths including Warmbly Cloud, making the two acquisition toggles mutually exclusive and naming them after what they actually select, and replacing the unsupported CNIL consent-exemption claim with what the guidance says and an explicit note that qualifying is a deployment-specific assessment this document does not make 2026-09-07 05:45:11 -07:00
Matthew Meszaros 8e9f67f46b feat: add cookieless PostHog analytics for the hosted marketing site and dashboard with server-side signup and subscription events, a first-party acquisition record written once at signup on a new organization_acquisition table registered in the org-transfer spec, an acquisition column and channel filter in the admin org list, and never a single request from a self-host because every key is unset by default 2026-09-07 04:18:33 -07:00
Matthew Meszaros cd5e1e9cf7 Merge pull request #72 from warmbly/research/warmup-abuse-cold-effectiveness
feat: deliverability research + observe-only SPF/DKIM/DMARC auth state and watch-band cold throttle
2026-07-17 05:29:58 +02:00
Matthew Meszaros 3ea2831040 feat: annotate the research roadmap with the observe-only auth-state and watch-band cold-throttle work already shipped on this branch 2026-07-17 03:24:18 +00:00
Matthew Meszaros a104b1e220 feat: add warmup/abuse/cold-send deliverability research report with prioritized Now/Next/Later roadmap in resources/warmup-abuse-cold-research.md 2026-07-01 02:11:57 +00:00
Matthew Meszaros 7813d173b1 Move development docs to docs/development/ 2026-06-28 12:25:08 +02:00
Matthew Meszaros bd7db069ba Merge origin/main into feature/integrations-3, resolving doc conflicts by accepting the docs-restructure deletions 2026-06-10 18:56:47 +02:00
Matthew Meszaros 5b735bb528 feat: replace ascii architecture diagrams in the realtime readme and resources docs with prose and tables 2026-06-10 18:27:42 +02:00
Matthew Meszaros fc33acc64d feat: document the per-organization DEK model
Update the encryption sections in the agent notes, internal API auth
doc, OAuth setup doc, and architecture notes, and state explicitly that
per-user DEKs must not be reintroduced.
2026-06-10 17:17:19 +02:00
Matthew Meszaros 3943b8a2e7 feat: update docs for postgres-backed secrets
Refresh developer docs, deployment notes, and public-site copy to describe the Postgres-backed encrypted key and message-map model after DynamoDB removal.
2026-06-02 15:55:01 +02:00
Matthew Meszaros ef9ce6a6e2 refactor: split make dev into make infra and make app
Every docker compose invocation in the Makefile now pins
-p warmbly so all git worktrees target the same compose
project. This means infra (postgres, redis, kafka, mailpit,
localstack, stripe-mock, cloud-tasks-emulator, zookeeper,
schema-registry) is brought up once and stays running across
worktree switches. App services (backend, consumer, worker,
tracking, realtime, web) recreate in place per worktree against
the bind-mounted source.

Removed targets:
- dev, dev-down, dev-logs (and the DEV_SVCS / SVCS vars)

Added targets:
- infra, infra-down
- app, app-down, app-logs

Daily flow becomes:

    make infra                     # once, from any worktree
    cd /path/to/worktree-a
    make app                       # bring up app code for branch A
    cd /path/to/worktree-b
    make app                       # recreates app against branch B;
                                   # infra untouched, caches warm

The named cache volumes already shared their content across
worktrees (warmbly_gomodcache, warmbly_gocache, warmbly_cargo_home,
warmbly_cargo_target, warmbly_mix_deps, warmbly_mix_build); pinning
the project name additionally makes container ownership shared,
which is what eliminates the per-worktree cold start.

README.md, resources/local-development.md, resources/deployment-guide.md,
deploy/README.md, and docker-compose.dev.yml all updated to reflect
the new targets.
2026-05-25 11:43:33 +00:00
Matthew Meszaros e5c0c8a448 chore(dev): standardize ports where they don't conflict
Most container ports go back to their natural defaults — the offsets
that existed weren't justified, they just made URLs harder to remember.
Now standard:

  backend         8080   (was always 8080)
  tracking        3000   (was 13000)
  realtime        4000   (was 14000)
  web             5173   (was 15173 — already changed)
  kafka           9092   (was 19092)
  schema-registry 8081   (was 18081)
  localstack      4566   (was 14566)
  cloud-tasks     8123   (was 18123)
  stripe-mock     12111  (always was)

Kept offset (the defaults conflict too often on real dev machines):

  postgres        15432  (system postgres / sibling project)
  redis           16379  (sibling docker projects with redis)
  mailpit ui      18025  (sibling docker projects with mailpit)
  mailpit smtp    11025  (same)
  kafka-ui        18090  (8080 already used by backend)

Touched: docker-compose.yml, Makefile (test-seed SEED_TEST_DB), READMEs
(root + deploy), local-development.md + deployment-guide.md. Internal
docker-network refs (kafka:29092, mailpit:1025, etc.) unchanged — only
host-port mappings moved. Compose validated, all default-profile
services come up healthy on the new ports.
2026-05-22 14:51:16 +00:00
Matthew Meszaros 34ab7ad266 fix(web): correct Request import depth + move web back to port 5173
The new admin API clients (audit, credentials, workers) imported Request
with four '..' segments instead of three. Vite's import-analysis failed
with "Failed to resolve import ../../../../Request" because that path
resolves to api/Request, not client/Request. tsc didn't catch it because
the resolver was permissive enough to keep going, but the runtime is
strict. Matched the existing pattern from roles/getRoles.ts (three dots
for Request, four for models).

Separately: web was on host port 15173, offset from the canonical 5173
to avoid colliding with a locally-running Vite outside Docker. Nobody
actually runs Vite locally in this setup, and the offset makes the URL
non-obvious. Moved back to 5173:5173 and updated VITE_APP_URL plus the
docs.

If a developer one day wants to run a host-side Vite alongside the
container, change the mapping back to "15173:5173" — the offset is the
escape hatch, not the default.
2026-05-22 03:55:34 +00:00
Matthew Meszaros f13b549928 feat(make): make logs takes positional service names
make logs                  # everything, --tail=200 + follow
  make logs backend          # one service
  make logs backend consumer # several

Same positional-args trick as `make restart`, reused. Ctrl-C to exit
the follow.
2026-05-22 03:45:03 +00:00
Matthew Meszaros 51dc2a54d9 chore(make): drop rebuild aliases — restart is the only name
Two names for the same action was just clutter. `restart` is enough.

If you ever need to genuinely restart without rebuilding (container
restart that preserves the binary), `docker compose restart <svc>`
works directly — that's a rare enough case to not need a wrapper.
2026-05-22 03:42:15 +00:00
Matthew Meszaros 02c01d8946 fix(dev): make restart/rebuild positional and actually do the rebuild
Previous attempt distinguished restart (no rebuild) from rebuild
(rebuild + restart). That distinction was useless in practice because
'docker compose restart' alone keeps the old binary — your code
change never appears. So every iteration was actually 'make rebuild',
and 'make restart' was a trap.

Collapsed both names into one behaviour. `restart` and `rebuild` are
aliases now; both do rebuild + restart, both take the service name
positionally:

  make restart backend       # was: make rebuild SVC=backend
  make rebuild backend       # same thing
  make restart-go            # all Go services
  make restart-all           # + Rust + Elixir

Positional argument plumbing via the standard Makefile trick:
captures non-target words after `restart`/`rebuild`, turns them into
no-op rules so make doesn't error.

If anyone genuinely needs the old container-restart-without-rebuild
behaviour (env var change, re-applying a migration the backend
already has), `docker compose restart <svc>` still works directly.
Documented that escape hatch.
2026-05-22 03:40:31 +00:00
Matthew Meszaros 92ef478d2a feat(dev): make targets for easy service restart / rebuild
Simpler than full hot reload for the Go side. The web service already
runs in dev mode (Vite HMR via the node container + ./web mount), so
frontend iteration was never the problem — only Go required a manual
docker rebuild + restart, which is a sequence everyone forgets.

Three new targets:

  make restart SVC=backend       restart without rebuild (config/env
                                  changes, re-applying migrations)
  make rebuild SVC=backend       rebuild + restart one service
  make rebuild-go                rebuild + restart all Go services
                                  (backend + consumer + worker)
  make rebuild-all               same plus tracking (Rust) + realtime
                                  (Elixir) — the safe one when you've
                                  touched things across stacks

local-development.md updated with an "Iterating on code" block so
this is discoverable.
2026-05-22 03:38:33 +00:00
Matthew Meszaros f95a12f2a1 Revert "feat(dev): hot reload for Go services in docker-compose"
This reverts commit 3a84e33155.
2026-05-22 03:37:34 +00:00
Matthew Meszaros 3a84e33155 feat(dev): hot reload for Go services in docker-compose
Until now, only the web service hot-reloaded (Vite HMR via the
node:22-alpine container + ./web mount). The Go services (backend,
consumer, worker) used their production multi-stage Dockerfiles, so
every code change meant `docker compose build <svc> && docker compose
up -d <svc>` — ~30s per service.

Switched all Go services to a shared dev image (go.dev.Dockerfile)
that ships:
  - full Go 1.25 toolchain on alpine
  - CGO deps for librdkafka (gcc, musl-dev, librdkafka-dev, pkgconf)
  - air v1.61.7 (the source watcher / hot-recompile tool)

docker-compose mounts the repo at /app and runs `air -c <config>`.
Each Go service has its own air.SERVICE.toml (build target +
exclusions). Named volumes for the Go module cache and build cache
so the first build is slow (~60s for module download) but subsequent
rebuilds after a save are ~2s.

Per-service compose changes:
  - backend, consumer: dockerfile, volumes, and command updated
  - worker-base (the YAML anchor used by all 3 workers): same

Production Dockerfiles in deploy/docker/{backend,consumer,worker}.
Dockerfile are unchanged and still used by release CI. The seed
one-shot in compose continues to use backend.Dockerfile (it's a
short-lived job, no benefit from the dev image).

Rust (tracking) and Elixir (realtime) still build-on-change. They
change far less often; documenting the workaround in
resources/local-development.md for now.
2026-05-22 03:37:05 +00:00
Matthew Meszaros f4345c3c54 docs: rewrite README and resources for current architecture
Old docs described a k8s/ArgoCD/Terraform deployment that no longer
exists, with ASCII-art system diagrams that hadn't aged well. Rewritten
to match how the project actually ships:

- README: control plane (Railway) + execution plane (per-VPS workers)
  split, dashboard-driven worker management, credentials/profiles,
  auto-update from GitHub releases, OS package updates, self-hosting
  knobs. Removed all ASCII art.

- resources/architecture.md: control vs execution plane, encryption
  model (worker SSH keys + platform secrets under the same KMS-envelope
  cipher as user secrets), worker identity from public IPv4, credentials
  model, push-driven release flow, anti-abuse layers, source anchors.

- resources/deployment-guide.md: end-to-end from "provision a VPS" to
  "auto-update on release". No more k8s, ArgoCD, kubectl, or Terraform.
  Step-by-step backend env, webhook setup, worker add flow, day-2 ops,
  rollback per plane.

- resources/local-development.md: the five make targets (dev / sim /
  seed / tools / reset), what each profile runs, LocalStack bootstrap,
  rich seed contents, native dev against containerized infra, the
  offset-port URL table.

- resources/cicd.md: the two-plane build/release flow, image tag scheme
  ({sha} / dev / vX.Y.Z / vX.Y / vX / prod), webhook setup, release
  process, security notes around HMAC and least-privilege worker AWS
  keys.

- deploy/README.md: tight version of the same.
2026-05-18 13:09:31 +00:00
Matthew Meszaros d8d88c7f69 feat: add warmup health tracking, migrate repos to postgres, and overhaul web UI 2026-04-03 06:08:52 +00:00
Matthew Meszaros c564b3ac95 feat: implement unibox replies, warmup conversations, and daily email limits 2026-02-20 04:54:46 +00:00
Matthew Meszaros 6c6d26d8f0 Update auth and onboarding flow 2026-02-14 05:38:27 +01:00
Matthew Meszaros 2635e9c6da frontend & email design; local email setup 2026-02-12 17:25:46 +01:00
Máté Mészáros (Laptop) ed35ab2dbc Realtime Updates 2026-01-30 15:32:58 +01:00