Matthew Meszaros
41d43f64be
feat: check that a verified root redirect actually reaches visitors by opening the domain over http and https and naming what answered instead (Traefik, nginx, Caddy, a rewritten Host header, a missing certificate, a closed port) with per-proxy fix steps in the sending domain drawer, and let a self-hosted instance linked to Warmbly Cloud have Cloud serve and certify its redirects (connect in place from the redirect tab, the bulk dialog or a page banner), with Cloud-side linked-instance redirect endpoints under a per-instance limit, migration 000237, labelled tracking answers and a 503 when the redirect lookup is unavailable, and the sending domains, Warmbly Cloud, data control, install, OpenAPI, API and error code docs updated
2026-09-30 06:04:00 -07:00
Matthew Meszaros
f4f219b7c5
feat: run contact file imports as background jobs (upload once, a whole-file check of new, existing, repeated and invalid rows, a chunked leased runner with live CONTACT_IMPORT_PROGRESS, history, cancel, a draft that autosaves and survives a reload, remembered mappings, and a failed-rows CSV under the file's own headers), match existing contacts across the workspace, batch updates and fail a bad row alone, keep imported verdicts on update, scope every import write to the importing workspace, rebuild the import wizard with icons and inline segment creation, show company logos and the inbox provider on the contact avatar, and hide contact columns the list has no data for
2026-09-27 21:50:10 -07:00
Matthew Meszaros
e58921484d
feat: rebuild mailbox import around column mapping and automatic host and sign-in detection (CSV, XLSX, pasted lists, saved mappings, retryable rows with fixes, migrations 000205-000206), connect whole Google Workspace domains and Microsoft 365 organizations through a proved administrator grant, import from inbox vendors (InboxKit, Zapmail, Mailforge, Infraforge, Maildoso, Cheap Inboxes, ScaledMail) with vendor-managed forwarding and DNS, add a sending domains page with per-domain tracking and verified root redirects, unify Add account into one Google and one Microsoft entry with per-method choices, mark per-mailbox Google sign-in as retiring with in-place moves to the admin grant or an app password, allow the loopback security mode in the credential columns (migration 000207), read semicolon-separated CSVs, and add a mock vendor API to the sandbox
2026-09-23 08:41:01 -07:00
Matthew Meszaros
e668a2a36b
feat: complete the ADA CASA v2.1.1 AL1 control set across authentication, sessions, access control, cryptography, input validation and configuration, adding a breached-password denylist and per-account login throttling, enforced multi-factor authentication on the admin panel, step-up confirmation before an action that mints a lasting credential, purpose-scoped session tokens, single-use TOTP steps, tenant verification on every cross-referenced identifier, security headers on every surface, encrypted webhook signing secrets, per-organization idempotency, PKCE and a minimal two-scope Gmail consent on the mailbox OAuth flow, bounded spreadsheet and archive decoding, a patched Go toolchain with govulncheck in CI, and the evidence pack under compliance/casa
2026-09-19 08:18:35 +02:00
Matthew Meszaros
e36a070595
Merge remote-tracking branch 'origin/main' into fix/worker-capacity-account-distribution
2026-09-17 08:02:45 -07:00
Matthew Meszaros
8a60c6bab3
feat: build realtime on OTP 27 and update vulnerable Elixir dependencies
2026-09-17 07:54:34 -07:00
Matthew Meszaros
c1e45b194a
feat: merge latest main before worker capacity queueing
2026-09-17 06:26:07 -07:00
Matthew Meszaros
b733d09f89
feat: make inbox follow-up labels safe for manual labels and automated mail
2026-09-17 04:57:52 -07:00
SUMAN JANA
3b8761d361
feat(inbox): explain every tag on hover, label live mail, and keep follow-ups working with no external service
2026-09-17 04:57:52 -07:00
SUMAN JANA
260898bf20
feat(inbox): automatic tagging and relevance scoring for inbound mail, optional and off by default
2026-09-17 04:57:24 -07:00
Matthew Meszaros
bab9f86727
feat: correct worker capacity, mailbox distribution, observed IPv4, fleet pagination, and premium pool promotion
2026-09-17 04:15:05 -07:00
Matthew Meszaros
4784ee7d39
feat: fetch the MaxMind databases instead of requiring a mounted file ( #529 )
...
* feat: let the backend, consumer and tracking service fetch their own MaxMind databases from GEODB_URL and TRACKING_SCANNER_ASN_DB_URL, reading the archive shape from the content so a permalink tar.gz, a gzipped mmdb and a bare mmdb all work, never replacing a file already at the path, opening the bytes before installing them so a licence-key error page cannot become the database forever, skipping the AppleDouble sidecars a macOS tar writes ahead of the real file, and treating both URLs as secrets because the permalink carries the licence key
* feat: drop the trailing blank line cargo fmt --check rejects at the end of tracking/src/asndb.rs
* feat: stream the downloaded ASN archive instead of decompressing it whole, sizing each buffer from the gzip footer and the tar header so the member is allocated exactly once, which drops the peak of unwrapping a permalink tar.gz from 38 MB to 11.9 MB, essentially the database itself
* feat: stop the MaxMind licence key reaching the logs through net/http's and reqwest's own error text, which both print the URL they were given and so defeated the redaction beside them, drop userinfo as well as the query when redacting, refuse plain http for a URL carrying a credential and refuse an https-to-http redirect, and apply the size cap to the decoded database rather than the compressed transfer so a gzip bomb cannot fill the disk
* feat: strip basic-auth userinfo as well as the query when the tracking service redacts its database URL, parsing it rather than cutting at the first question mark so where a credential sits is the URL library's problem and not a guess
2026-09-15 03:06:34 -07:00
Matthew Meszaros
0a1ed6f04e
feat: resolve scanner ASNs from a GeoLite2-ASN database so the catalogue's asn: entries match without a Cloudflare transform rule, ship the Proofpoint, Mimecast and Cisco ASNs enabled behind a new probable certainty that widens the consumer's machine window instead of deciding the verdict, make the tracking event dedupe claim one coalesced operation, and report an ASN database that opened cleanly but resolves nothing ( #440 )
2026-09-15 01:48:45 -07:00
Matthew Meszaros
7300b3b021
feat: full PostHog coverage: identify the signed-in user and workspace in the dashboard and admin panel with autocapture, heatmaps, dead and rage clicks, web vitals, network timing, console capture and session replay masking only password fields, send server-side signup, trial and subscription events under the user id with the organization as a group, keep the marketing site and form pages cookieless while capturing everything stateless plus a form funnel, upload the form app's source maps, and add WARMBLY_POSTHOG_SESSION_REPLAY
2026-09-13 20:58:22 -07:00
Matthew Meszaros
d456bc48c6
feat: fix the Warmbly Cloud pool link across both roles ( #262 ): take an enrolled mailbox out of this instance's own warmup pool so local partners stop writing to it and their unverifiable warmup stops landing in the owner's unibox, recognise the cloud's warmup mail whose verify header did not survive delivery through a new warmup-deliveries lookup that ignores consumed_at because instance and cloud read the same mailbox, move the managed-mailbox access token route behind NODE_BROKER_TOKEN so the internet-facing tracking and forms services can no longer mint a live provider token, scope pause and resume to the caller's workspace, keep an enrolled mailbox listed once it goes inactive, release the cloud copy when the local mirror row cannot be written, refuse the one-time handshake when CREDENTIALS_ENCRYPTION_KEY is missing, blank an expired code's plaintext instance token, and stop errx answering 200 for a status outside its table
2026-09-12 06:58:25 -07:00
Matthew Meszaros
a4f503f620
fix: realtime cannot complete either of its TLS connections ( #452 )
...
* fix: let the realtime service actually establish its TLS connections, adding the wildcard hostname match fun Redix needs for every managed Redis and a CA bundle option for an RDS Postgres that the system trust store cannot verify
* feat: drop the redundant wget package from the realtime image since busybox already provides it, and fix a comment that read as literal interpolation
* fix: treat a blank DATABASE_SSL_CA_FILE as unset in the realtime config, because compose passes every optional variable through as empty and cacertfile with an empty path is no CA source at all rather than a fallback to the system store
2026-09-12 00:29:34 -07:00
Matthew Meszaros
280a3e64ac
feat: publish Kafka-linked image variants ( #448 )
...
* feat: publish Kafka-linked images for backend, consumer, worker and tracking as -kafka tag variants built per-arch on native runners, because the cgo librdkafka link cannot cross-compile on the build-go path
* feat: carry BUILT_AT inside the Go Kafka targets' build-args instead of the shared build-native block, so tracking and realtime stop warning about an unconsumed arg, and emit the matrix with printf because echo expands the separating backslash-n in some shells
2026-09-11 22:31:41 -07:00
Matthew Meszaros
a962683511
feat: ship AWS's RDS truststore in the backend and consumer images and point the docs at it, because Amazon RDS chains to a root that is in no public trust store so the sslmode=verify-full those docs recommended failed every connection with x509 certificate signed by unknown authority, while deliberately not setting PGSSLROOTCERT by default since an RDS-only store would break a Postgres fronted by a public CA
2026-09-11 05:23:23 +02:00
Matthew Meszaros
a84ab48729
Merge branch 'main' into feature/posthog-error-tracking
2026-09-10 10:25:21 -07:00
Matthew Meszaros
ced741e352
feat: make PostHog the default error tracker across every runtime while keeping Sentry fully supported alongside or instead of it, by turning internal/observability/errs into a two-sink fan-out with a local-log fallback, adding $exception capture to the Go services, the Rust tracking service, the Elixir realtime service and the dashboard, admin and form apps, reporting gin panics with their route, request id, workspace and user, attaching that identity plus a route and failed-request trail to browser exceptions, and wiring POSTHOG_ERROR_TRACKING, the node join env, compose, source-map upload and the docs to match
2026-09-10 19:11:32 +02:00
Matthew Meszaros
512451c8d6
feat: give every BuildKit cache mount an explicit id so the Dockerfiles build on hosted builders that reject an unnamed one, which is what stopped the backend, consumer, worker, forms, cli, updater and tracking images from building on Railway with 'flag --mount=type=cache is missing an id argument'
2026-09-10 19:05:01 +02:00
Matthew Meszaros
a08ada6666
feat: address the review on the bus bundle by downloading the compose file and config the documented one-command install never fetched, resolving container names through compose so --install-dir stops inspecting containers that do not exist, renewing a certificate that is present but expired instead of starting a stack that refuses every connection, refusing to write an empty credential when openssl fails, and falling back to dig or host where getent does not consult DNS
2026-09-10 17:39:58 +02:00
Matthew Meszaros
5ef8b608c6
feat: record the split-cloud scripts and aws-bootstrap.sh as executable, which a checkout on a filesystem that does not preserve the mode had silently dropped so make lint died with Permission denied in CI only, and assert the recorded git mode in check-split-cloud so it cannot ship non-executable again
2026-09-10 17:30:16 +02:00
Matthew Meszaros
9e689e6d91
feat: fix the three defects that stopped the split-deployment bus bundle from starting at all (NATS given store_dir twice on the command line and in nats.conf, which it refuses rather than reconciling; Redis unable to read its own private key because the image's entrypoint drops from root with gosu and discards the added group; and a healthcheck on localhost when the monitor binds IPv4 loopback, leaving the service unhealthy while serving fine), add bus/setup.sh so standing the box up is one idempotent command that verifies with real clients before printing the control plane's URLs, and add scripts/check-split-cloud.sh to make lint so a bundle that cannot start fails CI
2026-09-10 17:22:14 +02:00
Matthew Meszaros
510ee692ba
feat: address the review on the split-deployment branch by moving the two broker routes onto their own NODE_BROKER_TOKEN so the internet-facing tracking and forms services no longer hold a credential that can open any organization's data key, refusing to presign any key outside the prefixes a node reaches, fixing IAM policies that named an alias ARN KMS never resolves in a Resource element, bounding both brokered HTTP clients because the sync loop's context never expires, no longer reporting a 403 from the object store as a missing body, and redacting the DSN and URL credentials the dry-run listing printed in clear
2026-09-10 14:19:53 +02:00
Matthew Meszaros
47ba13083e
feat: make a split deployment work end to end by fixing the three defects that made an off-host node impossible to configure (nodeEnvKeys shipped S3_BUCKET and KMS_KEY_ID, which nothing reads, so an AWS-backed node silently used the default bucket and key alias; a joined consumer never received PRIMARY_DB and died at boot; and node.env was rewritten on every join with no file an operator could add to), then removing the need for cloud credentials on a node at all with brokered KMS and blob providers that renderNodeEnv hands out automatically, plus deploy/split-cloud, scripts/aws-bootstrap.sh, two fleet instance checks and the docs
2026-09-10 13:58:59 +02:00
Matthew Meszaros
18a8c7b009
feat: give a self-hosted instance on-demand TLS for customer tracking and forms domains, gating Caddy's ask on a /tls/authorize endpoint that answers only for domains this instance has verified, so a workspace CNAME stops serving every tracked link and opt-out link with no certificate
2026-09-09 08:56:08 -07:00
Matthew Meszaros
494996ea8a
feat: stop the backend image copying scripts/install-worker.sh, which the pull-based fleet change deleted, and clear the rest of that removal's dangling references out of the backend systemd unit, deploy/README, architecture and troubleshooting docs
2026-09-09 08:56:05 -07:00
Matthew Meszaros
51dedc90ee
feat: put every runtime behind one optional error-reporting story: a single internal/observability/errs wrapper that is now the only package importing sentry-go, InitSentry for cmd/forms, release and environment tags on every service from the existing build stamp, optional Sentry in the admin panel and the public forms app, the sentry crate in the Rust tracking service, release tagging in realtime, CI source-map upload that only runs when a Sentry token is configured, and docs covering the DSN for each service
2026-09-07 03:51:06 -07:00
Matthew Meszaros
4e94d2b47b
feat: install the CLI in one command from warmbly.com/cli.sh or cli.ps1, with checksum-verified archives, a Homebrew formula and Scoop manifest generated alongside them, a ghcr cli image, and a make cli-check that installs from a local mirror and proves a tampered checksum stops the install
2026-09-04 20:14:38 -07:00
Matthew Meszaros
d68bbcd2ab
feat: add a one-command self-host installer at warmbly.com/install.sh with an interactive data-control wizard, give docker-compose.yml image keys and per-store volume variables, add an image-mode updater, move engagement/form/audit retention into instance settings, and add warmblyctl backup/restore
2026-09-04 05:49:54 -07:00
Matthew Meszaros
701dc404a2
Merge remote-tracking branch 'origin/main' into fix/click-without-open-tracking
2026-09-03 05:22:05 -07:00
Matthew Meszaros
3d5eb36ae5
feat: second review pass on self-hosted updates: git describe suffixes after a prerelease (rc.1-2-gabc1234) now parse as prerelease plus commit distance, Apply validates the updater for every target including explicit tags, a DNS failure only reads as the compose profile being off when UPDATER_URL names the updater service and stays unreachable for custom hosts, the admin dialog re-checks that the update can still start before launching it and drops the confirmation when it cannot, the bare-metal installer refuses symlinks anywhere inside a build directory, restores config.js with --remove-destination and bounds every health probe, and the bare-metal docs add the single sudoers rule before the upgrade command
2026-09-03 05:04:30 -07:00
Matthew Meszaros
bb35de0455
feat: address review on self-hosted updates: the updater no longer re-locks its mutex when a job finishes (every job used to deadlock at completion and freeze the status API), the backend caches the updater view so the member version pill, the health checks and the admin poll share one read and an absent updater is reported as not running rather than broken, the bare-metal upgrade builds unprivileged and hands off to a root-owned fixed-path installer that refuses symlinks so sudoers allows one command instead of install/cp/rm/chown/chmod/systemctl/ln, the installer fails when the backend does not come back, the seed image gets the version build args, the dashboard gates the update action on manage_settings and stops polling a backend that answers 404, and revived timestamps are typed as Date
2026-09-03 05:04:30 -07:00
Matthew Meszaros
b2ea1f1961
feat: add self-hosted update awareness and one-click updates: every binary is stamped with its version and commit, the backend polls GitHub Releases and a new host-side updater (cmd/updater, compose profile or systemd unit) reports the checkout's commit distance, the admin panel's top bar shows a version pill that turns into an update indicator and opens a dialog with confirmation, live step progress and log, restart tracking and result, the dashboard header shows the same pill to every member of a self-hosted instance with the full update flow for platform admins, Setup and health gains update_available and updater_unreachable checks, warmblyctl status prints the version, make upgrade and scripts/upgrade-bare-metal.sh cover the by-hand paths, and docs gain an Updates page plus configuration, health, deployment and API reference updates
2026-09-03 05:04:30 -07:00
Matthew Meszaros
a3264f0d39
feat: address the review on the rebuilt engagement branch: a deferred human click's effects are now durable and once-only (announce_pending on the click row written before the event is marked processed, a claim in finishHumanClick and a minute-by-minute sweep that finishes what a restart cut off, a burst relabel clears the flag), the tracking service keys the source-address token with TRACKING_IP_HASH_KEY so it cannot be enumerated back into an IPv4 address, the migration no longer adds a blocking index to the existing click table, and timeline click and open rows carry task_id
2026-09-03 04:13:39 -07:00
Matthew Meszaros
9485503e03
feat: merge main into the unsubscribe branch, keeping the new campaigns.kind column alongside unsubscribe_mode in every campaign scanner and renumbering the opt-out migration to 000123 because main took 000122 for campaign kind
2026-09-03 01:56:38 -07:00
Matthew Meszaros
5eb92c601c
feat: give every campaign email a working opt-out: a reply-to-opt-out line by default or an unsubscribe link (workspace setting under Settings > Sending with a per-campaign override and a {{.UnsubscribeLink}} variable), signed per-recipient unsubscribe links served on the API origin so the List-Unsubscribe header no longer points at a dead warmbly.com page, a confirm page on GET with RFC 8058 one-click on POST and a resubscribe button, reply opt-out detection through the whole-word compliance lexicon with quoted history stripped, a first-class suppression list (Contacts tab, GET/POST/DELETE /suppressions with address and domain entries, audited removal, contact drawer action), the contact Subscribed flag enforced in campaign routing, migration 000122 with a shared recipient_suppressed() predicate, and docs for all of it
2026-09-03 01:52:40 -07:00
Matthew Meszaros
16eed14501
feat: keep every frontend node_modules and build output out of the root Docker build context so a host forms/node_modules from a native make forms or make dev no longer overwrites the forms image's own pnpm install and aborts the build with ERR_PNPM_ABORTED_REMOVE_MODULES_DIR_NO_TTY, set CI=true in the forms, web and admin app build stages so pnpm reinstalls instead of prompting in a build with no TTY, and document the symptom on the troubleshooting page ( #292 )
2026-09-03 01:04:27 -07:00
Matthew Meszaros
290f1f00aa
Merge remote-tracking branch 'origin/main' into feature/public-forms
...
# Conflicts:
# cmd/backend/main.go
# docs/content/docs/guides/workspace-export-import.mdx
2026-09-01 09:29:53 -07:00
Matthew Meszaros
d6ddf1f170
feat: fix implicit-TLS SMTP on 465 and IMAP STARTTLS on 143 behind a stored per-mailbox security mode that accepts any port, stop worker ID churn orphaning mailbox assignments via flock-claimed persistent worker ids, give the unibox a standard mail-folder sidebar (inbox/sent/drafts/archive/spam/trash) backed by a provider-derived folder column, and expose the AI tool registry over REST for non-MCP function-calling agents ( #283 )
2026-09-01 03:53:19 -07:00
Matthew Meszaros
60c9e316d9
feat: hosted lead-capture forms end to end: drag-and-drop builder with field settings, design panel, embed/share and submissions tabs in the dashboard, a public TanStack form app (forms/) served by the new standalone forms service (cmd/forms + internal/formserver) on FORMS_DOMAIN with per-form frame-ancestors CSP, honeypot/fill-time/Turnstile/per-IP submit protection and a same-origin JSON API proxying the backend internal API, form submissions creating contacts with categories and campaign enrollment plus realtime, audit, webhook and org-transfer coverage, migration 000114, seed forms, CI jobs, Dockerfile, systemd/nginx/compose manifests and docs
2026-09-01 01:17:51 -07:00
Matthew Meszaros
623862b9f8
feat: point every server block in deploy/nginx/warmbly.conf at one Let's Encrypt certificate under live/app.example.com and explain that it must be issued standalone before the site is enabled
2026-08-30 00:21:16 -07:00
Matthew Meszaros
b9152ea460
feat: document the systemd and nginx directories in deploy/README.md and point the new Deploying without Docker section at the docs guide
2026-08-29 23:37:22 -07:00
Matthew Meszaros
ee3379e399
feat: add deploy/nginx/warmbly.conf serving the static web and admin builds with no-store index.html and config.js and reverse-proxying the API, websocket and tracking hosts over TLS
2026-08-29 23:37:22 -07:00
Matthew Meszaros
2d6bec4025
feat: add deploy/systemd units for backend, consumer, worker, tracking and realtime so a Docker-free install runs every service as the unprivileged warmbly user from /opt/warmbly with EnvironmentFile config
2026-08-29 23:37:22 -07:00
Matthew Meszaros
111a5a4034
feat: read the forwarded client address from exactly one operator-named header (TRACKING_CLIENT_IP_HEADER, default x-forwarded-for with the proxy-appended last entry, cf-connecting-ip only when configured) so a client-supplied CF-Connecting-IP passed through a generic trusted proxy can no longer choose the rate-limit bucket or the stored page-hit location, with tests and the configuration, env example and compose entries
2026-08-29 05:11:22 -07:00
Matthew Meszaros
8daefbe8c4
feat: address the Greptile review on website tracking by believing forwarded-IP headers only from TRACKING_TRUSTED_PROXIES (socket peer otherwise, proxy-appended last X-Forwarded-For entry, applied to pixel, click and page-hit paths), making IdentifyVisitor report whether it claimed the row so a lost identification race re-reads the visitor and splits onto a fresh record instead of attributing the hit to the wrong contact, forgetting the edge dedupe entry when a forward to the backend fails so the retry is counted, and trimming the new Go and Rust comments to the one-line style
2026-08-29 04:16:28 -07:00
Matthew Meszaros
27630eec0a
feat: add website visitor tracking for issue #255 with migration 000106 (website_tracking_settings, website_visitors, website_page_hits, all registered in the orgtransfer spec), a consent-gated dependency-free tracking.js served by the Rust tracking service with a rate-limited, size-capped, prefetch-filtered POST /p ingest that forwards to a new backend internal page-hits endpoint for server-side user-agent and GeoIP enrichment, contact identification only through the click ticket the redirect appends to registered hosts, a per-workspace retention job, page_hit events with an expandable detail view in the contact Activity timeline, a Settings > Website tracking page for the snippet and consent, location and retention configuration, realtime PAGE_HIT fanout, and a website tracking guide plus endpoint, export and configuration docs
2026-08-29 03:25:50 -07:00
Matthew Meszaros
9ab42cfd65
feat: build the browser half of social sign-in, which was never wired: GOOGLE_CLIENT_ID was read at boot and made the login screen render a Google button, but the button opened a popup at /auth/google/login which no route served, and authService.GoogleAuth/AppleAuth had no caller anywhere in the codebase; internal/app/socialauth now runs Google and Apple through the flow generic OIDC already used (one-time state, PKCE, nonce, id_token verified against the provider JWKS, identity keyed on issuer and subject, JIT provisioning, the ban and 2FA gates), the redirect URI defaults to API_PUBLIC_URL/v1/auth/<provider>/callback and is logged at boot because registering the dashboard origin instead is the mistake that produces a valid OAuth client and a dead button, /auth/config advertises only providers the backend can actually complete, the SSO landing page no longer swallows a two_fa_required response, and OIDC_PROVIDER_NAME finally reaches the button it documents
2026-08-28 01:33:08 -07:00
Matthew Meszaros
SUMAN JANA