starred/warmbly
1
0
Fork 0
mirror of https://github.com/warmbly/warmbly.git synced 2026-10-04 00:02:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,494 Commits 1 Branch 74 Tags
ffd27bc46d2eb07bb0e1ccae186406ecd670e116
Commit Graph
32 Commits
Author SHA1 Message Date
Matthew Meszaros e8186e5f52 fix: the tracking service must not write Avro to a JSON consumer (#450) 
* feat: make the Rust tracking publisher honour CODEC_PROVIDER on Kafka instead of always writing Avro, so a json consumer stops silently dropping every open and click, and refuse avro at boot when no Schema Registry is configured

* feat: build the tracking service's kafka feature in CI, because clippy on the default build never opens kafka.rs and that file now ships as the published tracking -kafka image

* feat: install libcurl and the rest of the librdkafka build dependencies for the tracking kafka clippy step, which fails at the first object without curl headers even with WITH_CURL=0
 2026-09-11 23:00:15 -07:00
Matthew Meszaros f1f5249153 feat: authenticate to NATS with a user JWT and nkey seed so a managed bus like Synadia Cloud can replace a self-run one, taking the credential from a file path for containers and from a single-line base64 value for the fleet, because a node receives environment variables rather than files and the env file docker reads cannot express the multi-line credentials format, in both the Go event bus and the Rust tracking publisher 2026-09-12 05:29:09 +02:00
Matthew Meszaros 80c3c79a31 feat: add live coverage that the machine windows survive the settings document's jsonb round trip and reach the classifier, a guard that every shipped scanner CIDR is written as its own network address since the loader truncates host bits silently, and correct the comments that claimed an edit lands on the very next event when the consumer reads through a thirty second cache in its own process 2026-09-11 03:11:35 -07:00
Matthew Meszaros 184a3dc08e feat: make the automated-open and automated-click windows operator-editable under Instance settings and raise their defaults to 60s and 30s, because the ten-second window was anchored on dispatch to the worker rather than on delivery and routinely expired before the recipient-side gateway it was meant to catch had even seen the message, and add Barracuda's published Email Gateway Defense blocks to the scanner catalogue with Proofpoint, Mimecast and Cisco shipped commented out because browser isolation renders a clicked page from the vendor's own network 2026-09-11 02:53:34 -07:00
Matthew Meszaros e7f25890d2 feat: install the ring crypto provider at tracking startup so the service can open a TLS connection at all, because rustls 0.23 refuses to choose when both aws-lc-rs and ring are in the tree and panics at the first handshake, which took the whole service down the moment it pointed at a tls:// bus and which plaintext local development never reveals 2026-09-11 05:19:52 +02:00
Matthew Meszaros ced741e352 feat: make PostHog the default error tracker across every runtime while keeping Sentry fully supported alongside or instead of it, by turning internal/observability/errs into a two-sink fan-out with a local-log fallback, adding $exception capture to the Go services, the Rust tracking service, the Elixir realtime service and the dashboard, admin and form apps, reporting gin panics with their route, request id, workspace and user, attaching that identity plus a route and failed-request trail to browser exceptions, and wiring POSTHOG_ERROR_TRACKING, the node join env, compose, source-map upload and the docs to match 2026-09-10 19:11:32 +02:00
Matthew Meszaros 2599d6abb2 feat: stop handing the website-tracking identification ticket to a recognised scanner, which filed its walk of a tracked link as the recipient's own page view, read TRACKING_SCANNER_BUILTINS the way the backend's configuration registry does so off no longer means on, and ship the whole-cloud Microsoft and Google ASN entries commented out because a recipient browsing from Azure or Google Cloud sits inside them 2026-09-10 04:32:52 -07:00
Matthew Meszaros ca4c1279ab feat: strip a scanner-catalogue comment before splitting the line on commas, so the shipped file's own prose stops being read as entries and warning about each fragment at every tracking boot, and count unreadable entries so a typo in an operator's list is visible 2026-09-10 03:41:10 -07:00
Matthew Meszaros 6684dafe3a feat: classify opens and clicks that arrive from a known mail-filtering network as automated, so Microsoft 365 Defender's delivery-time pixel fetches and Safe Links URL detonations stop counting as engagement (issue #410), with a shipped scanner catalogue, TRACKING_SCANNER_* overrides and optional ASN matching from a trusted edge header 2026-09-10 03:38:50 -07:00
Matthew Meszaros c403af6757 feat: give the recipient opt-out routes their own per-source budget in the tracking service instead of the counter pixels and clicks spend, so a NAT or mail gateway that exhausts the tracking budget cannot make the unsubscribe page answer 429 to the recipient behind it 2026-09-09 08:38:55 -07:00
Matthew Meszaros 7d58b874b8 feat: keep every recipient-facing and self-host-facing address on the deployment's own domain: mint unsubscribe links on a workspace's verified tracking domain (served by the tracking service, proxied to the backend that owns the pages), attach RFC 8058 one-click only over https, resolve all branding through config.Brand() gated on SelfHosted() so a self-host's email footer, sign-in links, stats card, API example and public form badge name nobody else, drop the app.warmbly.com fallback from AppBaseURL, blank TRACKING_DOMAIN and FORMS_DOMAIN on core-only installs, and have install.sh offer to configure a fresh interactive install instead of silently defaulting to localhost 2026-09-09 06:34:43 -07:00
Matthew Meszaros f043bfc761 feat: address the CodeRabbit review on the Sentry PR by escaping every runtime-config value the web and admin entrypoints write into a JavaScript string literal so a quote or newline in a DSN cannot inject or discard the whole config, reporting only the count and types of a failed query's parameters instead of their values now that the db scope actually applies, cloning the fallback hub in errs.Hub so scope cannot leak between requests, flushing the tracking service before every fatal exit since process::exit skips the guard's drop, treating a blank WARMBLY_RELEASE or APP_ENV as unset in realtime, stamping the environment and a service tag into the forms page, and correcting the docs claim that an unset DSN means the SDK is never initialised 2026-09-07 04:47:51 -07:00
Matthew Meszaros 51dedc90ee feat: put every runtime behind one optional error-reporting story: a single internal/observability/errs wrapper that is now the only package importing sentry-go, InitSentry for cmd/forms, release and environment tags on every service from the existing build stamp, optional Sentry in the admin panel and the public forms app, the sentry crate in the Rust tracking service, release tagging in realtime, CI source-map upload that only runs when a Sentry token is configured, and docs covering the DSN for each service 2026-09-07 03:51:06 -07:00
Matthew Meszaros a3264f0d39 feat: address the review on the rebuilt engagement branch: a deferred human click's effects are now durable and once-only (announce_pending on the click row written before the event is marked processed, a claim in finishHumanClick and a minute-by-minute sweep that finishes what a restart cut off, a burst relabel clears the flag), the tracking service keys the source-address token with TRACKING_IP_HASH_KEY so it cannot be enumerated back into an IPv4 address, the migration no longer adds a blocking index to the existing click table, and timeline click and open rows carry task_id 2026-09-03 04:13:39 -07:00
Matthew Meszaros 31dabea0a4 feat: rebuild the click-without-open fix on top of the per-link click attribution from #298: a person's click now also counts as an open and a burst that withdraws the click withdraws the open it implied unless a real open is on record, routing readers ignore machine opens as the docs promised, every open gets its own log row and every open and click records the mail client or proxy, browser, device, OS, country, region and city (migration 000124: origin columns on email_link_clicks plus an email_opens table), the tracking service publishes only the address's network in a nullable client_ip field which the consumer resolves with GeoLite and drops, the contact Activity tab shows each open and the origin of opens and clicks, the campaign overview gains a who-engaged-from-where breakdown exposed as engagement in campaign analytics, live open and click events carry occurred_at, client and location, the leads table explains why an open is not always counted, both logs are pruned daily after a year, email_opens joins the export registry, the consumer reads GEODB_PATH optionally, and the guides and API references are updated (fixes #294) 2026-09-03 03:49:44 -07:00
Matthew Meszaros 99be92f159 feat: attribute every campaign click to the exact link with a per-link click log (email_link_clicks) behind the contact activity timeline, campaign live feed, recent activity and the email_clicked webhook, add per-campaign automatic UTM tagging (utm_tracking with source, medium and campaign overrides, utm_content from the link text) applied at send time to the stored ticket destination, and classify opens and clicks as machine when they arrive within ten seconds of dispatch or when several links of one email are followed within five seconds, so scanner clicks are logged but never count as engagement, fire automations or send webhooks 2026-09-03 01:05:34 -07:00
Matthew Meszaros 2e376bcd4a feat: make the tracking publisher honor user:pass@ or token@ credentials in NATS_URL by lifting them into async-nats ConnectOptions, since async-nats ignores URL userinfo unlike the Go client, and redact the userinfo from the connect log line 2026-08-30 00:21:16 -07:00
Matthew Meszaros e84d47c492 feat: drop a page hit instead of storing it under the wrong contact when tying a browser to a ticket fails after a lost identification race (attach now fails closed and the edge retries), and make the edge dedupe an atomic request-owned claim so a failed concurrent forward can only release its own entry and never a successful request's 2026-08-29 05:20:26 -07:00
Matthew Meszaros 111a5a4034 feat: read the forwarded client address from exactly one operator-named header (TRACKING_CLIENT_IP_HEADER, default x-forwarded-for with the proxy-appended last entry, cf-connecting-ip only when configured) so a client-supplied CF-Connecting-IP passed through a generic trusted proxy can no longer choose the rate-limit bucket or the stored page-hit location, with tests and the configuration, env example and compose entries 2026-08-29 05:11:22 -07:00
Matthew Meszaros 8daefbe8c4 feat: address the Greptile review on website tracking by believing forwarded-IP headers only from TRACKING_TRUSTED_PROXIES (socket peer otherwise, proxy-appended last X-Forwarded-For entry, applied to pixel, click and page-hit paths), making IdentifyVisitor report whether it claimed the row so a lost identification race re-reads the visitor and splits onto a fresh record instead of attributing the hit to the wrong contact, forgetting the edge dedupe entry when a forward to the backend fails so the retry is counted, and trimming the new Go and Rust comments to the one-line style 2026-08-29 04:16:28 -07:00
Matthew Meszaros 27630eec0a feat: add website visitor tracking for issue #255 with migration 000106 (website_tracking_settings, website_visitors, website_page_hits, all registered in the orgtransfer spec), a consent-gated dependency-free tracking.js served by the Rust tracking service with a rate-limited, size-capped, prefetch-filtered POST /p ingest that forwards to a new backend internal page-hits endpoint for server-side user-agent and GeoIP enrichment, contact identification only through the click ticket the redirect appends to registered hosts, a per-workspace retention job, page_hit events with an expandable detail view in the contact Activity timeline, a Settings > Website tracking page for the snippet and consent, location and retention configuration, realtime PAGE_HIT fanout, and a website tracking guide plus endpoint, export and configuration docs 2026-08-29 03:25:50 -07:00
Matthew Meszaros f0846eb034 feat: stop a freshly connected mailbox being silently excluded from every campaign send, by making an unset mailbox timezone representable as the empty string the campaign scheduler already checks for, since email_accounts.timezone defaulted to 'UTC' while campaigns.timezone defaults to 'Europe/London' and nothing in the OAuth or SMTP onboarding paths ever set either, so a brand new mailbox looked deliberately placed in UTC, was compared against the differing campaign zone and dropped by the hardcoded 8am-8pm business-hours gate whenever the current UTC hour fell outside it, emptying the candidate pool and failing the campaign start, adding a migration that changes the column default and converts existing 'UTC' rows because until now no API field, dashboard control or onboarding path could set that column at all so every such row is the old default rather than a choice, adding the missing Timezone field to UpdateEmail with IANA validation so the setting the sending-behaviour UI already tells people to change is finally reachable and an unloadable zone is rejected instead of being silently coerced to UTC by the scheduler, and replacing the misleading 'no active email accounts found for campaign's email tags' response for a pool that exists but is entirely gated out with a distinct message naming the real cause, via an ErrNoEligibleMailbox that wraps ErrNoEmailAccounts so the three callers that pause a campaign on it are unaffected (#126) (#125) 2026-08-16 07:54:45 +02:00
Matthew Meszaros 3922333930 feat: run the tracking service on nats by default, kafka behind a cargo feature 2026-07-20 09:56:29 +02:00
Matthew Meszaros cc52878966 fix: apply cargo fmt to the tracking service (line wrapping in abuse.rs, config.rs, links.rs) so the Rust CI format check passes 2026-06-12 17:45:32 +02:00
Matthew Meszaros 8732805934 feat: replace signed click redirects with server-side link tickets (tracked_links store, internal resolver API, opaque /c/<id> URLs, layered anti-probe caches with miss budget and circuit breaker) removing TRACKING_LINK_SECRET entirely 2026-06-11 09:30:21 +02:00
Matthew Meszaros 2c5e8b2cbd feat: make TRACKING_LINK_SECRET a required boot-time secret on backend and tracking service with no unsigned mode and no rotation grace, so rotating the key revokes old links immediately 2026-06-11 09:00:04 +02:00
Matthew Meszaros 515efce991 feat: support TRACKING_LINK_SECRET_PREVIOUS rotation grace on the tracking service so rotating the click-signing key never breaks links in already-delivered emails 2026-06-11 08:53:29 +02:00
Matthew Meszaros 8b9277dabf feat: harden tracking service against abuse with per-IP rate limiting, prefetch/scanner filtering, URL length caps, and HMAC-signed click redirects (TRACKING_LINK_SECRET) closing the open-redirect hole 2026-06-11 08:11:05 +02:00
Matthew Meszaros d57febd1e3 ci: relax golangci-lint to real-bug rules, silence dead-code in tracking 
Go:
- Drop errcheck, unparam, prealloc, gosec, exportloopref from the
  enabled set. The legacy codebase has thousands of unchecked
  `tx.Rollback()` calls (idiomatic — Rollback after Commit is a
  no-op), prealloc suggestions the author chose not to follow, and
  gosec rules that don't apply to our control-plane code.
  Real-bug linters (govet, staticcheck, ineffassign, gosimple,
  unused, bodyclose, noctx, sqlclosecheck, typecheck) stay enabled.

Rust:
- Add #[allow(dead_code)] to `Config::from_aws` — legacy
  AWS-only loader kept as fallback while we migrate fully to the
  unified loader. Clippy's `-D warnings` was failing the build on
  the unused warning.
 2026-05-23 16:44:00 +00:00
Matthew Meszaros c12fd93def Add local Sentry logging 2026-02-14 05:49:49 +01:00
Máté Mészáros (Laptop) 7ed5719c15 Config & Deploy 2026-01-29 08:50:38 +01:00
Máté Mészáros (Laptop) 41624a6f79 Analytics & Tracking 2026-01-29 05:59:04 +01:00