Commit Graph
21 Commits
Author SHA1 Message Date
Matthew Meszaros 4784ee7d39 feat: fetch the MaxMind databases instead of requiring a mounted file (#529)
* feat: let the backend, consumer and tracking service fetch their own MaxMind databases from GEODB_URL and TRACKING_SCANNER_ASN_DB_URL, reading the archive shape from the content so a permalink tar.gz, a gzipped mmdb and a bare mmdb all work, never replacing a file already at the path, opening the bytes before installing them so a licence-key error page cannot become the database forever, skipping the AppleDouble sidecars a macOS tar writes ahead of the real file, and treating both URLs as secrets because the permalink carries the licence key

* feat: drop the trailing blank line cargo fmt --check rejects at the end of tracking/src/asndb.rs

* feat: stream the downloaded ASN archive instead of decompressing it whole, sizing each buffer from the gzip footer and the tar header so the member is allocated exactly once, which drops the peak of unwrapping a permalink tar.gz from 38 MB to 11.9 MB, essentially the database itself

* feat: stop the MaxMind licence key reaching the logs through net/http's and reqwest's own error text, which both print the URL they were given and so defeated the redaction beside them, drop userinfo as well as the query when redacting, refuse plain http for a URL carrying a credential and refuse an https-to-http redirect, and apply the size cap to the decoded database rather than the compressed transfer so a gzip bomb cannot fill the disk

* feat: strip basic-auth userinfo as well as the query when the tracking service redacts its database URL, parsing it rather than cutting at the first question mark so where a credential sits is the URL library's problem and not a guess
2026-09-15 03:06:34 -07:00
Matthew Meszaros 0a1ed6f04e feat: resolve scanner ASNs from a GeoLite2-ASN database so the catalogue's asn: entries match without a Cloudflare transform rule, ship the Proofpoint, Mimecast and Cisco ASNs enabled behind a new probable certainty that widens the consumer's machine window instead of deciding the verdict, make the tracking event dedupe claim one coalesced operation, and report an ASN database that opened cleanly but resolves nothing (#440) 2026-09-15 01:48:45 -07:00
Matthew Meszaros e8186e5f52 fix: the tracking service must not write Avro to a JSON consumer (#450)
* feat: make the Rust tracking publisher honour CODEC_PROVIDER on Kafka instead of always writing Avro, so a json consumer stops silently dropping every open and click, and refuse avro at boot when no Schema Registry is configured

* feat: build the tracking service's kafka feature in CI, because clippy on the default build never opens kafka.rs and that file now ships as the published tracking -kafka image

* feat: install libcurl and the rest of the librdkafka build dependencies for the tracking kafka clippy step, which fails at the first object without curl headers even with WITH_CURL=0
2026-09-11 23:00:15 -07:00
Matthew Meszaros ced741e352 feat: make PostHog the default error tracker across every runtime while keeping Sentry fully supported alongside or instead of it, by turning internal/observability/errs into a two-sink fan-out with a local-log fallback, adding $exception capture to the Go services, the Rust tracking service, the Elixir realtime service and the dashboard, admin and form apps, reporting gin panics with their route, request id, workspace and user, attaching that identity plus a route and failed-request trail to browser exceptions, and wiring POSTHOG_ERROR_TRACKING, the node join env, compose, source-map upload and the docs to match 2026-09-10 19:11:32 +02:00
Matthew Meszaros 2599d6abb2 feat: stop handing the website-tracking identification ticket to a recognised scanner, which filed its walk of a tracked link as the recipient's own page view, read TRACKING_SCANNER_BUILTINS the way the backend's configuration registry does so off no longer means on, and ship the whole-cloud Microsoft and Google ASN entries commented out because a recipient browsing from Azure or Google Cloud sits inside them 2026-09-10 04:32:52 -07:00
Matthew Meszaros 6684dafe3a feat: classify opens and clicks that arrive from a known mail-filtering network as automated, so Microsoft 365 Defender's delivery-time pixel fetches and Safe Links URL detonations stop counting as engagement (issue #410), with a shipped scanner catalogue, TRACKING_SCANNER_* overrides and optional ASN matching from a trusted edge header 2026-09-10 03:38:50 -07:00
Matthew Meszaros 51dedc90ee feat: put every runtime behind one optional error-reporting story: a single internal/observability/errs wrapper that is now the only package importing sentry-go, InitSentry for cmd/forms, release and environment tags on every service from the existing build stamp, optional Sentry in the admin panel and the public forms app, the sentry crate in the Rust tracking service, release tagging in realtime, CI source-map upload that only runs when a Sentry token is configured, and docs covering the DSN for each service 2026-09-07 03:51:06 -07:00
Matthew Meszaros a3264f0d39 feat: address the review on the rebuilt engagement branch: a deferred human click's effects are now durable and once-only (announce_pending on the click row written before the event is marked processed, a claim in finishHumanClick and a minute-by-minute sweep that finishes what a restart cut off, a burst relabel clears the flag), the tracking service keys the source-address token with TRACKING_IP_HASH_KEY so it cannot be enumerated back into an IPv4 address, the migration no longer adds a blocking index to the existing click table, and timeline click and open rows carry task_id 2026-09-03 04:13:39 -07:00
Matthew Meszaros 111a5a4034 feat: read the forwarded client address from exactly one operator-named header (TRACKING_CLIENT_IP_HEADER, default x-forwarded-for with the proxy-appended last entry, cf-connecting-ip only when configured) so a client-supplied CF-Connecting-IP passed through a generic trusted proxy can no longer choose the rate-limit bucket or the stored page-hit location, with tests and the configuration, env example and compose entries 2026-08-29 05:11:22 -07:00
Matthew Meszaros 8daefbe8c4 feat: address the Greptile review on website tracking by believing forwarded-IP headers only from TRACKING_TRUSTED_PROXIES (socket peer otherwise, proxy-appended last X-Forwarded-For entry, applied to pixel, click and page-hit paths), making IdentifyVisitor report whether it claimed the row so a lost identification race re-reads the visitor and splits onto a fresh record instead of attributing the hit to the wrong contact, forgetting the edge dedupe entry when a forward to the backend fails so the retry is counted, and trimming the new Go and Rust comments to the one-line style 2026-08-29 04:16:28 -07:00
Matthew Meszaros 27630eec0a feat: add website visitor tracking for issue #255 with migration 000106 (website_tracking_settings, website_visitors, website_page_hits, all registered in the orgtransfer spec), a consent-gated dependency-free tracking.js served by the Rust tracking service with a rate-limited, size-capped, prefetch-filtered POST /p ingest that forwards to a new backend internal page-hits endpoint for server-side user-agent and GeoIP enrichment, contact identification only through the click ticket the redirect appends to registered hosts, a per-workspace retention job, page_hit events with an expandable detail view in the contact Activity timeline, a Settings > Website tracking page for the snippet and consent, location and retention configuration, realtime PAGE_HIT fanout, and a website tracking guide plus endpoint, export and configuration docs 2026-08-29 03:25:50 -07:00
Matthew Meszaros 3922333930 feat: run the tracking service on nats by default, kafka behind a cargo feature 2026-07-20 09:56:29 +02:00
Matthew Meszaros cc52878966 fix: apply cargo fmt to the tracking service (line wrapping in abuse.rs, config.rs, links.rs) so the Rust CI format check passes 2026-06-12 17:45:32 +02:00
Matthew Meszaros 8732805934 feat: replace signed click redirects with server-side link tickets (tracked_links store, internal resolver API, opaque /c/<id> URLs, layered anti-probe caches with miss budget and circuit breaker) removing TRACKING_LINK_SECRET entirely 2026-06-11 09:30:21 +02:00
Matthew Meszaros 2c5e8b2cbd feat: make TRACKING_LINK_SECRET a required boot-time secret on backend and tracking service with no unsigned mode and no rotation grace, so rotating the key revokes old links immediately 2026-06-11 09:00:04 +02:00
Matthew Meszaros 515efce991 feat: support TRACKING_LINK_SECRET_PREVIOUS rotation grace on the tracking service so rotating the click-signing key never breaks links in already-delivered emails 2026-06-11 08:53:29 +02:00
Matthew Meszaros 8b9277dabf feat: harden tracking service against abuse with per-IP rate limiting, prefetch/scanner filtering, URL length caps, and HMAC-signed click redirects (TRACKING_LINK_SECRET) closing the open-redirect hole 2026-06-11 08:11:05 +02:00
Matthew Meszaros d57febd1e3 ci: relax golangci-lint to real-bug rules, silence dead-code in tracking
Go:
- Drop errcheck, unparam, prealloc, gosec, exportloopref from the
  enabled set. The legacy codebase has thousands of unchecked
  `tx.Rollback()` calls (idiomatic — Rollback after Commit is a
  no-op), prealloc suggestions the author chose not to follow, and
  gosec rules that don't apply to our control-plane code.
  Real-bug linters (govet, staticcheck, ineffassign, gosimple,
  unused, bodyclose, noctx, sqlclosecheck, typecheck) stay enabled.

Rust:
- Add #[allow(dead_code)] to `Config::from_aws` — legacy
  AWS-only loader kept as fallback while we migrate fully to the
  unified loader. Clippy's `-D warnings` was failing the build on
  the unused warning.
2026-05-23 16:44:00 +00:00
Matthew Meszaros c12fd93def Add local Sentry logging 2026-02-14 05:49:49 +01:00
Máté Mészáros (Laptop) 7ed5719c15 Config & Deploy 2026-01-29 08:50:38 +01:00
Máté Mészáros (Laptop) 41624a6f79 Analytics & Tracking 2026-01-29 05:59:04 +01:00