Commit Graph
71 Commits
Author SHA1 Message Date
Matthew Meszaros 31dabea0a4 feat: rebuild the click-without-open fix on top of the per-link click attribution from #298: a person's click now also counts as an open and a burst that withdraws the click withdraws the open it implied unless a real open is on record, routing readers ignore machine opens as the docs promised, every open gets its own log row and every open and click records the mail client or proxy, browser, device, OS, country, region and city (migration 000124: origin columns on email_link_clicks plus an email_opens table), the tracking service publishes only the address's network in a nullable client_ip field which the consumer resolves with GeoLite and drops, the contact Activity tab shows each open and the origin of opens and clicks, the campaign overview gains a who-engaged-from-where breakdown exposed as engagement in campaign analytics, live open and click events carry occurred_at, client and location, the leads table explains why an open is not always counted, both logs are pruned daily after a year, email_opens joins the export registry, the consumer reads GEODB_PATH optionally, and the guides and API references are updated (fixes #294) 2026-09-03 03:49:44 -07:00
Matthew Meszaros 99be92f159 feat: attribute every campaign click to the exact link with a per-link click log (email_link_clicks) behind the contact activity timeline, campaign live feed, recent activity and the email_clicked webhook, add per-campaign automatic UTM tagging (utm_tracking with source, medium and campaign overrides, utm_content from the link text) applied at send time to the stored ticket destination, and classify opens and clicks as machine when they arrive within ten seconds of dispatch or when several links of one email are followed within five seconds, so scanner clicks are logged but never count as engagement, fire automations or send webhooks 2026-09-03 01:05:34 -07:00
Matthew Meszaros 7a7c6051bc Merge remote-tracking branch 'origin/main' into feature/contact-segments 2026-08-30 02:07:56 -07:00
Matthew Meszaros 52916ab60d feat: verification evidence engine and Greptile fixes for #264: contact_verification_evidence ledger (migration 000111) fed by clean deliveries, human opens, clicks, replies and recipient-naming bounces from the send, tracking, reply and bounce paths, a decaying score that lets real mail outrank a probe and a newer bounce outrank older engagement, verification_confidence on every contact with a reasons list and animated Deliverability card in the contact drawer, per-organization probe breakers, undeliverable counts that ignore finished leads, tighter wiring comments, and docs 2026-08-29 23:49:33 -07:00
Matthew Meszaros daf946fc45 feat: add the add_to_segment and remove_from_segment sequence action steps, executed on both the scheduled campaign path and the instant reply path, wired into the backend and consumer 2026-08-29 23:45:05 -07:00
Matthew Meszaros 44b2c18906 feat: address review on cloud-managed mailboxes: the consumer now asks the cloud to vouch for a warmup token in a mailbox it warms (GET /pool-link/instance/mailboxes/:id/warmup-tokens/:token) and files anything unverified as ordinary mail instead of dropping on a sender-controlled header, disconnect keeps local mirrors and the link until the cloud confirms the instance is released so managed mailboxes cannot be stranded, and long narrative comments are cut to one line 2026-08-29 10:07:36 -07:00
Matthew Meszaros 2a831e9783 feat: let a linked self-hosted instance sign Google and Microsoft mailboxes in through Warmbly Cloud's own OAuth apps and send with cloud-brokered access tokens: the cloud runs the consent (pool_link_mailboxes.managed, brokered state in Redis, the existing /addresses/*/callback completes it and redirects to the instance's /cloud-oauth/done), keeps the refresh grant, mints short-lived tokens at /pool-link/instance/mailboxes/:id/token and refuses them for revoked links, removed, inactive or blocked mailboxes; the instance mirrors such mailboxes without a credential (cloud_link_mailboxes.managed), ships them to the worker as brokered so goog/msgraph init on a token source that pulls from /api/v1/internal/cloud-link/token/:id, lets the consumer ignore cloud warmup tokens for enrolled mailboxes, and can adopt mailboxes connected directly on the workspace; Add account shows the cloud path and the adoptable list, and the Warmbly Cloud guide documents the model 2026-08-29 09:50:52 -07:00
Matthew Meszaros b75fdcf86c Merge remote-tracking branch 'origin/main' into feature/self-hosted-warmup-pool-access
# Conflicts:
#	internal/scheduler/warmup_scheduler.go
2026-08-29 07:10:38 -07:00
Matthew Meszaros 37b60b59d3 feat: let a self-hosted instance warm its mailboxes in the hosted pool: device-code link approved at /connect, instance-token API that enrolls SMTP/IMAP mailboxes as warmup-only accounts (no history import, non-warmup mail dropped), free for 10 mailboxes and unlimited on the seeded $15 pool plan, tier fallback to proven healthy mailboxes when a pool runs thin, local warmup stands down for enrolled mailboxes, Settings > Warmbly Cloud step flow and linked-instances page, docs guide, marketing copy, and fix SetWarmupLifecycle re-reading the row with an org-scoped lookup so every warmup start/pause returned 404 2026-08-29 07:09:04 -07:00
Matthew Meszaros 84700bae8d Merge remote-tracking branch 'origin/main' into fix/issue-241
# Conflicts:
#	cmd/consumer/main.go
#	docs/content/docs/guides/deliverability.mdx
#	internal/app/auth/login_risk.go
#	internal/app/auth/provision.go
#	internal/app/contact/import.go
#	internal/app/orgrisk/service.go
2026-08-28 23:06:07 -07:00
Matthew Meszaros a5ea55bba5 feat: give a suspended workspace a way back, because risk_state was a one-way door: the derived band is no longer pinned at suspended by the UPDATE in pg_org_risk, an operator's decision is now an explicit risk_override that outranks the score and survives every later detector write until it is lifted, the one-shot detectors (signup origin, import list quality, repeated sign-in anomalies) file findings with a 30-day expiry that a six-hourly consumer sweep retires so a score falls on its own, migration 000104 backfills that expiry onto findings already on file, and four admin endpoints plus an Abuse posture panel in admin/ let an operator finally read the evidence the customer endpoint withholds, retract a finding, pin a band and lift the pin; also stops the reviewing admin's identity reaching the tenant's own audit feed (which resolves an actor to a name and email) by recording the platform as the actor there and the operator in the admin trail, and stops risk_signals riding along in a customer-downloadable org export 2026-08-28 22:42:57 -07:00
Matthew Meszaros 2b25e32f6b feat: judge an organization's abuse posture on what it did rather than on what it looks like, so an agency opening client workspaces from one office under one operator identity and connecting the mailboxes in an afternoon lands on watch instead of restricted: findings now carry a circumstantial/substantive class, shape is capped at 40 points and can never move a band past watch, the signup address and identity clusters count once as one family, the signup finding is substantive only when the domain is really disposable, list_quality retracts on a later clean import, and the nightly sweep gains a recipient-outcome pass scoring 30-day bounce and complaint rates against the provider bands so a band still has evidence it may act on 2026-08-28 22:41:44 -07:00
Matthew Meszaros b8ddf2bc83 chore: merge main, keeping both background jobs 2026-08-28 12:12:51 -07:00
Matthew Meszaros e31286d973 feat: retract findings that no longer hold, exclude private IPv6, and fix the containment operator that let loopback through: a cluster ageing out of the lookback or a mailbox burst ending left its weight on the workspace permanently, so the score could only ever climb, and each detector now clears itself from organizations that stopped matching before recording the current ones; the private-address filter covered only IPv4, so an IPv6-only self-hosted install would have clustered as a ring; and it used << rather than <<=, which is strictly-contained and therefore excludes an address from the prefix describing exactly it, so ::1 was never inside ::1/128 2026-08-28 12:12:28 -07:00
Matthew Meszaros 9f23c663c4 feat: give a cold mailbox a rotation lifecycle so a tired one can rest (#237)
* feat: give a cold mailbox a rotation lifecycle so a tired one can rest and come back, instead of running at full volume until a hard band trips: send_lifecycle is warming, active, resting or reserve and decides whether cold sender resolution offers the mailbox at all, which is a different axis from risk_band deciding which worker and IP host it, so a resting mailbox is still a clean-band mailbox that keeps its warmup traffic and its reputation; the hourly rebalancer rests a mailbox at throttled and worse but never at watch, since watch is defined as the band that changes nothing a customer can feel and leaving cold rotation is very much something they feel, and a rested mailbox returns only after three clean days so one good hour cannot bounce it back to full volume; reserve is the owner's hold and is never overridden, the default is active so no existing mailbox changes on deploy, and the state never travels in a workspace archive because it is this instance's decision about sending it watched

* feat: stop a query error re-admitting rested mailboxes, make probation measure healthy time, and rotate the candidate window so no mailbox starves: sendLifecycles returned a nil map on failure and an unresolved state reads as active, so one bad query quietly put every resting and reserved mailbox back into cold rotation, and the gate is now applied only when the states were actually read, with the skip logged rather than silent; ReadyToResume measured total time resting, so a mailbox that sat unhealthy for three days resumed on its first healthy tick having served no clean time, and an unhealthy evaluation now restarts the streak; and ordering candidates by send_lifecycle_since put every never-moved mailbox equal-first, so on an install with more than one page of them the same page was re-examined forever, which a checked-at stamp and its index fix
2026-08-28 12:08:52 -07:00
Matthew Meszaros 02a1ab524a feat: sweep for cross-account patterns nightly, since every other control watches one subject 2026-08-28 11:58:49 -07:00
Matthew Meszaros 8953b2a132 chore: delete repository junk that was never referenced by anything: the paseo worktree-tool config, the root .astro type output an astro run from the repo root left behind, the empty schema.sql, the stray root package.json plus pnpm-lock.yaml from an accidental pnpm add motion (every frontend tree owns its own manifest and lockfile, and CI only ever reads those), the zero-byte cmd/consumer/envsample, the empty web CampaignSearchProvider.tsx, six Go files holding nothing but a package clause, and models.WMailAdd which had no callers; root .gitignore now covers each of them so they cannot drift back in 2026-08-26 04:55:45 -07:00
Matthew Meszaros 15e139e15d feat: stop a campaign email going out twice when the progress write after dispatch is lost: a step is now RESERVED before its SEND_EMAIL reaches the bus (migration 000093 adds campaign_contact_progress.dispatched_at + dispatch_task_id, and ReserveSend takes the claim and the day's counters in one transaction) and routing treats a step as attempted on sent_at OR dispatched_at, so a crash or a failed stamp in the dispatch window can no longer read as "never sent" and email the same person again; the ON CONFLICT claim is exactly-once so two ticks racing the same pair cannot both send (the loser ends skipped_duplicate), the stamp is retried and escalated to the campaign feed instead of warned and swallowed, HandleEmailSent repairs a lost stamp from the worker's own confirmation, ReleaseSend gives a reservation back only when the command provably never left (a publish failure is ambiguous via ErrSendDispatchUnknown and keeps it), and StartStuckSendReclaimer walks back a reservation nobody answered after 30 minutes so a worker that died mid-send cannot park a lead in flight forever; live-tested in TestLiveLostProgressWriteDoesNotResend, TestLiveDispatchedSendIsNeverOfferedTwice, TestLiveConcurrentTicksSendOnce, TestLiveStuckDispatchIsReclaimed, TestLiveReclaimBelievesADeliveredSend and TestLiveInFlightSendIsNotOfferedAgain 2026-08-24 09:15:06 -07:00
Matthew Meszaros 8e0043878c feat: consume the worker's EMAIL_SENT and EMAIL_FAILED results in the consumer, which until now dropped them with a warning so a send the worker could not complete stayed stamped as sent and the lead sat at 'processing' forever: a failure now marks the task failed, walks the step back, gives the day's counters back, writes a red entry to the campaign activity feed, reopens a campaign that completed while the send was in flight, marks the lead failed after five attempts, and routes a recipient refused at RCPT into the bounce pipeline instead of retrying it; a one-off compose or reply failure is pushed to the mailbox owner; covered by an opt-in live test against Postgres and documented in the events page, the campaign guide, the contacts API reference and the agent notes 2026-08-23 10:18:40 -07:00
Matthew Meszaros a75ea012a0 feat: import a mailbox's recent history on connect and govern sync by fair use: a backfill on every provider (newest first, inside an operator-editable window and cap, resumable through a durable per-provider cursor relayed as SYNC_STATE), a per-mailbox sync governor with priority, live and backfill lanes on shared Redis windows that defers over-budget mail with the cursor held instead of dropping it and only deactivates a mailbox for a flood or chronic daily overage, sync.* budgets on the admin instance settings shipped inside ADD_EMAIL, saved IMAP folder cursors and last_synced_at finally written, a Sync card in the mailbox drawer fed by GET /emails/:id/sync, and docs 2026-08-18 09:09:52 -07:00
Matthew Meszaros 0ae4db2c41 feat: make self-hosted auth work without a mail relay by rewriting the platform SMTP transport with real AUTH and TLS (it did neither, so SMTP_USERNAME/SMTP_PASSWORD were dead and every documented relay was unreachable), adding MAIL_TRANSPORT=smtp|log|ses with a log transport that prints codes so a fresh install can sign in with no relay, demoting the emailed login code to AUTH_LOGIN_CODE=always|new_device|off (off on self-host, per NIST SP 800-63B and OWASP ASVS), claiming the first owner through a single-use setup link or WARMBLY_BOOTSTRAP_* instead of register-then-psql, deriving every emailed URL from APP_URL rather than a hardcoded app.warmbly.com that leaked live reset tokens to the vendor, fixing the confirm hooks that read path params against paramless routes and broke login, register and reset confirmation in the dashboard everywhere, adding generic OIDC with PKCE, one-time state, verified nonce and (issuer,subject) identity binding, enforcing 2FA on the social paths that skipped it, adding a per-IP limiter and trusted-proxy handling to the unthrottled auth group, refusing boot on the published default secrets, and dropping mailpit from the default stack (#99) 2026-08-14 14:57:09 +02:00
Matthew Meszaros bb040828b1 feat: rework automation AI steps to mirror campaign step types with one agentic AI step plus an AI switch, per-case switch handles on the canvas, a Stop node, and a step-type add menu 2026-07-22 10:58:35 +02:00
Matthew Meszaros 93b64e2afa feat: add shared run time listing and arg resolving for ai agents 2026-07-21 17:35:11 +02:00
Matthew Meszaros 8055c2f05f Merge remote-tracking branch 'origin/main' into self-host/local-stack
# Conflicts:
#	cmd/backend/main.go
2026-07-20 11:53:21 +02:00
Matthew Meszaros 543d97f6f3 feat: wire the no-cloud providers into the consumer and worker 2026-07-20 09:56:29 +02:00
Matthew Meszaros a534ef1908 feat: notification email channel becomes digest-first: email-channel notifications queue as pending rows (migration 000076 adds group_key/email_state/email_due_at/email_attempts) with a due time from a new per-user email_digest cadence (instant/smart 15m/hourly/daily, security sign-ins always immediate), a 30s flush loop with SKIP LOCKED claims bundles a user's pending rows into one digest email and coalesces org-shared group_key events into a single email with every recipient in To, reading a notification in-app cancels its pending email, and NotifyOrg targets only members holding a permission (Slack fires once per group) with new producers: dead-worker downtime to manage_emails members (SetNX-deduped per incident), trial expiry to manage_billing members via new billing_alert category (replacing the direct owner email), and invitation accepts to manage_team members via new team_activity category 2026-07-20 07:16:25 +02:00
Matthew Meszaros 053ca6a71d feat: usage-based AI credit system — every AI call reserves its flat minimum then settles the real token cost per model (light 1500/standard 400 tokens per credit, drain-to-zero overage settle wired into writing assistant, reply drafts, dashboard agent, inbox agent, research, automation nodes, and campaign switches), org spend controls in a new org_ai_settings table (day/week/month hard limits enforced in Consume, low-balance alert threshold, auto top-up config), a credit-watch monitor hook that fires BILLING_CREDITS_LOW realtime alerts once per day and buys the configured pack off-session via a new Stripe AutoTopUpCredits (idempotent on the PaymentIntent, bounded per month, Redis-locked), GET /subscription/credits/usage + GET/PATCH settings endpoints, and an AI usage & spend controls billing card (spend vs limits, 30-day chart, per-feature and per-model breakdowns, limits/reminder/auto-top-up form) with a realtime low-credit toast 2026-07-18 08:39:41 +02:00
Matthew Meszaros cd5e1e9cf7 Merge pull request #72 from warmbly/research/warmup-abuse-cold-effectiveness
feat: deliverability research + observe-only SPF/DKIM/DMARC auth state and watch-band cold throttle
2026-07-17 05:29:58 +02:00
Matthew Meszaros 29e3bb3a64 feat: drop the legacy OPENAI_*/ANTHROPIC_API_KEY/AI_LOCAL_MODEL env fallbacks (no prod deployments exist): AI_PROVIDER/AI_API_KEY/AI_MODEL/AI_BASE_URL/AI_FREE are the only AI vars, empty provider means openai, the anthropic preset and writing client key off AI_API_KEY, and the warmup Batch generator runs only when the selected provider is OpenAI 2026-07-16 06:39:06 +02:00
Matthew Meszaros 917451b105 feat: select the AI provider via AI_PROVIDER + AI_* vars in the backend and consumer (legacy OPENAI_*/ANTHROPIC_API_KEY still honored as fallbacks), logging a clear message when the provider is misconfigured 2026-07-16 06:14:19 +02:00
Matthew Meszaros d94bd70f88 feat: wire AI_LOCAL_MODEL into the backend and consumer provider config and pass the agent voice preamble into the dashboard agent service 2026-07-15 18:51:51 +02:00
Matthew Meszaros 78ff73e45c feat: ground inbox-agent reply drafts in the counterpart contact's CRM record - the inbox agent now folds the replying contact's name, company, and known custom fields into the draft prompt (mirroring the manual unibox reply draft) via an optional ContactReader wired from repository.ContactRepository in both the backend and consumer, best-effort so a lookup miss just drops the grounding, so suggested replies reference what Warmbly already knows about the person 2026-07-14 07:29:56 +02:00
Matthew Meszaros 56de65380b feat: inbox agent that drafts a suggested unibox reply on inbound human replies for human approve/edit/discard - paid + per-org opt-in (organizations.inbox_agent_enabled) feature where the consumer's reply hook, on a non-automated reply, detaches a goroutine (panic-contained, never blocks ingest) that checks entitlement, dedupes via ai_thread_drafts partial unique indexes (one pending per thread, unique source_message_id), pre-checks balance, grounds a reply in the thread history + org voice + skills via generation.Provider.Complete, reserves a draft row then charges 5 credits (idempotency inbox_agent:<draft.ID>, row unwound on a fresh context if the charge fails so no unpaid draft lingers), and emits an org-scoped AI_DRAFT_READY event gated on access_unibox; the agent never sends - only a human POST to /unibox/agent-drafts/:id/approve sends through the normal reply path, claiming pending->approved before send with an approved->pending revert on send failure, alongside list + discard endpoints; plus the unibox awaiting_agent_draft badge + agent_drafts search scope, an AgentDraftCard in the thread view with inline-editable approve-and-send/discard, the workspace settings opt-in toggle, CanUseInboxAgent paid gate, migration 000065, and docs (inbox-agent guide, endpoints, realtime) 2026-07-14 06:13:27 +02:00
Matthew Meszaros a067bf5922 feat: AI action nodes in automations - ai_classify/ai_extract/ai_generate run one generation.Provider completion per node over the event data and merge the result back as a variable (ai_class, output_keys[], ai_text) so downstream conditions branch on it, charging 1 credit per node with a run_id+node_id idempotency key, refund on provider failure, out-of-credits fails only that node while the run continues, and a per-automation ai_credit_failures counter (migration 000064) auto-pauses a flow after 20 consecutive credit misses; classify/extract run deterministic (temperature 0), dry-run executes AI nodes for real and is charged but never touches the auto-pause lifecycle, provider + credit ledger wired into both backend and consumer integration services, plus the reply-classifier Layer 3 ported off its bespoke OpenAI client onto the platform provider via an injected ModelClassifyFunc (platform-paid, deterministic), a CompletionRequest.Temperature primitive threaded through the OpenAI and Anthropic providers, the flow builder palette/config panels/validation for the three AI steps, and a new AI-steps-in-automations guide 2026-07-14 05:30:55 +02:00
Matthew Meszaros d6b7885e8b feat: mobile push notifications end to end - APNs provider-token client, device_tokens table with session-scoped register/delete endpoints, a push channel in notification preferences (web + iOS toggles), and Redis-backed immediate-then-digest batching (first event pushes now, bursts summarize when the 5h window closes) wired in backend and consumer, with iOS registration/badge sync and docs for the channel, endpoints, and APNS_* deploy env 2026-07-13 16:11:15 +02:00
Matthew Meszaros 6bc2073379 feat: honor CODEC_PROVIDER=json across backend and consumer (previously worker-only) and round-trip untyped event bodies into their typed payloads in both dispatchers - the worker command/result envelopes carry any-typed bodies that Avro cannot serialize, so worker messaging only works on the JSON codec; also wires the credentials encrypter into both mains 2026-07-11 17:31:02 +02:00
Matthew Meszaros eb0a5654fd feat: persist the opaque per-folder Graph delta cursor in a new email_delta_links table via a repository and a GRAPH_DELTA_UPDATE consumer handler wired into the consumer service 2026-07-04 08:07:49 +02:00
Matthew Meszaros ffe823cfa8 feat: persist and surface per-mailbox SPF/DKIM/DMARC authentication state via an hourly observe-only consumer sweep, threaded through the mailbox list/detail API and documented in the deliverability guide and mailboxes API reference 2026-07-01 02:32:23 +00:00
Matthew Meszaros c5dfa5e4e7 feat: remove the HTTP-request action from campaign steps and automations in favor of signed webhooks, keep fire_event for custom payloads, and drop the now-unused outbound quota plumbing 2026-06-15 08:11:35 +02:00
Matthew Meszaros 20935ef061 feat: add a nil-safe per-org daily outbound-action quota (Redis daily counter, anti-abuse ceiling on the HTTP-request automation node, wired in both backend and consumer, fail-open) to bound webhook relay abuse 2026-06-13 13:41:20 +02:00
Matthew Meszaros 7901e9a92c feat: wire the Pub/Sub realtime provisioning, the instant-branch automation runner, and the campaign-chain reconciler into the backend and consumer binaries 2026-06-13 06:27:06 +02:00
Matthew Meszaros ba89a932cd feat: run the label_email action on reply — campaign instant and scheduled paths via advanced LabelThread/LabelLatestThreadForContact, automations via execNativeAction reading thread_id and _user_id from the reply payload — and move the native-actions adapter to a shared package wired into the consumer too so reply/bounce/warmup automation actions stop silently failing 2026-06-12 16:44:16 +02:00
Matthew Meszaros ba3ab98e1e feat: bridge realtime events over Redis pub/sub when Google Pub/Sub is unconfigured (RedisBus publisher in backend/consumer plus Realtime.Redis.EventSubscriber and a shared EventBroadcaster) so dashboard live updates and presence-driven collaboration actually fire in local dev 2026-06-12 04:49:58 +02:00
Matthew Meszaros 160dc0bc76 feat: implement the coming-soon notification delivery channels — Email (SES/SMTP to the account email) and Slack (posts to the org's connected workspace via a new integration NotifySlack), wired in both backend and consumer with per-channel gating, real toggles replacing the coming-soon labels, and updated docs 2026-06-11 12:21:48 +02:00
Matthew Meszaros abdfd05c34 feat: org-scope realtime events (inbox, campaign, tracking, account health) and emit EMAIL_SENT/EMAIL_REPLIED/EMAIL_DELETED pulses so the whole team's dashboard updates live 2026-06-11 07:43:57 +02:00
Matthew Meszaros 4013b33008 feat: expose security notification APIs
Wires notification and two-factor services into the backend and consumer, adds the authenticated notification feed/preferences endpoints, and exposes the public 2FA login verification endpoint.
2026-06-08 15:04:47 +02:00
Matthew Meszaros 51c99458ad feat: fire instant actions from tracking events 2026-06-07 17:24:32 +02:00
Matthew Meszaros da82993846 feat: add campaign workflow backend support
Add sequence action-node storage and execution, template conditional rendering, lead progress state, profile updates, webhook fan-out throttling, and supporting repository fixes.
2026-06-06 07:49:52 +02:00
Matthew Meszaros 35fbf73347 feat: persist warmup engagement dwell
Store delayed warmup engagement actions in Postgres and drain them from the consumer so read, important, and star actions survive worker restarts.

Keep foldering and spam rescue immediate while routing delayed actions to the mailbox's current worker at fire time.
2026-06-03 11:14:02 +02:00
Matthew Meszaros 59d253ef1d feat: harden warmup action handling
Wire warmup health webhooks in the consumer and log skipped warmup actions when a recipient mailbox has no assigned worker.

Run foldering and spam rescue immediately while keeping only low-stakes engagement signals behind the dwell delay.
2026-06-03 06:40:35 +02:00