Commit Graph
70 Commits
Author SHA1 Message Date
Matthew Meszaros 52916ab60d feat: verification evidence engine and Greptile fixes for #264: contact_verification_evidence ledger (migration 000111) fed by clean deliveries, human opens, clicks, replies and recipient-naming bounces from the send, tracking, reply and bounce paths, a decaying score that lets real mail outrank a probe and a newer bounce outrank older engagement, verification_confidence on every contact with a reasons list and animated Deliverability card in the contact drawer, per-organization probe breakers, undeliverable counts that ignore finished leads, tighter wiring comments, and docs 2026-08-29 23:49:33 -07:00
Matthew Meszaros a02ff7c936 feat: address verification overhaul for #264: MillionVerifier as a pay-as-you-go integration plugin with org-sealed key and automatic fallback to the built-in check, built-in prober gains domain cache, Microsoft/Yahoo fingerprinting, MX fallback, disposable/role sub-statuses and a self-check breaker, imports and POST /contacts accept verification results in any known provider vocabulary with auto-detected columns, verdict provenance and expiry columns (migration 000110), campaigns park at paused_undeliverable with re-verify/send-anyway instead of finishing, POST/GET /contacts/verification bulk actions and overview, launch gate override via acknowledge_list_risk, animated verification marks, banner and settings card in the dashboard, and docs 2026-08-29 23:11:22 -07:00
Matthew Meszaros 6bb2a3d0c2 Merge remote-tracking branch 'origin/main' into fix/issue-245-resolution 2026-08-28 22:51:37 -07:00
Matthew Meszaros 4ead1b200f feat: answer the review on #251 by filing a signup's throwaway-domain finding separately from its soft ones so the aggregate score no longer carries one class, and by measuring import quality across everything a workspace has imported instead of the newest file, so a small clean upload cannot retract a large bad list whose addresses are still stored while the finding still fades as good data outweighs it, with the running counts kept as evidence on the finding and a finding filed before those counts existed folded in as the smallest list that could have flagged it 2026-08-28 22:51:23 -07:00
Matthew Meszaros ec7b7bbed3 feat: count anchors and bare URLs together instead of taking the larger of the two, so three labeled links plus three written-out URLs no longer score a clean 100, clear the editor's pending flag when the draft is emptied mid-request since the cancelled request's finally can no longer do it, fail the preflight content check when a campaign's attachments cannot be read rather than scoring as if there were none and passing copy the send path then warns about, and condense the new comments to the one-line form the repo convention asks for 2026-08-28 21:59:53 -07:00
Matthew Meszaros 4ddcb24bcd feat: make the campaign content check actually fire, fixing a link cap that counted zero links on every HTML email because the URLs live in href attributes that tag-stripping discards, a preflight that scored wait and action nodes as copy and so failed every campaign using one at 55/100 because GetSequencesByCampaignID never selected the kind column, a content_warning log level of "warning" that never matched the dashboard's "warn" amber tier, an unvalidated min_content_score an API caller could set to 5000, attachments the send path scored but preflight ignored, and a docs claim of live editor scoring that was really a manual button 2026-08-28 21:54:29 -07:00
Matthew Meszaros 5e0916b266 feat: count only readable messages in the warmup batch failure suffix, so a blank provider error entry no longer renders "(+1 more)" and sends the operator hunting for a second reason that does not exist, and condense the new BatchState, batchFailureReason and failed-batch comments to the one-line form the repo convention asks for 2026-08-28 20:03:07 -07:00
joao-crm 0e4a696c2f feat: record the provider's own reason when a warmup generation batch fails as a whole, instead of leaving the job saying only "batch failed": a batch that never ran has no error file to read the refusal out of, so the message OpenAI puts on the batch itself is the only account of why, and GetBatch now returns it as part of a BatchState the poller writes into the job row 2026-08-29 01:12:22 +00:00
Matthew Meszaros 78ec522758 feat: challenge a sign-in that could not physically be the same person, which a remembered device cannot catch because an attacker holding the cookie looks familiar: sign-ins were only ever a Redis device fingerprint with a TTL, so nothing durable recorded WHERE an account was used and there was nothing to compare a new one against; login_history keeps a bounded window per user, the geo lookup gains the coordinates the city database already had, and a journey implying more than 1000 km/h forces the emailed code even on a known device; it declines to judge a first sign-in, a hop under 500 km, an interval under two minutes or a missing position, because a false challenge locks a real person out of their own account and that is the worse failure, and repeated anomalies inside a fortnight reach the workspace posture while one odd trip does not 2026-08-28 12:08:17 -07:00
Matthew Meszaros 8ef6ee5545 feat: measure an uploaded list at import, and say what it looks like (#236)
* feat: refuse a launch whose list is known to be largely undeliverable

* feat: count the deliverable audience in SQL instead of subtracting two overlapping totals, surface the unverified advice, and stop a query error skipping the gate silently: a contact can be both suppressed and unsubscribed, so Total minus both removed it twice and inflated every share computed against the remainder, which for a fully overlapping list produced a negative deliverable clamped to zero; the unverified-list branch set a summary and remediation but neither Block nor Warn, and a preflight report only surfaces checks that did not pass, so that advice could never be displayed; and a failed audience query was treated as a pass, which is fail-open on a safety gate without even a log line saying the check did not run

* feat: measure an uploaded list at import time and report what it looks like

* feat: document the import assessment and show it in the wizard

* feat: count only sendable leads as invalid, and stop guessing which column held a malformed address: an invalid lead that was also suppressed sat in the numerator while Deliverable excluded it from the denominator, so a campaign whose sendable list was clean could project above 100% and be refused; every verification count now shares the deliverable predicate, and a row whose MAPPED address will not parse is recorded as malformed directly rather than scanning other cells for an at sign, which could pick up a notes field
2026-08-28 11:50:48 -07:00
Matthew Meszaros 1e37fb6aa3 feat: capture and score a signup's origin instead of discarding it (#234)
* feat: give an organization one fused abuse posture, because every existing control watches a single subject and an actor slightly wrong on several axes sits under all of them: organizations gains risk_state, risk_score, risk_reason and an append-only risk_signals evidence blob, modelled on the warmup participant health machine that already works rather than a second vocabulary for the same idea; restricted cuts per-mailbox cold volume to a quarter and forces the free warmup pool so a risky tenant cannot spend the paid pool's shared reputation, suspended stops sending at the send gate, and watch deliberately changes nothing a customer can feel so evidence accumulates before anything is taken away; an operator's suspension outranks the derived band so a detector clearing cannot release a workspace a human suspended, transitions ride the audit spine to every teammate's dashboard, a banner says which limit is active and why rather than letting volume drop silently, and the posture never travels in a workspace archive since it is one platform's verdict reached from evidence the destination never saw

* feat: make the suspension actually stop sending, and emit the audit transitions the PR claimed: emailsend.SendEmail is only the manual and API path, so campaign and warmup sends went nowhere near the gate and a suspended workspace kept sending on its schedule, while the restricted multiplier floored every mailbox at one a day which quietly turned suspension into a trickle rather than a stop; the campaign scheduler now defers the whole campaign with a reason and the warmup task skips as org-suspended, since warmup is outbound mail from the same domains; separately the band change emitted no audit entry at all despite the entity type and the frontend spine entry both existing, so no banner moved for a teammate and there was no trail of who was restricted when, and only a real transition is logged so a detector re-recording the same finding cannot fill the feed; one of my own live tests also asserted how far out a slot lands, which depends on the hour the suite runs, and now asserts the property it was about

* feat: capture and score a signup's origin instead of discarding it

* feat: document what a signup records, and prove the origin write is on the account path with a seam test rather than only testing the scorer

* chore: drop the em dashes and trim the comments the review flagged
2026-08-28 11:26:44 -07:00
Matthew Meszaros 1b8c51ad06 feat: read spam complaints and domain-auth refusals, the last two open delivery signals (#232)
* feat: close the two halves of the delivery-signal loop that were still open, complaints and domain-auth refusals: a spam complaint arrives as mail long after the send succeeded, and nothing read those reports, so the strongest negative signal a sender gets never reached the complaint rate, the suppression list or the breaker; internal/pkg/arf parses RFC 5965 feedback reports worker-side alongside the existing DSN path, takes the LAST Message-ID because the reported mail's headers follow the report's own, and records only abuse-type reports so a not-spam report cannot be inverted into a complaint; separately a receiving server refusing mail because the SENDING DOMAIN failed its authentication (5.7.515, 5.7.26) was classified as SERVER_UNREACHABLE and retried forever, and it is now its own hard code that blames the domain rather than the address, leaves the recipient unsuppressed, and brings that domain's DNS re-check forward so the sweep confirms the verdict the send gate acts on

* feat: make the complaint path actually reachable, and stop it trusting the report: selectTextParts only ever picked text/plain and text/html, so an RFC 5965 report's message/feedback-report and message/rfc822 parts never reached the worker and both Feedback-Type and the reported Message-ID were invisible, which would have left this feature inert and has been quietly weakening DSN parsing too; report parts are now selected as plain text; the complainer is taken from the RESOLVED SEND rather than the report body, because a report is unauthenticated mail anyone able to reach the mailbox could forge and honouring the address it names would let a forger suppress a contact the send never went to; Original-Mail-From is no longer read as the complainer since that address is the sender; and a domain-auth refusal now releases the reservation instead of spending one of the lead's attempts, because the recipient received nothing and the problem is the mailbox's domain, so another mailbox in the pool picks the lead up

* feat: fix the same missing-parts gap in the Gmail adapter, and say plainly that Microsoft Graph cannot see reports at all: goog.extractBody took only text/plain and text/html exactly as the IMAP path did, so a feedback report synced from Gmail was as invisible as one synced over IMAP, and the tests that would have caught either called the parser directly rather than going through the adapter where it actually broke; the new tests exercise that seam, and Graph returns one rendered body with no parts so reports there are undetectable without a MIME fetch that is not built, which the docs now state rather than implying full coverage
2026-08-28 11:02:45 -07:00
Matthew Meszaros ca5acb5aaf feat: score campaign copy as sent, not just as written (#229)
* feat: score campaign copy where it actually matters, on the message the recipient receives rather than only the template in the editor: warmlint.Score already ran at save time through /templates/score, but nothing checked the copy after merge fields, spintax, A/B selection and AI blocks resolved, which is exactly where a clean template becomes Hi comma or picks the one spammy spintax branch; the send path now scores the rendered message and writes one campaign-feed warning per step per day rather than one per recipient, RunPreflight gains the same check across every step, warmlint gains image-heavy, many-images and attachment heuristics, and the whole thing is advisory behind Settings > Sending > Content checks; the launch dialog also finally calls POST /campaigns/:id/preflight, whose report had no UI at all and so never reached the person about to send

* feat: fix three defects the review found in the content lint: a preflight run that could not read the campaign's steps left the worst score at 100 and reported the check as PASSED, which is the one outcome a check that did not run must never produce, so it now reports failed with the reason; the send-time path read organization settings while preflight read campaign-effective ones, so a campaign that turned the check off or moved its floor was still warned, and EffectiveSettings is now exposed and used by both; and the check-then-insert dedupe let concurrent recipients of the same low-scoring step each find nothing and each write, so it becomes one advisory-locked conditional insert proven by twelve racers producing exactly one feed entry
2026-08-28 10:29:39 -07:00
Matthew Meszaros 9ab42cfd65 feat: build the browser half of social sign-in, which was never wired: GOOGLE_CLIENT_ID was read at boot and made the login screen render a Google button, but the button opened a popup at /auth/google/login which no route served, and authService.GoogleAuth/AppleAuth had no caller anywhere in the codebase; internal/app/socialauth now runs Google and Apple through the flow generic OIDC already used (one-time state, PKCE, nonce, id_token verified against the provider JWKS, identity keyed on issuer and subject, JIT provisioning, the ban and 2FA gates), the redirect URI defaults to API_PUBLIC_URL/v1/auth/<provider>/callback and is logged at boot because registering the dashboard origin instead is the mistake that produces a valid OAuth client and a dead button, /auth/config advertises only providers the backend can actually complete, the SSO landing page no longer swallows a two_fa_required response, and OIDC_PROVIDER_NAME finally reaches the button it documents 2026-08-28 01:33:08 -07:00
Matthew Meszaros 5241256274 feat: classify OAuth token refusals by what the provider actually said instead of calling every non-5xx refusal a dead grant: internal/pkg/mailauth reads the RFC 6749 error code, so a revoked or expired grant (invalid_grant and the interaction family) is an authentication error the customer must reconnect, while a 429 from the token endpoint, a 5xx, an unrecognised code and above all invalid_client stay retryable, because an expired app secret returns invalid_client for every Outlook mailbox on the install at once and deactivating all of them into a re-consent that cannot work either is a far worse outage than retrying until it is rotated; the Graph client logs the provider's own error code and description next to the verdict, and the tests drive real refusals through the real oauth2 transport on the fetch, list and send paths 2026-08-27 20:35:01 -07:00
Matthew Meszaros 0a8c3314d0 feat: stop one undeliverable address from wedging a whole campaign, and stop the pre-send verifier from inventing the verdict that caused it: campaign routing in FindNextRoutedPair now excludes contacts whose address verification refused them (invalid, or risky with the campaign's send-to-risky toggle off) because the pre-send gates in campaign_task.go skip those without recording progress, so the finder re-served the same lead every tick and the campaign never reached the healthy leads behind it; the SMTP prober no longer reads every 5xx reply to RCPT TO as a dead mailbox, since Postfix defers HELO/sender/policy rejections to RCPT time and the old default greeting of localhost made them arrive as 504 5.5.2 Helo command rejected against the recipient, so a 5xx now becomes invalid only when the reply names the recipient and degrades to unknown otherwise; the HELO name falls back to the APP_URL host and the probe is skipped outright without a public FQDN; migration 000095 resets the verdicts the corrected classifier would no longer reach so poisoned contacts are re-verified; and skipped leads now report as undeliverable in the Leads view and in the campaign completion log instead of sitting at Queued forever 2026-08-26 06:03:20 -07:00
Matthew Meszaros 8953b2a132 chore: delete repository junk that was never referenced by anything: the paseo worktree-tool config, the root .astro type output an astro run from the repo root left behind, the empty schema.sql, the stray root package.json plus pnpm-lock.yaml from an accidental pnpm add motion (every frontend tree owns its own manifest and lockfile, and CI only ever reads those), the zero-byte cmd/consumer/envsample, the empty web CampaignSearchProvider.tsx, six Go files holding nothing but a package clause, and models.WMailAdd which had no callers; root .gitignore now covers each of them so they cannot drift back in 2026-08-26 04:55:45 -07:00
joao-crm ac40eca920 feat: read a fully failed batch's error file instead of reporting "empty output file id", so a generation job that had every request refused names the provider's actual reason in the admin panel rather than an internal guard message 2026-08-25 20:46:27 +00:00
joao-crm 93a2fa0962 feat: reflect the structured-output schema inline and anonymously so OpenAI stops rejecting every warmup generation request with "$ref cannot have keywords {'$id'}", which made AI warmup content impossible to generate on any install 2026-08-25 20:46:27 +00:00
Matthew Meszaros cbf4190f42 feat: make a custom tracking domain verifiable instead of permanently "Pending DNS": the CNAME target is now this install's TRACKING_DOMAIN rather than a hardcoded t.warmbly.com that resolves nowhere, matching is exact on the label boundary (or on shared addresses, so a provider-flattened CNAME stops reading as no record at all) instead of strings.Contains, and every outcome carries the reason plus what DNS actually returned, including when the tracking host the customer is told to point at has no record of its own; a pasted URL is normalized to its host and a malformed one is rejected up front instead of saved and left pending forever; only a VERIFIED mailbox domain is used at send time with the shared host as the fallback and a campaign-feed entry saying why; pixels and click tickets are built from the configured host, and with none configured mail ships untracked rather than carrying links to another deployment's tracking service; adds GET /emails/:id/track and POST /emails/:id/track/verify plus an hourly re-resolution sweep so a record that propagates later starts being used and one that breaks stops routing links; and scopes the tracking-domain write by organization like the read, which also fixes GET /emails/:id passing a user id to an org-scoped query and 404ing for every caller 2026-08-24 09:02:29 -07:00
Matthew Meszaros 3739a36b67 feat: enforce the persisted SPF/DKIM/DMARC state as a real cold-send and warmup gate behind a 72h grace clock and an operator toggle, after first fixing the DMARC organizational-domain fallback in dnsauth so a dedicated sending subdomain covered by its parent's record stops reading as unauthenticated, adding auth_state to the four mailbox loaders that never selected it (which would have made the gate dead code), stamping auth_failing_since on entry to failing so a resolver hiccup can never stop a campaign, notifying the org on that transition, and reporting an all-gated pool as ErrDomainAuthFailing instead of a message about sending windows (#160) 2026-08-22 09:37:26 -07:00
Matthew Meszaros 94cf21d95e Fix truncated, unformatted and mis-encoded email content in the unibox (#137)
* feat: add internal/pkg/mailhtml, a mail-oriented HTML sanitizer and text flattener, because rendering a received message body means rendering the sender's markup: Sanitize builds on bluemonday's UGC policy but keeps what real email is made of (table layout attributes, inline CSS through the property-allowlisted style sanitizer, legacy font/center, data: and https: images) while dropping script, iframe, object and the text content of style/head blocks so a marketing email's stylesheet cannot render as body copy, forcing target=_blank plus nofollow/noreferrer on links and allowing only http, https, mailto and tel; ToText flattens the same input for previews, turning block boundaries into newlines and decoding entities back to the characters they stand for so an already-escaped body does not surface as literal &amp;; LooksLikeHTML reports whether a stored body is actually markup, which is how a body recorded as HTML by an older sync but containing no tag at all can be recognised as the plain text it really is

* feat: add internal/pkg/mailhdr for RFC 5322 header values, since headers are ASCII on the wire and every transport was writing raw UTF-8 into Subject and display names: Subject and AddressList RFC 2047-encode non-ASCII (a no-op on plain ASCII, and a bare address stays bare rather than being wrapped in angle brackets), DecodeWords reverses encoded-words with a charset hook wired to go-message so legacy encodings Go does not handle natively still decode, and Bare/BareList strip a display name down to the routable address for SMTP envelope commands where 'Ana <a@b.com>' in RCPT TO is a syntax error, promoting go-message from an indirect to a direct dependency

* feat: encode outbound Subject and address headers on all three transports, so a subject or sender name containing an accent, a currency sign or an emoji reaches the recipient as the characters the user typed instead of mojibake: SMTP and Graph were writing the raw string into Subject (only the Gmail transport encoded it) and Graph built its From by fmt.Sprintf rather than mail.Address, so a non-ASCII display name went out unencoded there too, and all three joined To/Cc/Bcc entries verbatim so an encoded display name never appeared even when the caller supplied one; additionally the SMTP envelope now takes bare addresses through mailhdr.BareList, because an API caller may pass 'Name <addr>' (the compose handler has a bareAddress helper precisely because that arrives) and passing that to RCPT TO gets the recipient rejected by the server

* feat: rewrite the IMAP body reader, which was the reason received mail from SMTP/IMAP mailboxes came back corrupted: it built one FetchItemBodySection with a hardcoded Part []int{1} and a comment saying it would adjust when recursing, which it never did, so on a multipart/alternative the text/plain bytes were fetched twice and the second copy was stored as the HTML body (plain text rendered as markup loses every line break, shows & as an entity and swallows anything inside angle brackets), and decodeIfNeeded never reversed Content-Transfer-Encoding at all, leaving quoted-printable bodies full of =E2=80=99 runs and = soft breaks and base64 bodies unreadable, while its charset detection parsed params off a media-type string that never carried any and its mail.ReadMessage call could silently eat leading body lines as headers; the reader now walks the body structure for real part paths, fetches every text leaf in a single FETCH with a server-side Partial size cap, decodes quoted-printable and base64 (tolerating a tail cut mid-quantum by the cap) then converts the part's charset to UTF-8 with go-message, skips attachment-disposition parts so a .txt attachment cannot stand in for the body, takes one part per type inside a multipart/alternative but treats sibling inline parts in mixed/related as additive, and is bounded at five text parts per message; the stored body cap also goes from 200 KB to 512 KB because 200 KB cuts real HTML newsletters mid-document

* feat: decode Gmail's raw headers and entity-escaped snippets, because the Gmail API hands header values back exactly as they arrived on the wire, so a message from a sender whose subject or display name was RFC 2047-encoded showed in the dashboard as =?utf-8?q?caf=C3=A9?= rather than as the text it stands for, and the API's own snippet field is HTML-escaped, so a preview containing an apostrophe surfaced as &#39; in the conversation list and, until the thread reader stopped rendering snippets as message bodies, inside the message itself; getSingleHeader now runs values through mailhdr.DecodeWords (a no-op unless the value actually contains an encoded-word, so Message-ID and the warmup token header are untouched), the comma-split fallback in getAddressList does the same for display names net/mail could not parse, and the snippet is unescaped once on the way in

* feat: fix the conversation-list snippet, which collapsed whitespace before splitting on newlines so the quoted-line and signature filters below it could never match a thing, stripped HTML with bluemonday's strict policy and then showed the escaped output verbatim so an ampersand in an HTML-only message read as &amp; and a marketing email's stylesheet text rendered as body copy, and cut at 100 bytes with text[:100] so a multi-byte character or emoji at the boundary became a replacement glyph; it now flattens HTML through mailhtml.ToText (entities decoded, style and script content dropped) including when a sender puts markup in their text/plain part, filters quoted history and everything past the RFC 3676 signature delimiter while the text still has lines, collapses whitespace afterwards, and truncates on a rune boundary at 200 characters

* feat: make GET /unibox/:id serve a display-safe body and stop it failing outright, sanitizing body_html through mailhtml before it leaves the API so every consumer gets markup that cannot execute rather than each call site having to defend itself, degrading a body blob that cannot be read to the message's preview text with a new body_truncated flag instead of returning 500 (which made a message with a missing blob unopenable, and hit every seed, sandbox and dev-history fixture row since only the '<seed-' prefix was recognised while the sandbox uses '<sbx-' and dev history '<dev-'), and treating a stored HTML body that contains no tag at all as the plain text it really is, because mail synced before the IMAP reader addressed parts individually recorded the plain part under both bodies and serving that as HTML is exactly what collapsed a ten-line message onto one line

* feat: escape composer text before turning it into the HTML part of an outgoing email, replacing body_html: trimmedBody.replace(/\n/g, '<br />') in both the compose window and the reply composer with a shared plainToHtml that escapes the five markup characters first, so an email containing 'Terms & conditions' no longer ships a broken entity and one containing anything in angle brackets ('<see attached>', 'a < b', a pasted tag) no longer has the rest of the paragraph swallowed by the recipient's mail client as an unclosed tag, while runs of spaces survive as non-breaking spaces and bare URLs become links without eating the sentence punctuation after them; the same unescaped plain-to-HTML pattern in the campaign step editor's applyTemplate now goes through promptToHtml, which escapes as it paragraph-wraps

* feat: render the real message body in the unibox thread reader instead of the list preview, which is the whole of the reported bug: ThreadView mapped each thread row to a UniboxEmail whose body was '<p>' + escapeHtml(m.snippet) + '</p>' and MessageBubble rendered that as the message, but a snippet is a preview capped at 100 characters with every run of whitespace collapsed to one space, so a ten-line email displayed as roughly two lines on a single continuous line, and Gmail's already-escaped snippet was escaped a second time so an apostrophe read as &#39;; each expanded message now loads its own body from GET /unibox/:id (the newest message and anything unread open on mount, older messages collapse to their preview line so a long thread does not fetch every body at once) and renders it in a sandboxed iframe carrying no allow-scripts, which keeps a sender's stylesheet from restyling the dashboard and means nothing in the message can run even though the API already sanitized it, sizing itself from the inner document as images load, with the preview kept as the fallback when a body cannot be fetched and a notice when only a preview is stored

* feat: document how a message body is read and returned, adding a 'Reading a message' section to the unibox guide covering the expand-on-open behaviour, that formatting and special characters are preserved as sent, that the conversation list preview is a summary and not the message, and that HTML mail renders in an isolated frame with links opening in a new tab, plus a paragraph in the API endpoint reference stating that GET /unibox and GET /unibox/thread return previews carrying snippet while GET /unibox/:id returns body_plain and a sanitized body_html, and what body_truncated means

* feat: add email_accounts.save_to_sent, the per-mailbox switch for filing a copy of outbound mail in the Sent folder, defaulting on because plain SMTP submission leaves nothing behind in the sender's account while Gmail and Outlook file their own copy through their APIs, making it a per-mailbox choice rather than a global one since a submission server that files the copy itself (Gmail's SMTP, Fastmail, Zoho) would otherwise end up with two of everything, which is exactly why every desktop mail client ships the same switch, and wiring the column through the Email model, the mailbox read paths and UpdateEmail so it is readable and writable from the dashboard and the API

* feat: teach the IMAP client to APPEND a sent message and the SMTP client to hand back the exact bytes it submitted, the two transport pieces the Sent-folder copy needs: AppendToSent resolves the folder from the RFC 6154 \\Sent special-use attribute first (requesting it only when the server advertises SPECIAL-USE) and falls back to matching the known names against both the full mailbox name and its leaf, since servers namespace as INBOX.Sent and localize the label, caches the result for the life of the connection, files the message flagged \\Seen and dated when it was sent, and returns a sentinel rather than an error when the account has no Sent folder at all; APPEND addresses its mailbox by argument and never touches the selected mailbox, so unlike the warmup MOVE/STORE actions it is safe to run while the sync loop is mid-fetch on the same connection

* feat: file a copy of every SMTP send in the mailbox's Sent folder, closing the gap where a message sent from Warmbly through an SMTP/IMAP mailbox existed only in the recipient's inbox: nothing appeared in the customer's own mail client, and nothing appeared in the unibox either, whose thread reader can only show messages the sync found in a folder, so a user who sent from the dashboard and then went looking for what they sent found no record of it at all; the worker now APPENDs the exact bytes the SMTP client submitted after a successful send, best effort so a failed append never turns a delivered message into a failed task, skipping warmup traffic because filing dozens of machine-generated messages a day would bury the customer's real sent mail, and skipping Gmail and Graph mailboxes entirely since their APIs file their own copy; the per-mailbox setting rides along on the add-email worker payload as a pointer so an older control plane that does not send the field is read as unset and takes the default rather than as an explicit no

* feat: expose the Sent folder copy as a mailbox setting in the dashboard, adding a 'Keep a copy of sent mail' toggle to the Settings tab of the mailbox drawer that only renders for SMTP/IMAP mailboxes (Gmail and Outlook file their own copy, so the control would be a lie there), tracked by the drawer's save bar alongside the other editable fields, and worded so the one case where it should be turned off is obvious: a provider that already saves its own copy, where leaving it on means seeing every sent message twice

* feat: document the Sent folder copy in the mailboxes guide and the API reference, explaining why the toggle exists at all (SMTP submission leaves nothing in the sender's own account, so without it a sent message shows in neither the customer's mail client nor the unibox thread), when to turn it off (a provider such as Gmail, Fastmail or Zoho that already files its own copy of anything submitted over SMTP, where leaving it on doubles every message), that OAuth Gmail and Outlook mailboxes never show the control because their APIs file the copy themselves, that warmup traffic is deliberately excluded, and that PATCH /emails/:id takes save_to_sent

* feat: add unibox_emails.body_text and its search index, because unibox search ran against search_tsv, a generated column built from subject and snippet, and a snippet is a truncated one-line preview, so searching for a phrase that appears in the third paragraph of an email returned nothing at all and read as broken search rather than as search that only covers the first line; message bodies stay in object storage where they belong, and what lands in Postgres is a bounded 16 KB plain-text rendering carried on the new-email worker event, indexed with a GIN expression index rather than a second stored generated column since adding one of those rewrites the whole table while this builds against a column that is empty on every existing row

* feat: index what a message actually says, adding mailhtml.SearchText (HTML flattened, entities decoded, whitespace collapsed, quoted history deliberately kept because a phrase someone quoted back at you should still find the conversation, truncated on a rune boundary) and computing it on all three sync paths so IMAP, Gmail and Graph mail all arrive with searchable text, writing it on insert, and widening the unibox search filter to match either the existing subject-and-preview vector or the body expression, written exactly as the new index declares it so the index is actually used

* feat: backfill the searchable text of messages that were synced before bodies were indexed, so search covers the archive a customer already has instead of only mail that arrives from now on, which would have made the feature useless on day one for exactly the people who need it; the sweep pages through unibox_emails by id, reads each body from object storage under the mailbox owner's key, renders it with the same helper the sync path uses and writes it back, at 100 rows per 30 seconds because nothing waits on it, and returns for good once a pass finds nothing left to visit, with rows whose stored body really is empty simply revisited after the next restart rather than needing a tried-and-failed marker in the schema

* feat: document that unibox search now covers message text and not just subjects and previews, in the search paragraph of the unibox guide where the old wording only promised that search stays inside the current scope

* feat: add generation.RenderThread, the shared way to put a conversation in front of a model, because every AI surface was grounding on preview snippets and a draft written from the first hundred characters of each email answers the greeting rather than the question; it strips quoted history and signatures (the earlier messages are already in the prompt on their own, so quoting them again spends the budget twice, though a reply written underneath the quote is kept rather than thrown away when there is nothing meaningful above the attribution line), spends a bounded character budget newest-message-first since the message being replied to matters most, degrades older messages to their preview line instead of dropping them once the budget runs low, and renders oldest-first so the transcript reads in order

* feat: add grounding reads to the unibox service and repository, returning message text (the stored body, falling back to the preview for mail synced before bodies were indexed) for a thread or for all correspondence with one address, kept deliberately separate from the preview queries and given their own result type so a 16 KB body can never leak into a list response by accident, capped at twenty messages whatever a caller asks for, and paired with a RenderGrounding helper so every AI surface formats a conversation the same way instead of each one rolling its own transcript loop

* feat: ground every AI writing surface in what the messages actually say, switching the unibox reply draft, the compose draft's correspondence history, the inbox agent's thread history and the assistant's read-thread tool from preview snippets to real message text through the new grounding reads, which is what makes a drafted reply answer the question that was asked rather than the first sentence of the email; the inbox agent's triviality gate also reads the reply's full text now, since a preview line cannot tell a one-word ack apart from a long message that happens to open with one, and the assistant tool returns a bounded body per message with quoted history stripped instead of a snippet field

* feat: say in the docs that AI drafting reads the messages and not their previews, in both the unibox reply-draft section (adding that quoted history is stripped and the newest messages get the most room, so a draft answers what was asked rather than the opening sentence) and the inbox agent's grounding section, where 'the full thread so far' was true of the message list but not of how much of each message the model actually saw

* feat: renumber the two new migrations to 000087 and 000088 after rebasing onto main, which landed its own 000085 (org data transfer) and 000086 (email sync state) in the meantime, so the sequence has no duplicate versions

* feat: add the two new API fields to the OpenAPI spec that landed on main while this branch was open, documenting save_to_sent on the Mailbox and MailboxUpdate schemas and body_truncated on UniboxEmail, and saying on body_html that what the API returns is already sanitized so a client can render it directly
2026-08-21 18:45:09 +02:00
Matthew Meszaros 16b672e6f6 feat: wire OnTokenRefresh on the Gmail worker client so every send and sync stops panicking, since goog.Client was constructed with all four message and label callbacks but no token callback while goog.Init unconditionally wrapped the token source in stoken, whose Token() calls that callback on every single request from inside the oauth2 transport's RoundTrip, making the nil func value a guaranteed nil-pointer dereference on the first Gmail API call any mailbox made (the Outlook path immediately below it set the same field correctly, so no Microsoft mailbox was affected), additionally guarding both goog.Init and msgraph.Init so the stoken wrapper is only installed when there is somewhere to persist a refreshed token to, hardening stoken.Token itself against a nil callback because it runs inside RoundTrip where a panic takes down the caller's request rather than surfacing as an error, and adding a regression test that panics without the guard and passes with it (#118) 2026-08-16 07:46:00 +02:00
Matthew Meszaros 0ae4db2c41 feat: make self-hosted auth work without a mail relay by rewriting the platform SMTP transport with real AUTH and TLS (it did neither, so SMTP_USERNAME/SMTP_PASSWORD were dead and every documented relay was unreachable), adding MAIL_TRANSPORT=smtp|log|ses with a log transport that prints codes so a fresh install can sign in with no relay, demoting the emailed login code to AUTH_LOGIN_CODE=always|new_device|off (off on self-host, per NIST SP 800-63B and OWASP ASVS), claiming the first owner through a single-use setup link or WARMBLY_BOOTSTRAP_* instead of register-then-psql, deriving every emailed URL from APP_URL rather than a hardcoded app.warmbly.com that leaked live reset tokens to the vendor, fixing the confirm hooks that read path params against paramless routes and broke login, register and reset confirmation in the dashboard everywhere, adding generic OIDC with PKCE, one-time state, verified nonce and (issuer,subject) identity binding, enforcing 2FA on the social paths that skipped it, adding a per-IP limiter and trusted-proxy handling to the unthrottled auth group, refusing boot on the published default secrets, and dropping mailpit from the default stack (#99) 2026-08-14 14:57:09 +02:00
Matthew Meszaros 443dcbf4b5 Merge pull request #83 from warmbly/ai-content-blocks
AI content blocks and a much more capable dashboard assistant
2026-07-22 17:59:36 +02:00
Matthew Meszaros 887106c3d9 feat: add BuildInlineSnippetRules humanization prompt for AI variables so inline fragments read plainly and fit the surrounding sentence instead of copywriting rhythm 2026-07-22 17:05:08 +02:00
Matthew Meszaros c1c9a31b1f Merge pull request #82 from warmbly/warmup/messaging-automation
Automate warmup content and replies
2026-07-22 12:44:04 +02:00
Matthew Meszaros f768029ec0 feat: automate warmup conversation generation, coherent replies, adaptive rotation, and admin observability 2026-07-22 12:11:19 +02:00
Matthew Meszaros bb040828b1 feat: rework automation AI steps to mirror campaign step types with one agentic AI step plus an AI switch, per-case switch handles on the canvas, a Stop node, and a step-type add menu 2026-07-22 10:58:35 +02:00
Matthew Meszaros fc91d71492 feat: make google, apple and turnstile optional at boot 2026-07-20 09:56:17 +02:00
Matthew Meszaros 4e576daddf feat: stream the dashboard agent's text token by token: the OpenAI-compatible provider now runs chat completions with stream=true whenever a listener is attached, forwarding content deltas as a new EventTextDelta (SSE text_delta) while stitching tool-call fragments by index, capturing usage via stream_options with a sticky compatibility fallback, tolerating backends that ignore streaming, and always closing each block with the authoritative full-text event (now also emitted for commentary alongside tool calls) 2026-07-19 10:41:45 +02:00
Matthew Meszaros 6f8db3ae05 feat: de-template the cold-email humanizer (drop the quotable example asks and subject that models parroted verbatim, add a VARIATION section mandating a fresh skeleton per email and capping punchy standalone lines at one) and add compose drafts: autosaved per-user working copies (client-generated ids, idempotent PUT on a 1.2s debounce, migration 000072) with Saving/Saved in the window header, close-keeps-draft instead of the discard confirm, delete on send or when emptied, and a Drafts list under the rail Compose button to resume or delete, documented in the guide and endpoints scope map 2026-07-19 08:52:56 +02:00
Matthew Meszaros e6b11f8d07 feat: fix compose draft quality and review chrome: a compose-specific humanizer frame (BuildComposeRules) that bans copywriting rhythm outright (no standalone punch lines, no problem-agitate-pitch, no market generalizations, 40-70 words, plain first sentence) and gives no example phrasings for the model to parrot, clear the grounding report and voice-profile nudge when the draft flow ends instead of lingering after Keep, restructure the Draft ready card into breathing rows (title plus usage, grounding line, right-aligned action row), and portal the From mailbox menu to the body with viewport-aware flip so the compose window's overflow clipping can't cut it off 2026-07-19 08:45:47 +02:00
Matthew Meszaros cd5e1e9cf7 Merge pull request #72 from warmbly/research/warmup-abuse-cold-effectiveness
feat: deliverability research + observe-only SPF/DKIM/DMARC auth state and watch-band cold throttle
2026-07-17 05:29:58 +02:00
Matthew Meszaros 7176bbce3c feat: add dnsauth unit tests covering Result.State() classification (unknown on empty/transient-error, passing on SPF+DMARC, DKIM advisory) and DMARC policy parsing 2026-07-17 03:22:48 +00:00
Matthew Meszaros 73c80530c3 feat: adapt the OpenAI-compatible provider to the gpt-5.x/o-series request shape at runtime: a 400 naming max_tokens or temperature flips a sticky per-provider flag (max_completion_tokens / omit temperature) and retries, so newer OpenAI models work while Ollama/Groq/OpenRouter keep the legacy params 2026-07-16 07:25:08 +02:00
Matthew Meszaros e64b0b161d feat: sweep stale OPENAI_API_KEY/AI_LOCAL_MODEL mentions out of comments, the provider-not-configured and warmup-admin error strings, env.example, and the deployment guide now that AI_* is the only config surface 2026-07-16 06:39:07 +02:00
Matthew Meszaros 29e3bb3a64 feat: drop the legacy OPENAI_*/ANTHROPIC_API_KEY/AI_LOCAL_MODEL env fallbacks (no prod deployments exist): AI_PROVIDER/AI_API_KEY/AI_MODEL/AI_BASE_URL/AI_FREE are the only AI vars, empty provider means openai, the anthropic preset and writing client key off AI_API_KEY, and the warmup Batch generator runs only when the selected provider is OpenAI 2026-07-16 06:39:06 +02:00
Matthew Meszaros 9d76685fb2 feat: add generation.Resolve mapping an AI_PROVIDER preset (openai/openrouter/groq/ollama/anthropic/custom) plus AI_API_KEY/AI_MODEL/AI_BASE_URL/AI_FREE into a provider config, erroring on a base-less custom provider instead of leaking the key to OpenAI 2026-07-16 06:14:19 +02:00
Matthew Meszaros 42f2a9bf77 feat: add generation.BuildAgentVoiceRules - the shared humanizer bans plus the org voice grounding, scoped for the dashboard agent so it does not hijack the agent role 2026-07-15 18:51:51 +02:00
Matthew Meszaros 80452ce4b4 feat: add IsLocal() to the WritingGenerator interface and its hosted writing clients so the writing surfaces can detect a free/local model 2026-07-15 18:51:51 +02:00
Matthew Meszaros 84424fec8f feat: add a free/local model flag to the generation provider - ProviderConfig.Local, Provider.IsLocal(), and a local default model (llama3.1) for OpenAI-compatible endpoints 2026-07-15 18:51:51 +02:00
Matthew Meszaros a067bf5922 feat: AI action nodes in automations - ai_classify/ai_extract/ai_generate run one generation.Provider completion per node over the event data and merge the result back as a variable (ai_class, output_keys[], ai_text) so downstream conditions branch on it, charging 1 credit per node with a run_id+node_id idempotency key, refund on provider failure, out-of-credits fails only that node while the run continues, and a per-automation ai_credit_failures counter (migration 000064) auto-pauses a flow after 20 consecutive credit misses; classify/extract run deterministic (temperature 0), dry-run executes AI nodes for real and is charged but never touches the auto-pause lifecycle, provider + credit ledger wired into both backend and consumer integration services, plus the reply-classifier Layer 3 ported off its bespoke OpenAI client onto the platform provider via an injected ModelClassifyFunc (platform-paid, deterministic), a CompletionRequest.Temperature primitive threaded through the OpenAI and Anthropic providers, the flow builder palette/config panels/validation for the three AI steps, and a new AI-steps-in-automations guide 2026-07-14 05:30:55 +02:00
Matthew Meszaros 2aec6745fa feat: connect external MCP servers whose tools the AI assistant can use - ai_mcp_servers table with bearer tokens sealed by the org DEK cipher (never returned) and SSRF-validated https urls, a dependency-free streamable-HTTP JSON-RPC client (initialize/tools/list/tools/call over safehttp dial-time IP blocking with SSE+JSON handling and body caps), an mcp service that discovers tools on connect and contributes only enabled servers' tools to the dashboard agent as namespaced mcp_<server>_<tool> defs that are always write-class and never auto-allowed, a registry DynamicToolSource hook so per-org tools join the agent's tool set and resume executes them through ToolDefs, /ai/connections CRUD gated on manage_settings with an mcp_server audit entity and spine, a Connections settings page to add servers and review/enable discovered tools, and docs 2026-07-13 20:05:24 +02:00
Matthew Meszaros 8d208d8c41 feat: org voice profile grounding every AI writing surface plus context-grounded unibox reply drafts - organizations gains product_description/icp_notes/voice_profile (000059) with a manage_settings workspace settings section, generation.BuildVoiceRules and BuildReplyRules fold the humanizer rules and org grounding into every prompt, WritingGenerator.GenerateWriting now takes a VoiceContext and a provider.Complete primitive powers single completions, POST /unibox/reply/draft assembles thread history plus the counterpart contact (custom fields and campaigns) plus the voice profile into a reply-tuned prompt that charges 2 credits with refund-on-failure and a client idempotency key and never sends, a Draft reply button fills the composer, and docs cover the voice profile and AI reply drafts 2026-07-13 19:04:57 +02:00
Matthew Meszaros 7f425c1624 feat: dashboard-wide AI assistant with streamed tool-use runs, per-action approvals, and per-iteration credits - agent_sessions/agent_messages/ai_tool_policies migration, aiagent service runs the M2 registry through the provider loop over a resumable jsonb transcript, streams text/tool-step/approval/done events over SSE, gates write tools behind approve/deny/always-allow (send always per-action) and charges 1 credit per iteration (budget 20, refund on provider failure, out-of-credits 402 insufficient_credits, cap 429 usage_cap_exceeded) with a resume-safe idempotency namespace, provider PreIteration budget hook, /ai/sessions endpoints (cursor list + two SSE runs) gated by membership with tools bound to the member's org-permission bits, APIPermAIAgent bit 22, ai_session audit entity + spine, and a right-side resizable panel (Cmd+I / sparkle button) with streamed text, collapsible tool steps, inline approval and draft-artifact deep-link cards, stop, new chat, and a credits/budget meter; tenancy enforced at the SQL layer and docs for the assistant, endpoints, permissions, and error codes 2026-07-13 18:46:43 +02:00
Matthew Meszaros e07c33d3cd feat: provider-agnostic LLM agent-loop layer (OpenAI-first, Anthropic self-host connector) with RunAgent tool-use loop, approval/resume contract, pluggable web search, and extracted voice-rules builder - generation.Provider.RunAgent executes Go tool handlers over a resumable transcript with a read-auto/write-approve gate, openAIProvider drives it via a lean HTTP client retargetable with OPENAI_BASE_URL for any OpenAI-compatible endpoint and also serves the writing assistant, anthropicProvider mirrors it with tool_use/tool_result multi-turn plus hosted web_search, NewProvider prefers OPENAI_API_KEY, SearchClient supports Serper/SearXNG with a clean not-configured error, and BuildVoiceRules extracts the humanizer prompt for reuse; refreshed stale model ids to sonnet-4-6 and haiku-4-5 2026-07-13 17:52:09 +02:00
Matthew Meszaros 99c1c4afa9 feat: seal SMTP/IMAP credentials at rest with a CREDENTIALS_ENCRYPTION_KEY-backed encrypter - the email repository Encrypt field was never wired anywhere, so smtp_imap accounts could never load onto workers (plaintext rows failed decode; hex-looking rows would nil-panic); adds encrypt.FromEnv and nil-guards that fail with a captured error instead 2026-07-11 17:30:49 +02:00
Matthew Meszaros a20ba926cb feat: add native Apple/Google sign-in to the backend - POST /auth/apple and /auth/google exchange provider-signed ID tokens for sessions, GET /auth/providers exposes configured providers for app discovery, with a JWKS-verifying idtoken package, auth config, stable error codes, and API/security docs 2026-07-07 05:56:35 +02:00
Matthew Meszaros 7abde17c26 feat: add a conservative DSN bounce parser that extracts permanence, failed recipient, and original message id from delivery-status reports, treating only 5.x.x/failed as permanent so transient 4.x.x failures never over-suppress 2026-07-04 11:36:47 +02:00