mirror of
https://github.com/warmbly/warmbly.git
synced 2026-10-03 16:02:02 +00:00
feat: prevent sender views from tracking opens and keep sent messages out of the default Inbox (#542)
This commit is contained in:
Matthew Meszaros
@@ -229,7 +229,7 @@ The `/unibox/drafts` endpoints hold autosaved compose drafts, scoped to the call
|
||||
|
||||
`PATCH /emails/:id` accepts `save_to_sent` (boolean) on SMTP/IMAP mailboxes: when true, which is the default, the worker files a copy of each outbound message in the mailbox's Sent folder. It has no effect on Gmail and Outlook mailboxes, whose APIs file their own copy. See [keeping a copy of sent mail](/guides/mailboxes/#keeping-a-copy-of-sent-mail).
|
||||
|
||||
`GET /unibox` and `GET /unibox/thread` return message previews: each row carries `snippet`, a one-line summary, not the message body. Read a full message with `GET /unibox/:id`, which returns `body_plain` plus `body_html`. The HTML is sanitized before it leaves the API (scripts, event handlers, embedded frames, and unsafe URL schemes are removed), so it is safe to render, and links carry `target="_blank"` with `rel="noopener"`. `body_truncated` is `true` on the rare message whose stored body could not be read, where `body_plain` falls back to the snippet.
|
||||
`GET /unibox` and `GET /unibox/thread` return message previews: each row carries `snippet`, a one-line summary, not the message body. Read a full message with `GET /unibox/:id`, which returns `body_plain` plus `body_html`. The HTML is sanitized before it leaves the API (scripts, event handlers, embedded frames, and unsafe URL schemes are removed), so it is safe to render, and links carry `target="_blank"` with `rel="noopener"`. Warmbly open-tracking pixels are also removed from this display copy, including quoted history, so rendering it does not record a campaign open. Stored and delivered copies keep their pixels. `body_truncated` is `true` on the rare message whose stored body could not be read, where `body_plain` falls back to the snippet.
|
||||
|
||||
`GET /unibox` also accepts `address=<value>` (matches sender or recipient, for "every conversation with this person") and `direction=sent|received` (resolved against your own mailbox addresses).
|
||||
|
||||
|
||||
@@ -18,9 +18,10 @@ Campaign analytics are shared across the workspace. With **View analytics**, you
|
||||
| Bounces | Hard and soft bounces | Bounce rate |
|
||||
| Complaints | Marked as spam | Complaint rate |
|
||||
|
||||
Six rules govern the counts:
|
||||
Seven rules govern the counts:
|
||||
|
||||
- **Tracking must be on.** Opens and clicks need open or link tracking enabled on the campaign, and a tracking host on the install. A [custom tracking domain](/guides/mailboxes/#custom-tracking-domain) per mailbox is optional; without one they go through the shared host and still count. With tracking off they stay at zero while sends and replies still count.
|
||||
- **Reading in Warmbly does not count as an open.** Unibox removes Warmbly's tracking pixels from the displayed message, including quoted history. This applies to sent messages and every other folder; recipient copies retain their pixels.
|
||||
- **Replies are human replies.** Out-of-office and autoresponders never count, never stamp the contact as replied, and never trip stop-on-reply.
|
||||
- **Bots are filtered.** Crawlers, CLI agents, prefetches, chat link previews, and security gateways that announce themselves are served normally but never counted. The ones that do not announce themselves are caught by where they came from and by what they do. Requests from a known mail-filtering network are treated as automated whatever their user agent claims, and a click from one is still redirected, just without the ticket that would file the visit against the recipient. Microsoft 365's own filtering layer, where Safe Links and delivery-time link scanning run, is recognised out of the box, as is Barracuda's. Proofpoint, Mimecast and Cisco Secure Email are recognised too, with one difference: those three also offer browser isolation, which renders a clicked page in the vendor's cloud for the recipient to read, so a click from them can be a real person. A match there is treated as automated only when it also arrives soon enough after the send to be the delivery-time scan, and a recipient who clicks through isolation later still counts as a person. The rest are caught by behaviour: an open within a minute of the send, or a click within thirty seconds, and clicks on two or more links of one email from the same source within five seconds (a scanner walking the message). The clock on the first two starts when the send is handed to a worker, before the mail has even been delivered, so those windows cover the sending provider's queue and the transit to the recipient as well as reading time. A self-hosted instance can change those windows under Instance settings, including the one the isolation vendors are measured against, when its recipients' gateways are slower or faster than that. Otherwise one corporate scanner would "click" every link seconds after delivery. A self-hosted instance can name its own scanner networks; see [tracking service configuration](/development/configuration/).
|
||||
- **Auto-opens are labeled, not hidden.** Privacy proxies like Apple Mail Privacy Protection, and instant opens, still count (they confirm delivery) but are tagged and shown separately (`12 auto`). A later real open upgrades them to human. **Auto-opens never trigger opened-based branches or automations.**
|
||||
|
||||
@@ -49,6 +49,8 @@ The rail is two short groups and then your mailboxes. The first group is where y
|
||||
|
||||
**Scheduled** sits between Sent and Archive. It is a view rather than a folder, listing replies queued to send later, and it reads naturally next to Sent.
|
||||
|
||||
Opening **Inbox** from the main navigation starts in the Inbox folder. Sent messages live in **Sent**; choose **All mail** when you want inbound and outbound messages together. An Inbox conversation still shows its complete history, including your replies, when you open it.
|
||||
|
||||
Each message starts in the folder the provider has it in: IMAP special-use folder attributes, Gmail labels, and Outlook well-known folders all map to the same six. Moves at the provider (junking a message, clearing it out of spam) follow on the next sync. Drafts are also reconciled: a draft the mailbox no longer holds is removed here too, so the fresh copy Gmail saves on every autosave does not leave the previous ones behind. You can also file a conversation yourself, which moves it here without moving it at the provider; see [filing a conversation](#filing-a-conversation). The active row is highlighted grey, unread counts sit on the right in blue, and hovering a folder reveals a three-dot menu with **Mark all as read**.
|
||||
|
||||
Spam and Trash stay out of every other view: the **All mail** view and the unread badge only count the folders you actually work.
|
||||
@@ -94,6 +96,8 @@ What you see is the message as it was sent: paragraphs, line breaks, lists, tabl
|
||||
|
||||
HTML mail renders in an isolated frame, so a sender's styling cannot leak into the dashboard and nothing in the message can run code. Links open in a new tab and carry no referrer. If the stored copy of a message cannot be read, Warmbly shows the preview it has and says so instead of failing to open the conversation.
|
||||
|
||||
Warmbly removes its open-tracking pixels from the displayed copy, including quoted messages. Reading mail here, whether in Inbox, Sent or another view, does not record a campaign open. The delivered email keeps its tracking pixel for recipient opens, and ordinary images still display.
|
||||
|
||||
## Categories and labels
|
||||
|
||||
Categories are the workspace's conversation labels, shared with the rest of Warmbly, so `Interested` means the same thing on a contact as it does here. They are shared with the rest of the team too: a category anyone creates is available to everyone, and a conversation one member labels shows that label to the next. Label with the tag button in the conversation header or `c`: search existing categories, tick what applies, or type a name and **Create**. A conversation can carry several.
|
||||
|
||||
@@ -93,7 +93,7 @@ func Sanitize(raw string) string {
|
||||
if strings.TrimSpace(raw) == "" {
|
||||
return ""
|
||||
}
|
||||
return displayPolicy.Sanitize(raw)
|
||||
return stripOpenTrackingPixels(displayPolicy.Sanitize(raw))
|
||||
}
|
||||
|
||||
// tagPattern matches an HTML tag opener. A plain-text body that merely
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
package mailhtml
|
||||
|
||||
import (
|
||||
"net/url"
|
||||
"strings"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"golang.org/x/net/html"
|
||||
)
|
||||
|
||||
// Remove Warmbly beacons from display copies, including pixels in quoted replies.
|
||||
func stripOpenTrackingPixels(body string) string {
|
||||
z := html.NewTokenizer(strings.NewReader(body))
|
||||
var out strings.Builder
|
||||
for {
|
||||
kind := z.Next()
|
||||
if kind == html.ErrorToken {
|
||||
return out.String()
|
||||
}
|
||||
raw := string(z.Raw())
|
||||
if kind == html.StartTagToken || kind == html.SelfClosingTagToken {
|
||||
token := z.Token()
|
||||
if token.Data == "img" && hasOpenTrackingSource(token.Attr) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
out.WriteString(raw)
|
||||
}
|
||||
}
|
||||
|
||||
func hasOpenTrackingSource(attrs []html.Attribute) bool {
|
||||
for _, attr := range attrs {
|
||||
if attr.Key != "src" {
|
||||
continue
|
||||
}
|
||||
u, err := url.Parse(strings.TrimSpace(attr.Val))
|
||||
if err != nil || !strings.HasPrefix(u.Path, "/t/o/") {
|
||||
continue
|
||||
}
|
||||
// Match the endpoint on shared, custom and previous tracking domains.
|
||||
id := strings.TrimSuffix(strings.TrimPrefix(u.Path, "/t/o/"), ".png")
|
||||
if _, err := uuid.Parse(id); err == nil {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
package mailhtml
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSanitizeRemovesWarmblyOpenPixels(t *testing.T) {
|
||||
const id = "550e8400-e29b-41d4-a716-446655440000"
|
||||
for _, markup := range []string{
|
||||
`<img src="https://tracking.customer.test/t/o/` + id + `.png" width="1" height="1" style="display:none;" alt="" />`,
|
||||
`<blockquote><p>Quoted reply</p><img src="https://old-domain.test/t/o/` + id + `.png"></blockquote>`,
|
||||
`<IMG SRC='http://localhost:3000/t/o/` + id + `'>`,
|
||||
`<img src="//custom.test/t/o/` + id + `.png?x=1&y=2">`,
|
||||
`<img src="https://custom.test/t/o/` + id + `.png">`,
|
||||
`<img src="https://custom.test/t/o/%35` + id[1:] + `.png">`,
|
||||
} {
|
||||
t.Run(markup, func(t *testing.T) {
|
||||
out := Sanitize(`<p>Hello</p>` + markup)
|
||||
if strings.Contains(out, "<img") {
|
||||
t.Fatalf("displaying the message would request its open pixel: %s", out)
|
||||
}
|
||||
if !strings.Contains(out, "<p>Hello</p>") {
|
||||
t.Fatalf("message content lost: %s", out)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestSanitizeKeepsOrdinaryImagesAndLinks(t *testing.T) {
|
||||
const image = "https://cdn.customer.test/logo.png"
|
||||
const link = "https://custom.test/c/550e8400-e29b-41d4-a716-446655440000"
|
||||
out := Sanitize(`<p>café & more</p><img src="` + image + `" width="1" height="1">` +
|
||||
`<img src="data:image/png;base64,iVBORw0KGgo=">` +
|
||||
`<img src="https://cdn.customer.test/t/o/logo.png"><a href="` + link + `">Visit</a>`)
|
||||
for _, want := range []string{image, link, "data:image/png;base64,iVBORw0KGgo=", "/t/o/logo.png", "café & more"} {
|
||||
if !strings.Contains(out, want) {
|
||||
t.Errorf("display content %q lost: %s", want, out)
|
||||
}
|
||||
}
|
||||
if again := Sanitize(out); again != out {
|
||||
t.Errorf("sanitizing twice changed the display: %s", again)
|
||||
}
|
||||
}
|
||||
@@ -6,8 +6,28 @@ import (
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/warmbly/warmbly/internal/models"
|
||||
"github.com/warmbly/warmbly/internal/pkg/mailhtml"
|
||||
)
|
||||
|
||||
func TestOpenTrackingIsRemovedOnlyFromDisplayCopy(t *testing.T) {
|
||||
const original = `<html><body><p>Hello</p><img src="https://example.com/logo.png"></body></html>`
|
||||
for _, host := range []string{"t.warmbly.com", "custom.example.com", "localhost:3000"} {
|
||||
t.Run(host, func(t *testing.T) {
|
||||
delivered := AddOpenTrackingPixel(original, uuid.New(), host)
|
||||
displayed := mailhtml.Sanitize(delivered)
|
||||
if strings.Contains(displayed, "/t/o/") {
|
||||
t.Fatalf("reading the sent copy would track an open: %s", displayed)
|
||||
}
|
||||
if !strings.Contains(delivered, "/t/o/") {
|
||||
t.Fatalf("recipient copy lost tracking: %s", delivered)
|
||||
}
|
||||
if !strings.Contains(displayed, "https://example.com/logo.png") {
|
||||
t.Fatalf("ordinary image lost: %s", displayed)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestAddOpenTrackingPixelUsesTheConfiguredHost(t *testing.T) {
|
||||
html := "<html><body>hi</body></html>"
|
||||
out := AddOpenTrackingPixel(html, uuid.New(), "t.acme.com")
|
||||
|
||||
@@ -78,7 +78,7 @@ export default function UniboxPage() {
|
||||
// opaque mailbox/tag/label id for those scopes) is the only query param left.
|
||||
// Accounts are no longer in the URL: the thread fetch scans every mailbox the
|
||||
// user owns, which is the right default for a unified inbox.
|
||||
const urlScope = routeParams.scope ?? "all";
|
||||
const urlScope = routeParams.scope ?? "inbox";
|
||||
const urlThread = routeParams.threadId ?? null;
|
||||
const urlScopeRef = searchParams.get("ref");
|
||||
|
||||
|
||||
@@ -9,7 +9,7 @@
|
||||
|
||||
import React from "react";
|
||||
import { describe, it, expect, vi, beforeAll, beforeEach } from "vitest";
|
||||
import { screen, act, fireEvent } from "@testing-library/react";
|
||||
import { screen, act, fireEvent, waitFor } from "@testing-library/react";
|
||||
import { useAppStore } from "@/stores";
|
||||
import { visibleShortcuts } from "@/hooks/useKeyboardShortcuts";
|
||||
import {
|
||||
@@ -22,6 +22,8 @@ import {
|
||||
SUITE,
|
||||
} from "./uniboxHarness";
|
||||
|
||||
const searchRequests = vi.hoisted((): string[] => []);
|
||||
|
||||
beforeAll(() => {
|
||||
installLayoutShims();
|
||||
setViewportWidth(1512);
|
||||
@@ -29,6 +31,9 @@ beforeAll(() => {
|
||||
|
||||
vi.mock("@/lib/api/client/Request", () => ({
|
||||
default: async (cfg: { url?: string }) => {
|
||||
if (cfg.url === "/unibox" || cfg.url?.startsWith("/unibox?")) {
|
||||
searchRequests.push(cfg.url);
|
||||
}
|
||||
const { route } = await import("./uniboxHarness");
|
||||
return route(String(cfg?.url ?? ""));
|
||||
},
|
||||
@@ -71,11 +76,28 @@ const selected = () => useAppStore.getState().selectedThreadId;
|
||||
|
||||
describe("unibox list shortcuts (#484)", SUITE, () => {
|
||||
beforeEach(() => {
|
||||
searchRequests.length = 0;
|
||||
resetScrollTops();
|
||||
useAppStore.setState({ selectedThreadId: null, navCollapsed: false });
|
||||
useAppStore.getState().clearSequence();
|
||||
});
|
||||
|
||||
it("opens Inbox from the main navigation and keeps Sent and All mail explicit", async () => {
|
||||
const router = await mount("/app/unibox");
|
||||
await waitFor(() => expect(searchRequests.length).toBeGreaterThan(0));
|
||||
expect(searchRequests.every((url) => new URL(url, "https://test.local").searchParams.get("folder") === "inbox")).toBe(true);
|
||||
|
||||
searchRequests.length = 0;
|
||||
await act(async () => { await router.navigate("/app/unibox/sent"); });
|
||||
await waitFor(() => expect(searchRequests.length).toBeGreaterThan(0));
|
||||
expect(searchRequests.every((url) => new URL(url, "https://test.local").searchParams.get("folder") === "sent")).toBe(true);
|
||||
|
||||
searchRequests.length = 0;
|
||||
await act(async () => { await router.navigate("/app/unibox/all"); });
|
||||
await waitFor(() => expect(searchRequests.length).toBeGreaterThan(0));
|
||||
expect(searchRequests.every((url) => !new URL(url, "https://test.local").searchParams.has("folder"))).toBe(true);
|
||||
});
|
||||
|
||||
it("moves, jumps to the ends, opens and deselects", async () => {
|
||||
await mount("/app/unibox/all");
|
||||
await settle();
|
||||
|
||||
@@ -108,12 +108,12 @@ export function ConversationList({
|
||||
return next;
|
||||
}, [params, debouncedSearch]);
|
||||
|
||||
const q = useUniboxSearch(merged);
|
||||
const q = useUniboxSearch(merged, scopeKey);
|
||||
const emails = q.emails;
|
||||
const totalShown = emails.length;
|
||||
const activeFilters = countUserFilters(params, baseParams);
|
||||
|
||||
// A scope, search or filter change keeps the previous rows on screen while
|
||||
// A search or filter change keeps the previous rows on screen while
|
||||
// the new ones load (placeholderData). That is the moment to show progress:
|
||||
// a bar along the top and the stale rows dimmed. Background refetches from
|
||||
// realtime events do not qualify, so nothing flickers while reading. The
|
||||
|
||||
@@ -62,6 +62,7 @@ export default function ComposeHistoryPanel({
|
||||
// History is reference material: include snoozed threads too.
|
||||
snoozed: "any",
|
||||
},
|
||||
`history:${address}:${tab}`,
|
||||
!!address,
|
||||
);
|
||||
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
import type { ReactNode } from "react";
|
||||
import { QueryClient, QueryClientProvider } from "@tanstack/react-query";
|
||||
import { renderHook } from "@testing-library/react";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import type { UniboxSearchParams } from "@/lib/api/models/app/unibox/UniboxSearch";
|
||||
import useUniboxSearch from "./useUniboxSearch";
|
||||
|
||||
vi.mock("@/lib/api/client/app/unibox/searchIncoming", () => ({
|
||||
default: () => new Promise(() => {}),
|
||||
}));
|
||||
|
||||
describe("unibox scope transitions", () => {
|
||||
it.each<{
|
||||
name: string; initial: UniboxSearchParams; next: UniboxSearchParams;
|
||||
scope: string; nextScope: string; keep?: boolean;
|
||||
}>([
|
||||
{ name: "Sent to Inbox", scope: "folder:sent", nextScope: "folder:inbox", initial: { folder: "sent" }, next: { folder: "inbox" } },
|
||||
{ name: "All mail to Inbox", scope: "all", nextScope: "folder:inbox", initial: {}, next: { folder: "inbox" } },
|
||||
{ name: "All mail to Unread", scope: "all", nextScope: "unread", initial: {}, next: { unseen: true } },
|
||||
{ name: "mailbox change", scope: "mailbox:a", nextScope: "mailbox:b", initial: { accountIds: ["a"] }, next: { accountIds: ["b"] } },
|
||||
{ name: "history recipient change", scope: "history:a:all", nextScope: "history:b:all", initial: { address: "a" }, next: { address: "b" } },
|
||||
{ name: "search within Inbox", scope: "folder:inbox", nextScope: "folder:inbox", initial: { folder: "inbox" }, next: { folder: "inbox", query: "hello" }, keep: true },
|
||||
{ name: "filter within All mail", scope: "all", nextScope: "all", initial: {}, next: { unseen: true }, keep: true },
|
||||
])("keeps previous rows only within the same scope: $name", ({ initial, next, scope, nextScope, keep = false }) => {
|
||||
const client = new QueryClient({ defaultOptions: { queries: { retry: false } } });
|
||||
const row = {
|
||||
id: "sent-message", email_id: "mailbox", thread_id: "outbound-thread",
|
||||
from_addr: ["me@example.com"], to_addr: ["client@example.com"],
|
||||
subject: "Sent message", snippet: "Hello", internal_date: "2026-09-16T00:00:00Z",
|
||||
seen: true, message_count: 1, has_unread: false, labels: [],
|
||||
};
|
||||
client.setQueryData(["unibox", "search", initial, scope], {
|
||||
pages: [{ data: [row], pagination: { has_more: false, next_cursor: null } }],
|
||||
pageParams: [null],
|
||||
});
|
||||
const wrapper = ({ children }: { children: ReactNode }) => (
|
||||
<QueryClientProvider client={client}>{children}</QueryClientProvider>
|
||||
);
|
||||
const { result, rerender, unmount } = renderHook(
|
||||
({ params, scope }: { params: UniboxSearchParams; scope: string }) => useUniboxSearch(params, scope),
|
||||
{ initialProps: { params: initial, scope }, wrapper },
|
||||
);
|
||||
expect(result.current.emails).toEqual([row]);
|
||||
rerender({ params: next, scope: nextScope });
|
||||
expect(result.current.emails).toEqual(keep ? [row] : []);
|
||||
expect(result.current.isPending).toBe(!keep);
|
||||
unmount();
|
||||
client.clear();
|
||||
});
|
||||
});
|
||||
@@ -1,20 +1,20 @@
|
||||
// Infinite scroll over the inbox search endpoint.
|
||||
|
||||
import { keepPreviousData, useInfiniteQuery, type InfiniteData } from "@tanstack/react-query";
|
||||
import { useInfiniteQuery, type InfiniteData } from "@tanstack/react-query";
|
||||
import searchIncoming from "@/lib/api/client/app/unibox/searchIncoming";
|
||||
import type { UniboxSearchParams } from "@/lib/api/models/app/unibox/UniboxSearch";
|
||||
|
||||
type Page = Awaited<ReturnType<typeof searchIncoming>>;
|
||||
|
||||
export default function useUniboxSearch(params: UniboxSearchParams, enabled = true) {
|
||||
export default function useUniboxSearch(params: UniboxSearchParams, scopeKey: string, enabled = true) {
|
||||
const q = useInfiniteQuery<
|
||||
Page,
|
||||
Error,
|
||||
InfiniteData<Page, string | null>,
|
||||
["unibox", "search", UniboxSearchParams],
|
||||
["unibox", "search", UniboxSearchParams, string],
|
||||
string | null
|
||||
>({
|
||||
queryKey: ["unibox", "search", params],
|
||||
queryKey: ["unibox", "search", params, scopeKey],
|
||||
queryFn: ({ pageParam }) =>
|
||||
searchIncoming({ ...params, cursor: pageParam ?? undefined, limit: 50 }),
|
||||
initialPageParam: null,
|
||||
@@ -24,10 +24,9 @@ export default function useUniboxSearch(params: UniboxSearchParams, enabled = tr
|
||||
// page the user had loaded, which is what the remembered scroll offset
|
||||
// needs to land on (issue #396).
|
||||
gcTime: 30 * 60 * 1000,
|
||||
// Scope/filter switches change the query key; keep showing the
|
||||
// previous list while the new one loads instead of flashing the
|
||||
// whole pane to skeletons on every switch.
|
||||
placeholderData: keepPreviousData,
|
||||
// Keep filter results during loading only within the same view.
|
||||
placeholderData: (previousData, previousQuery) =>
|
||||
previousQuery?.queryKey[3] === scopeKey ? previousData : undefined,
|
||||
enabled,
|
||||
});
|
||||
|
||||
|
||||
+1
-1
@@ -400,7 +400,7 @@ const router = createBrowserRouter([
|
||||
},
|
||||
{
|
||||
// Path-based, readable inbox URLs: /app/unibox/<scope>[/<threadId>].
|
||||
// Both segments optional, so /app/unibox is the default "all" view.
|
||||
// Both segments optional, so /app/unibox opens the Inbox folder.
|
||||
// Both are state inside one page, not different pages, so the shell
|
||||
// keeps the page mounted across them and the conversation list holds
|
||||
// its scroll offset when a thread opens (issue #396).
|
||||
|
||||
Reference in New Issue
Block a user