Commit Graph
34 Commits
Author SHA1 Message Date
Matthew Meszaros 15e139e15d feat: stop a campaign email going out twice when the progress write after dispatch is lost: a step is now RESERVED before its SEND_EMAIL reaches the bus (migration 000093 adds campaign_contact_progress.dispatched_at + dispatch_task_id, and ReserveSend takes the claim and the day's counters in one transaction) and routing treats a step as attempted on sent_at OR dispatched_at, so a crash or a failed stamp in the dispatch window can no longer read as "never sent" and email the same person again; the ON CONFLICT claim is exactly-once so two ticks racing the same pair cannot both send (the loser ends skipped_duplicate), the stamp is retried and escalated to the campaign feed instead of warned and swallowed, HandleEmailSent repairs a lost stamp from the worker's own confirmation, ReleaseSend gives a reservation back only when the command provably never left (a publish failure is ambiguous via ErrSendDispatchUnknown and keeps it), and StartStuckSendReclaimer walks back a reservation nobody answered after 30 minutes so a worker that died mid-send cannot park a lead in flight forever; live-tested in TestLiveLostProgressWriteDoesNotResend, TestLiveDispatchedSendIsNeverOfferedTwice, TestLiveConcurrentTicksSendOnce, TestLiveStuckDispatchIsReclaimed, TestLiveReclaimBelievesADeliveredSend and TestLiveInFlightSendIsNotOfferedAgain 2026-08-24 09:15:06 -07:00
Matthew Meszaros c3066f9cc9 feat: unbox campaign start dates and make follow-up pacing real: accept today as "start now" and let an explicit null clear start/end dates on PATCH /campaigns (models.NullableTime distinguishes absent from null, which used to silently no-op while the error message told users to send null), reschedule an active campaign's parked wakeup when any schedule field changes so clearing a future start date takes effect immediately instead of at the old slot, let a completed campaign be started again and turn the past-end-date start 500 into a clear 400, gate the campaign task on the step's hard-constraint floor (wait_after, start date, windows, day capacity, mailbox min-gap) via ErrCampaignDeferred so an early successor tick can no longer send a wait-3-days follow-up seconds after step one (live-tested in TestLiveFollowUpWaitIsHonored), disable past days in the schedule date picker, and fix the sandbox seed leaving worker 1a01 free-tier after make seed which unassigned the paid org's mailboxes and failed every send 2026-08-24 05:20:26 -07:00
Matthew Meszaros 7c2d08f204 feat: renumber the campaign_send_outcome migration to 000091 so it stops colliding with 000089_email_auth_enforcement, which made golang-migrate refuse to init its source driver and left the backend restart-looping at boot on main, and guard the class of bug with scripts/check-migrations.sh (duplicate versions, gaps, unpaired up/down, unparseable filenames) wired into make check-migrations, make lint, a dedicated Migrations CI job and a TestEmbeddedMigrationsLoad that boots the real iofs source over the embedded FS 2026-08-23 21:26:14 -07:00
Matthew Meszaros 8e0043878c feat: consume the worker's EMAIL_SENT and EMAIL_FAILED results in the consumer, which until now dropped them with a warning so a send the worker could not complete stayed stamped as sent and the lead sat at 'processing' forever: a failure now marks the task failed, walks the step back, gives the day's counters back, writes a red entry to the campaign activity feed, reopens a campaign that completed while the send was in flight, marks the lead failed after five attempts, and routes a recipient refused at RCPT into the bounce pipeline instead of retrying it; a one-off compose or reply failure is pushed to the mailbox owner; covered by an opt-in live test against Postgres and documented in the events page, the campaign guide, the contacts API reference and the agent notes 2026-08-23 10:18:40 -07:00
Matthew Meszaros a75ea012a0 feat: import a mailbox's recent history on connect and govern sync by fair use: a backfill on every provider (newest first, inside an operator-editable window and cap, resumable through a durable per-provider cursor relayed as SYNC_STATE), a per-mailbox sync governor with priority, live and backfill lanes on shared Redis windows that defers over-budget mail with the cursor held instead of dropping it and only deactivates a mailbox for a flood or chronic daily overage, sync.* budgets on the admin instance settings shipped inside ADD_EMAIL, saved IMAP folder cursors and last_synced_at finally written, a Sync card in the mailbox drawer fed by GET /emails/:id/sync, and docs 2026-08-18 09:09:52 -07:00
Matthew Meszaros 93e8451738 feat: organization data export and import for moving a workspace between instances (#132)
* feat: add the org_export_jobs and org_import_jobs tables plus the models behind them, so a whole organization can be written to a portable archive and read back on another instance, keeping the option columns typed (a text[] of data groups, an include_secrets boolean, a conflict_strategy check constraint) rather than a settings blob because the option set is small and fixed, and reserving jsonb only for the genuinely free-form parts that are read back for display alone (the source archive's manifest, per-table row counts, the import warning list), with partial indexes on the in-flight and expiring rows so the maintenance sweep stays cheap however much transfer history accumulates, an OrgDataGroup catalog that names the twelve slices of a workspace and carries the dependencies between them, and an org_archive audit entity so an export or import rides the existing audit spine into every teammate's dashboard

* feat: add the schema-generic repository behind workspace archives, which reads and writes tables by name rather than through typed structs because that is the only way an archive stays correct as the schema grows, moving rows as jsonb in both directions via to_jsonb on the way out and jsonb_populate_recordset on the way in so Postgres performs every type conversion and no hand-written Go column mapping can drift from arrays, jsonb, tsvector, inet or enums, lifting the pool's 60s statement_timeout inside the export transaction because a full inbox read legitimately runs longer than that, introspecting generated, identity and not-null columns plus primary keys and foreign keys from the catalog rather than trusting a compiled list, and treating identifier safety as structural: table names come from the compiled registry and column names are always intersected against the destination catalog before reaching a query, so nothing out of an uploaded archive is ever interpolated

* feat: add the workspace archive registry and on-disk format, covering all 110 organization-owned relations with their scope SQL, dependency order and per-table policy, plus 10 explicitly excluded ones each carrying the reason it must never travel (the KMS-wrapped org data key, in-flight OAuth handshakes, the websocket outbox, live sessions, a pending deletion that would otherwise schedule the destination workspace for destruction), naming the two key domains separately because Warmbly seals mailbox credentials under the instance CREDENTIALS_ENCRYPTION_KEY and everything else under the per-organization DEK and confusing them produces mailboxes that authenticate against nothing, defining the archive as a plain zip of newline-delimited JSON so an operator can unzip it and read the data in a text editor and so the manifest can be written last yet still be read first, and sealing archive secrets under an argon2id passphrase key with parameters deliberately heavier than the login hash since it is derived once per archive and guards every credential in the workspace against offline grinding

* feat: implement the workspace export and import engines, streaming rows straight through untouched for the tables that have neither secrets nor blobs so a million-row inbox export stays cheap and only decoding the rows that must change, opening every sealed value against whichever key domain wrote it and re-sealing it under the archive passphrase on the way out then against the destination's own keys on the way in, blanking a credential rather than sinking the whole export when one mailbox cannot be read and clearing the guard flag alongside it so no row is left claiming ciphertext it no longer holds, applying an import inside a single transaction because a half-applied workspace is far worse than a long-running one, rewriting the organization id and matching members to destination accounts by email with unresolvable people blanked where the column is nullable and redirected to the importer where it is not, and running transfers in the accepting process rather than through a queue for the one reason that matters: the passphrase is then never written down anywhere

* feat: make the per-organization DEK cache nil-safe in internal/app/cipher so a process built without Redis falls through to KMS on every call instead of dereferencing a nil cache handle, which is what lets warmblyctl run the workspace export and import commands at all: it deliberately attaches Redis as optional because the whole point of that CLI is working while the rest of the instance is down, and the decrypted-key cache was always an optimisation rather than a requirement

* feat: add the hourly workspace-archive maintenance job that deletes finished archives past their seven-day retention window, since each one is a complete copy of a workspace sitting in object storage and must not accumulate, and closes out any export or import whose process died mid-run, which is the necessary counterpart to executing transfers in the accepting process so the passphrase is never persisted: without this sweep a restart would leave a job reporting running forever

* feat: expose workspace export and import over the JWT-only organization routes and wire the service into the backend, gating every endpoint on workspace ownership through the existing requireOrgOwner check rather than a permission bit because an export with credentials is the single most sensitive artifact this product can produce and an import rewrites the workspace wholesale, so both belong at the same level as deleting it, spooling uploads to a temporary file since a zip needs random access and a length that a multi-gigabyte archive cannot supply from memory, handing that file's ownership to the background import so it outlives the request and is closed exactly when the job ends, streaming downloads with the archive's sha256 in a response header, and constructing the service with both key domains plus object storage so an archive can be opened, re-keyed and stored

* feat: add warmblyctl org list, export and import so a self-hoster can move a workspace from the box without a browser, running the same engine in-process against Postgres and adding no HTTP surface to a CLI whose entire trust model is container or host access, resolving --org from whichever handle the operator has (id, slug, or the owner's email), streaming the archive to a file or to stdout so it can be piped straight into ssh with progress still readable on stderr, prompting for the credential passphrase twice through the existing password prompt so the terminal and pipe rules stay identical across every command, and defaulting the import path to a preflight report that names what already exists here and which members have no account before anything is written, with --dry-run to stop there

* feat: add the dashboard API layer for workspace archives, fetching the data-group catalog from the server rather than restating it in the client so a new group appears the moment the backend knows about it, mirroring the server's group-dependency closure in expandGroups so the toggles a user sees always match what the archive actually gets, polling only while a transfer is in flight and dropping to no interval the moment none are active since a running job has no realtime event of its own, and downloading a finished archive as a blob through the authenticated client because the endpoint is bearer-authenticated and a plain anchor href cannot carry the token

* feat: build the Settings and Data dashboard page for exporting and importing a workspace, following the settings section conventions and the in-app confirm rather than window.confirm, defaulting the export to every data group because a migration that quietly leaves data behind is worse than one that takes a while, marking the heavy groups so nobody exports a decade of inbox history unaware, requiring the credential passphrase twice behind a confirm that states plainly what the file will contain, and making the import a two-step flow where a preflight reads the archive and reports its origin, row counts, unsealable credentials, existing rows and unknown members before a single byte is written, so confirming is never a leap of faith

* feat: register the Data settings section in the dashboard rail, route and realtime spine, placing it under Advanced beside the danger zone and gating it to the workspace owner so the nav matches what the endpoints actually allow, and mapping the new org_archive audit entity to the export and import query keys in useRealtimeEvents so an archive starting or landing refreshes the page for every teammate through the existing audit spine rather than a bespoke event

* feat: document workspace export and import as a customer guide registered under Account and team, covering what each of the twelve data groups contains and which four dominate archive size, why credentials need a passphrase to travel at all and what happens to mailboxes when they do not, how members are matched to destination accounts by email and what becomes of anyone without one, the difference between keeping existing rows and replacing them, and a table of what deliberately does not import with the reason for each, because billing, plan overrides, worker placement, sync checkpoints and warmup pool membership belong to an instance rather than to a workspace

* feat: document org list, export and import in the warmblyctl reference and point the deployment guide at them as the supported route between a self-hosted install and the hosted service in either direction, adding every flag with what it does, the two extra environment variables those commands read and the difference between them (a missing KMS provider stops the command because sealed values cannot be opened, while a missing CREDENTIALS_ENCRYPTION_KEY is only a warning that mailbox credentials will not move), the behaviour when Redis is down, and the warning that an archive carrying credentials is the most sensitive file this product produces

* feat: record in AGENTS.md that a migration adding an organization-scoped table is not finished until that table is registered in internal/app/orgtransfer/spec.go, either in Tables with its group and scope or in ExcludedTables with the reason it must not travel, because data left out of the registry is silently absent from every archive and nobody discovers it until a customer's migration lands on the other side missing a feature's data, and spelling out the four things that are easy to get wrong when adding one: dependency order, the group boundary that needs a Requires entry only when a NOT NULL foreign key crosses it, which of the two key domains seals a ciphertext column, and which columns name something only the source instance knows
2026-08-18 07:53:39 -07:00
Matthew Meszaros c39c29ab6b feat: rebuild the new-campaign wizard with animated step transitions, a numbered stepper, the shared Toggle instead of a broken hand-rolled switch, per-step validation that explains itself and a discard guard, register PopoverMenu's click-outside in the capture phase so dropdowns inside dialogs close on click-away, add a Campaigns back link and clickable breadcrumb crumbs, add a From contacts leads picker with category filter and select-all-matching backed by the bulk add_campaigns path whose SQL now scopes campaigns by organization instead of the caller, and stop self-hosted no-billing deployments presenting as a free trial or plan-metered by exposing billing_enabled on GET /auth/config, showing a Self-hosted badge, hiding Billing and Refer & earn, and reporting AI credits as unlimited with the header gauge and cost copy hidden 2026-08-18 07:48:58 -07:00
Matthew Meszaros 8bd2c2b57a feat: make self-hosting work end to end and rewrite the guide around what was tested (#97) 2026-08-13 09:47:46 +02:00
Matthew Meszaros fe6d3769cc feat: document make dev at the top of the AGENTS.md local development targets (readiness waits, seeding, seeded login, Ctrl-C leaves infra up) 2026-07-16 09:51:32 +02:00
Matthew Meszaros 30e8d3c13a feat: restyle the docs site without decorative sidebar icons (frontmatter icons dropped, lucide source plugin disabled, AGENTS.md conventions updated), add LangTab language-logo code tabs, and document the new agent tools plus panel resize/placement/shortcuts in the assistant guide and MCP reference 2026-07-16 07:50:24 +02:00
Matthew Meszaros e2ccd0d75a feat: record the new operational contracts in the agent notes and env example - CODEC_PROVIDER=json wherever workers run, CREDENTIALS_ENCRYPTION_KEY for credential sealing, the two native workers, and the make sandbox entry point 2026-07-11 17:31:58 +02:00
Matthew Meszaros 0baa48991b Publish development docs to docs.warmbly.com 2026-06-28 13:25:08 +02:00
Matthew Meszaros 7813d173b1 Move development docs to docs/development/ 2026-06-28 12:25:08 +02:00
Matthew Meszaros 8732805934 feat: replace signed click redirects with server-side link tickets (tracked_links store, internal resolver API, opaque /c/<id> URLs, layered anti-probe caches with miss budget and circuit breaker) removing TRACKING_LINK_SECRET entirely 2026-06-11 09:30:21 +02:00
Matthew Meszaros 2c5e8b2cbd feat: make TRACKING_LINK_SECRET a required boot-time secret on backend and tracking service with no unsigned mode and no rotation grace, so rotating the key revokes old links immediately 2026-06-11 09:00:04 +02:00
Matthew Meszaros 515efce991 feat: support TRACKING_LINK_SECRET_PREVIOUS rotation grace on the tracking service so rotating the click-signing key never breaks links in already-delivered emails 2026-06-11 08:53:29 +02:00
Matthew Meszaros a365aa1605 feat: document realtime collaboration architecture (audit spine, presence conventions, org event gating) and tracking anti-abuse layer in agent notes 2026-06-11 08:17:04 +02:00
Matthew Meszaros bd7db069ba Merge origin/main into feature/integrations-3, resolving doc conflicts by accepting the docs-restructure deletions 2026-06-10 18:56:47 +02:00
Matthew Meszaros 49a4e1e8ec feat: shorten verbose code comments in the docs app and add a one-line comment rule to the agent notes 2026-06-10 18:36:29 +02:00
Matthew Meszaros 519aeacfc7 feat: require docs updates for user-visible changes and tighten commit message guidance in the agent notes 2026-06-10 18:27:42 +02:00
Matthew Meszaros fc33acc64d feat: document the per-organization DEK model
Update the encryption sections in the agent notes, internal API auth
doc, OAuth setup doc, and architecture notes, and state explicitly that
per-user DEKs must not be reintroduced.
2026-06-10 17:17:19 +02:00
Matthew Meszaros 82f3677a94 feat: add site copy style guidance
Document the house rule for avoiding overused em dashes in user-facing site and web copy.
2026-06-06 11:54:34 +02:00
Matthew Meszaros 8b93230c19 feat: document data modeling guidance
Add repository guidance for typed columns versus jsonb configuration blobs and track the schema placeholder.
2026-06-06 07:50:46 +02:00
Matthew Meszaros 78decdb0f5 feat: document dashboard interaction rules 2026-06-04 08:52:57 +02:00
Matthew Meszaros 3943b8a2e7 feat: update docs for postgres-backed secrets
Refresh developer docs, deployment notes, and public-site copy to describe the Postgres-backed encrypted key and message-map model after DynamoDB removal.
2026-06-02 15:55:01 +02:00
Matthew Meszaros d08c984c42 feat: snapshot current dashboard changes 2026-05-30 13:56:27 +00:00
Matthew Meszaros 44c837d8ac Merge branch 'main' into feature/dashboard-plans
# Conflicts:
#	AGENTS.md
2026-05-30 09:49:12 +00:00
Matthew Meszaros d28c607013 feat: document api release standards 2026-05-30 04:35:26 +00:00
Matthew Meszaros 60773b3d8e feat: make dashboard realtime 2026-05-30 04:17:10 +00:00
Matthew Meszaros 7f830b976f feat: sync org session, harden campaign loading, add mailbox bulk-remove 2026-05-30 03:42:39 +00:00
Matthew Meszaros 090cc1c832 feat: rework dev workflow with native go services and dockerized infra 2026-05-30 02:57:27 +00:00
Matt e73ff0aa86 ci(go): gofmt fixes and AGENTS.md CI rules
Two files were unformatted, tripping the golangci-lint gate that runs
gofmt:

  - cmd/backend/main.go: dailythrottle import out of alphabetical order
  - internal/repository/pg_admin_outreach.go: numbered list comment used
    three-space indentation; gofmt wants two

Both fixed by running gofmt -w against the offending files.

Add a "Working In This Repo" section near the top of AGENTS.md (which
CLAUDE.md symlinks to) documenting the rules this PR violated:

  - go build is not the ship signal; CI runs gofmt via golangci-lint
  - always gofmt -w changed Go files before considering work done
  - run the right typecheck/lint step in each frontend tree before
    pushing
  - commit messages do not carry Co-Authored-By or other AI/agent
    attribution footers
2026-05-29 04:57:31 +02:00
Matthew Meszaros 734d149072 site: social avatar + full SEO icon set + Mindroot Ltd footer + home pricing restore
Brand and assets:
- New /public/brand/social-avatar.jpg: the official Warmbly social profile
  picture (paper-plane mark on a sky-with-clouds canvas, 1024x1024).
- scripts/gen-icons.mjs generates the full favicon set from that source
  using sharp. Rounded variants (16/32/48/96) so the icon reads as a
  soft chip in browser tabs, square apple-touch-icon (iOS rounds it),
  square maskable manifest icons (192/512), and a 1200x630 OG card on a
  sky gradient with the rounded avatar on the left.

SEO / head:
- Layout.astro head now declares all favicon sizes, full Open Graph
  set (type, site_name, url, locale, image w/h/alt), Twitter card with
  summary_large_image, application-name and PWA capability meta, dual
  light/dark theme-color tied to the sky tokens, format-detection off
  for phone-number autolink, and crawler hints with
  max-image-preview:large.
- Two JSON-LD blocks: Organization (legalName Mindroot Ltd, postal
  address, two ContactPoints, Companies House identifier, sameAs
  GitHub) and WebSite (publisher reference). Drives rich-result
  eligibility and gives crawlers a clean entity to attach signals to.
- site.webmanifest expanded: real name, description, start_url, scope,
  full icon set including the 96 rounded chip and apple-touch-icon
  alongside the maskable 192/512s, sky-token theme/background colors.

Brand page:
- New Social avatar section after Marks: 1024 source preview, round
  and square crop previews at 48/64/80/112, download CTA pointing at
  /brand/social-avatar.jpg. Asset manifest gets the avatar entry.

About page:
- Repo surface list now includes site/ (this marketing site, Astro 5
  + Tailwind v4), deploy/, docs/, resources/, scripts/. AGENTS.md
  (CLAUDE.md symlinked) System Shape section updated with the same
  extra paths so the agent context stays in sync.

Footer:
- UK Companies Act s.82 disclosure restored with real entity info:
  Mindroot Ltd, company number 16543299, registered office at
  71-75 Shelton Street, London, England, WC2H 9JQ. Copyright line
  flipped to Mindroot Ltd.

Home pricing regression fix:
- The home pricing cards had been flattened into a single edge-to-edge
  grid with no rings, no featured-card highlight, and no animations,
  while the toggle thumb had no inline width/left so it disappeared
  whenever motion.dev had not booted yet (the 'defected switcher'
  the user reported).
- Restored the premium card layout: each plan is a separated rounded
  card with its own ring, the featured plan picks up ring-2 of the
  brand color, a cloud bloom in the top-right, a continuously sweeping
  highlight (pricing-sweep keyframes already present in the page
  stylesheet), and a 'Most popular' pill.
- Toggle thumb now starts at left:116px width:112px so the Annual
  default is visually selected on first paint even before the JS
  upgrade animates it.
2026-05-27 05:19:52 +00:00
Matthew Meszaros d8d88c7f69 feat: add warmup health tracking, migrate repos to postgres, and overhaul web UI 2026-04-03 06:08:52 +00:00