starred/warmbly
1
0
Fork 0
mirror of https://github.com/warmbly/warmbly.git synced 2026-09-06 08:01:24 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,681 Commits 1 Branch 8 Tags
4fa20644c93a7c7ed05b2e94de972c05f551563d
Commit Graph
54 Commits
Author SHA1 Message Date
Matthew Meszaros 15e139e15d feat: stop a campaign email going out twice when the progress write after dispatch is lost: a step is now RESERVED before its SEND_EMAIL reaches the bus (migration 000093 adds campaign_contact_progress.dispatched_at + dispatch_task_id, and ReserveSend takes the claim and the day's counters in one transaction) and routing treats a step as attempted on sent_at OR dispatched_at, so a crash or a failed stamp in the dispatch window can no longer read as "never sent" and email the same person again; the ON CONFLICT claim is exactly-once so two ticks racing the same pair cannot both send (the loser ends skipped_duplicate), the stamp is retried and escalated to the campaign feed instead of warned and swallowed, HandleEmailSent repairs a lost stamp from the worker's own confirmation, ReleaseSend gives a reservation back only when the command provably never left (a publish failure is ambiguous via ErrSendDispatchUnknown and keeps it), and StartStuckSendReclaimer walks back a reservation nobody answered after 30 minutes so a worker that died mid-send cannot park a lead in flight forever; live-tested in TestLiveLostProgressWriteDoesNotResend, TestLiveDispatchedSendIsNeverOfferedTwice, TestLiveConcurrentTicksSendOnce, TestLiveStuckDispatchIsReclaimed, TestLiveReclaimBelievesADeliveredSend and TestLiveInFlightSendIsNotOfferedAgain 2026-08-24 09:15:06 -07:00
Matthew Meszaros 8e0043878c feat: consume the worker's EMAIL_SENT and EMAIL_FAILED results in the consumer, which until now dropped them with a warning so a send the worker could not complete stayed stamped as sent and the lead sat at 'processing' forever: a failure now marks the task failed, walks the step back, gives the day's counters back, writes a red entry to the campaign activity feed, reopens a campaign that completed while the send was in flight, marks the lead failed after five attempts, and routes a recipient refused at RCPT into the bounce pipeline instead of retrying it; a one-off compose or reply failure is pushed to the mailbox owner; covered by an opt-in live test against Postgres and documented in the events page, the campaign guide, the contacts API reference and the agent notes 2026-08-23 10:18:40 -07:00
Matthew Meszaros a75ea012a0 feat: import a mailbox's recent history on connect and govern sync by fair use: a backfill on every provider (newest first, inside an operator-editable window and cap, resumable through a durable per-provider cursor relayed as SYNC_STATE), a per-mailbox sync governor with priority, live and backfill lanes on shared Redis windows that defers over-budget mail with the cursor held instead of dropping it and only deactivates a mailbox for a flood or chronic daily overage, sync.* budgets on the admin instance settings shipped inside ADD_EMAIL, saved IMAP folder cursors and last_synced_at finally written, a Sync card in the mailbox drawer fed by GET /emails/:id/sync, and docs 2026-08-18 09:09:52 -07:00
Matthew Meszaros 0ae4db2c41 feat: make self-hosted auth work without a mail relay by rewriting the platform SMTP transport with real AUTH and TLS (it did neither, so SMTP_USERNAME/SMTP_PASSWORD were dead and every documented relay was unreachable), adding MAIL_TRANSPORT=smtp|log|ses with a log transport that prints codes so a fresh install can sign in with no relay, demoting the emailed login code to AUTH_LOGIN_CODE=always|new_device|off (off on self-host, per NIST SP 800-63B and OWASP ASVS), claiming the first owner through a single-use setup link or WARMBLY_BOOTSTRAP_* instead of register-then-psql, deriving every emailed URL from APP_URL rather than a hardcoded app.warmbly.com that leaked live reset tokens to the vendor, fixing the confirm hooks that read path params against paramless routes and broke login, register and reset confirmation in the dashboard everywhere, adding generic OIDC with PKCE, one-time state, verified nonce and (issuer,subject) identity binding, enforcing 2FA on the social paths that skipped it, adding a per-IP limiter and trusted-proxy handling to the unthrottled auth group, refusing boot on the published default secrets, and dropping mailpit from the default stack (#99) 2026-08-14 14:57:09 +02:00
Matthew Meszaros bb040828b1 feat: rework automation AI steps to mirror campaign step types with one agentic AI step plus an AI switch, per-case switch handles on the canvas, a Stop node, and a step-type add menu 2026-07-22 10:58:35 +02:00
Matthew Meszaros 93b64e2afa feat: add shared run time listing and arg resolving for ai agents 2026-07-21 17:35:11 +02:00
Matthew Meszaros 8055c2f05f Merge remote-tracking branch 'origin/main' into self-host/local-stack 
# Conflicts:
#	cmd/backend/main.go
 2026-07-20 11:53:21 +02:00
Matthew Meszaros 543d97f6f3 feat: wire the no-cloud providers into the consumer and worker 2026-07-20 09:56:29 +02:00
Matthew Meszaros a534ef1908 feat: notification email channel becomes digest-first: email-channel notifications queue as pending rows (migration 000076 adds group_key/email_state/email_due_at/email_attempts) with a due time from a new per-user email_digest cadence (instant/smart 15m/hourly/daily, security sign-ins always immediate), a 30s flush loop with SKIP LOCKED claims bundles a user's pending rows into one digest email and coalesces org-shared group_key events into a single email with every recipient in To, reading a notification in-app cancels its pending email, and NotifyOrg targets only members holding a permission (Slack fires once per group) with new producers: dead-worker downtime to manage_emails members (SetNX-deduped per incident), trial expiry to manage_billing members via new billing_alert category (replacing the direct owner email), and invitation accepts to manage_team members via new team_activity category 2026-07-20 07:16:25 +02:00
Matthew Meszaros 053ca6a71d feat: usage-based AI credit system — every AI call reserves its flat minimum then settles the real token cost per model (light 1500/standard 400 tokens per credit, drain-to-zero overage settle wired into writing assistant, reply drafts, dashboard agent, inbox agent, research, automation nodes, and campaign switches), org spend controls in a new org_ai_settings table (day/week/month hard limits enforced in Consume, low-balance alert threshold, auto top-up config), a credit-watch monitor hook that fires BILLING_CREDITS_LOW realtime alerts once per day and buys the configured pack off-session via a new Stripe AutoTopUpCredits (idempotent on the PaymentIntent, bounded per month, Redis-locked), GET /subscription/credits/usage + GET/PATCH settings endpoints, and an AI usage & spend controls billing card (spend vs limits, 30-day chart, per-feature and per-model breakdowns, limits/reminder/auto-top-up form) with a realtime low-credit toast 2026-07-18 08:39:41 +02:00
Matthew Meszaros cd5e1e9cf7 Merge pull request #72 from warmbly/research/warmup-abuse-cold-effectiveness 
feat: deliverability research + observe-only SPF/DKIM/DMARC auth state and watch-band cold throttle
 2026-07-17 05:29:58 +02:00
Matthew Meszaros 29e3bb3a64 feat: drop the legacy OPENAI_*/ANTHROPIC_API_KEY/AI_LOCAL_MODEL env fallbacks (no prod deployments exist): AI_PROVIDER/AI_API_KEY/AI_MODEL/AI_BASE_URL/AI_FREE are the only AI vars, empty provider means openai, the anthropic preset and writing client key off AI_API_KEY, and the warmup Batch generator runs only when the selected provider is OpenAI 2026-07-16 06:39:06 +02:00
Matthew Meszaros 917451b105 feat: select the AI provider via AI_PROVIDER + AI_* vars in the backend and consumer (legacy OPENAI_*/ANTHROPIC_API_KEY still honored as fallbacks), logging a clear message when the provider is misconfigured 2026-07-16 06:14:19 +02:00
Matthew Meszaros d94bd70f88 feat: wire AI_LOCAL_MODEL into the backend and consumer provider config and pass the agent voice preamble into the dashboard agent service 2026-07-15 18:51:51 +02:00
Matthew Meszaros 78ff73e45c feat: ground inbox-agent reply drafts in the counterpart contact's CRM record - the inbox agent now folds the replying contact's name, company, and known custom fields into the draft prompt (mirroring the manual unibox reply draft) via an optional ContactReader wired from repository.ContactRepository in both the backend and consumer, best-effort so a lookup miss just drops the grounding, so suggested replies reference what Warmbly already knows about the person 2026-07-14 07:29:56 +02:00
Matthew Meszaros 56de65380b feat: inbox agent that drafts a suggested unibox reply on inbound human replies for human approve/edit/discard - paid + per-org opt-in (organizations.inbox_agent_enabled) feature where the consumer's reply hook, on a non-automated reply, detaches a goroutine (panic-contained, never blocks ingest) that checks entitlement, dedupes via ai_thread_drafts partial unique indexes (one pending per thread, unique source_message_id), pre-checks balance, grounds a reply in the thread history + org voice + skills via generation.Provider.Complete, reserves a draft row then charges 5 credits (idempotency inbox_agent:<draft.ID>, row unwound on a fresh context if the charge fails so no unpaid draft lingers), and emits an org-scoped AI_DRAFT_READY event gated on access_unibox; the agent never sends - only a human POST to /unibox/agent-drafts/:id/approve sends through the normal reply path, claiming pending->approved before send with an approved->pending revert on send failure, alongside list + discard endpoints; plus the unibox awaiting_agent_draft badge + agent_drafts search scope, an AgentDraftCard in the thread view with inline-editable approve-and-send/discard, the workspace settings opt-in toggle, CanUseInboxAgent paid gate, migration 000065, and docs (inbox-agent guide, endpoints, realtime) 2026-07-14 06:13:27 +02:00
Matthew Meszaros a067bf5922 feat: AI action nodes in automations - ai_classify/ai_extract/ai_generate run one generation.Provider completion per node over the event data and merge the result back as a variable (ai_class, output_keys[], ai_text) so downstream conditions branch on it, charging 1 credit per node with a run_id+node_id idempotency key, refund on provider failure, out-of-credits fails only that node while the run continues, and a per-automation ai_credit_failures counter (migration 000064) auto-pauses a flow after 20 consecutive credit misses; classify/extract run deterministic (temperature 0), dry-run executes AI nodes for real and is charged but never touches the auto-pause lifecycle, provider + credit ledger wired into both backend and consumer integration services, plus the reply-classifier Layer 3 ported off its bespoke OpenAI client onto the platform provider via an injected ModelClassifyFunc (platform-paid, deterministic), a CompletionRequest.Temperature primitive threaded through the OpenAI and Anthropic providers, the flow builder palette/config panels/validation for the three AI steps, and a new AI-steps-in-automations guide 2026-07-14 05:30:55 +02:00
Matthew Meszaros d6b7885e8b feat: mobile push notifications end to end - APNs provider-token client, device_tokens table with session-scoped register/delete endpoints, a push channel in notification preferences (web + iOS toggles), and Redis-backed immediate-then-digest batching (first event pushes now, bursts summarize when the 5h window closes) wired in backend and consumer, with iOS registration/badge sync and docs for the channel, endpoints, and APNS_* deploy env 2026-07-13 16:11:15 +02:00
Matthew Meszaros 6bc2073379 feat: honor CODEC_PROVIDER=json across backend and consumer (previously worker-only) and round-trip untyped event bodies into their typed payloads in both dispatchers - the worker command/result envelopes carry any-typed bodies that Avro cannot serialize, so worker messaging only works on the JSON codec; also wires the credentials encrypter into both mains 2026-07-11 17:31:02 +02:00
Matthew Meszaros eb0a5654fd feat: persist the opaque per-folder Graph delta cursor in a new email_delta_links table via a repository and a GRAPH_DELTA_UPDATE consumer handler wired into the consumer service 2026-07-04 08:07:49 +02:00
Matthew Meszaros ffe823cfa8 feat: persist and surface per-mailbox SPF/DKIM/DMARC authentication state via an hourly observe-only consumer sweep, threaded through the mailbox list/detail API and documented in the deliverability guide and mailboxes API reference 2026-07-01 02:32:23 +00:00
Matthew Meszaros c5dfa5e4e7 feat: remove the HTTP-request action from campaign steps and automations in favor of signed webhooks, keep fire_event for custom payloads, and drop the now-unused outbound quota plumbing 2026-06-15 08:11:35 +02:00
Matthew Meszaros 20935ef061 feat: add a nil-safe per-org daily outbound-action quota (Redis daily counter, anti-abuse ceiling on the HTTP-request automation node, wired in both backend and consumer, fail-open) to bound webhook relay abuse 2026-06-13 13:41:20 +02:00
Matthew Meszaros 7901e9a92c feat: wire the Pub/Sub realtime provisioning, the instant-branch automation runner, and the campaign-chain reconciler into the backend and consumer binaries 2026-06-13 06:27:06 +02:00
Matthew Meszaros ba89a932cd feat: run the label_email action on reply — campaign instant and scheduled paths via advanced LabelThread/LabelLatestThreadForContact, automations via execNativeAction reading thread_id and _user_id from the reply payload — and move the native-actions adapter to a shared package wired into the consumer too so reply/bounce/warmup automation actions stop silently failing 2026-06-12 16:44:16 +02:00
Matthew Meszaros ba3ab98e1e feat: bridge realtime events over Redis pub/sub when Google Pub/Sub is unconfigured (RedisBus publisher in backend/consumer plus Realtime.Redis.EventSubscriber and a shared EventBroadcaster) so dashboard live updates and presence-driven collaboration actually fire in local dev 2026-06-12 04:49:58 +02:00
Matthew Meszaros 160dc0bc76 feat: implement the coming-soon notification delivery channels — Email (SES/SMTP to the account email) and Slack (posts to the org's connected workspace via a new integration NotifySlack), wired in both backend and consumer with per-channel gating, real toggles replacing the coming-soon labels, and updated docs 2026-06-11 12:21:48 +02:00
Matthew Meszaros abdfd05c34 feat: org-scope realtime events (inbox, campaign, tracking, account health) and emit EMAIL_SENT/EMAIL_REPLIED/EMAIL_DELETED pulses so the whole team's dashboard updates live 2026-06-11 07:43:57 +02:00
Matthew Meszaros 4013b33008 feat: expose security notification APIs 
Wires notification and two-factor services into the backend and consumer, adds the authenticated notification feed/preferences endpoints, and exposes the public 2FA login verification endpoint.
 2026-06-08 15:04:47 +02:00
Matthew Meszaros 51c99458ad feat: fire instant actions from tracking events 2026-06-07 17:24:32 +02:00
Matthew Meszaros da82993846 feat: add campaign workflow backend support 
Add sequence action-node storage and execution, template conditional rendering, lead progress state, profile updates, webhook fan-out throttling, and supporting repository fixes.
 2026-06-06 07:49:52 +02:00
Matthew Meszaros 35fbf73347 feat: persist warmup engagement dwell 
Store delayed warmup engagement actions in Postgres and drain them from the consumer so read, important, and star actions survive worker restarts.

Keep foldering and spam rescue immediate while routing delayed actions to the mailbox's current worker at fire time.
 2026-06-03 11:14:02 +02:00
Matthew Meszaros 59d253ef1d feat: harden warmup action handling 
Wire warmup health webhooks in the consumer and log skipped warmup actions when a recipient mailbox has no assigned worker.

Run foldering and spam rescue immediately while keeping only low-stakes engagement signals behind the dwell delay.
 2026-06-03 06:40:35 +02:00
Matthew Meszaros 15ef9d4994 feat: add warmup content controls 
Add warmup content generation and admin review surfaces, plus mailbox warmup appeal/status APIs.

Track warmup engagement and tampering signals so unsafe mailboxes can be handled by the warmup flow.
 2026-06-03 05:05:53 +02:00
Matthew Meszaros 2430095197 feat: move infrastructure state off dynamodb 
Remove DynamoDB-backed storage paths, add Postgres/HTTP repositories for mailbox state maps, wire the internal message-map API, and add provisioning runner/migration plumbing.
 2026-06-02 15:54:12 +02:00
Matthew Meszaros 83eae84d49 feat: merge main into warmup branch 
Resolve the main-branch conflict in realtime event publishing by keeping both warmup account-health and audit-created events. Renumber the warmup migrations after the current main migration tail.
 2026-06-01 11:36:04 +02:00
Matthew Meszaros a04d7450bc feat: guard warmup reconciliation by access 
Make the warmup task reconciler re-check account state and org warmup access before scheduling replacement tasks. Remove pool membership for accounts whose org can no longer use warmup to avoid repeated no-access task churn.
 2026-06-01 11:19:44 +02:00
Matthew Meszaros 4500aeba0f feat: add integration oauth connections 
Adds OAuth-backed integration connection management across the API, repository, event dispatch, migrations, docs, and dashboard UI.

Includes realtime invalidation and small dashboard type compatibility fixes needed for the web typecheck gate.
 2026-06-01 04:25:14 +02:00
Matthew Meszaros d2414ad29f ci(go): gofmt all flagged files 
Go CI fails on golangci-lint's gofmt check. Ran gofmt -w against
every file the linter named plus a handful of others that drifted
during the autonomous-fleet work. No semantic changes — alignment
of struct field whitespace and one mis-indented import block.

gofmt -l ./... is now empty; go build + go vet are clean.
 2026-05-27 16:40:23 +00:00
Matthew Meszaros 51506d98f4 cmd: wire pluggable infra (KMS, encryptedkeys, eventbus, codec, settings) 
cmd/backend/main.go:
  - kms.FromEnv replaces kms.New (defaults to AWS, accepts local)
  - encryptedkeys.FromEnv with Deps{DB, Dynamo}; default postgres
  - codec.NewAvroFromClient wraps the existing Schema Registry client
  - eventbus.FromEnv with kafka default; KafkaBus.Producer().WithAvrov2
    preserves the existing Avro wire format on Kafka
  - events.NewPublisher takes (bus, codec) instead of (producer, avrov2)
  - settings.Registrar reflects KMS / EncryptedKeys / Blob / EventBus
    choices into storage_backends on boot
  - Handler gains EncryptedKeys + StorageBackendRepo for the new admin
    and internal endpoints

cmd/consumer/main.go: same eventbus + codec + encryptedkeys swap; the
legacy kafkaProducer is kept around for the consumer's tracking pipeline
which still uses *kafka.Consumer directly (follow-up refactor).
 2026-05-27 14:43:51 +00:00
Matthew Meszaros 7e02bb2a5a feat(consumer): hourly risk rebalancer migrates mailboxes between risk pools 
New background job in the consumer process:

  1. Pulls up to 1000 mailbox candidates joined with their worst warmup
     health state (across all pools they participate in) and their
     current worker's risk_pool. Dedicated workers are excluded — single
     tenant, segregation not applicable.
  2. Recomputes risk_band from health state via RiskBandFromHealth.
     If it changed, writes the new band.
  3. If the band's matching pool doesn't equal the worker's pool, picks
     a new worker via SelectSharedWorkerForBand and migrates the mailbox.
     Increments/decrements account counts.
  4. Logs each migration to admin_audit_log with action=
     "risk_rebalance_migrate" so operators see what moved and why.

Boot-time run + hourly ticker. Rebalancing is intentionally batch, not
event-driven: warmup health states change on a slow rolling-window basis
(warmup_health_sweep is also hourly), so reacting in real time gains
nothing and would cause thundering-herd migrations.

JobsService gets an AssignmentService dep. Nil disables the job (lets
self-hosters opt out by simply not wiring it).
 2026-05-19 05:39:27 +00:00
Matthew Meszaros 2aefb7da02 feat(consumer): log auto-reassignment events to admin_audit_log 
When the dead-worker job reassigns email accounts from a worker whose
heartbeat expired, write a row into admin_audit_log so the dashboard's
audit viewer surfaces these system actions alongside admin-driven ones.

admin_user_id is uuid.Nil (the platform identity), so admins searching
the log can distinguish "system did this" from "an admin did this" by
filtering on that ID. Details include the replacement worker, account
count, and reason.

JobsService gets an optional AdminRepo dep. Nil disables logging — keeps
the contract loose for any other call site that doesn't have one.
 2026-05-19 05:21:26 +00:00
Matthew Meszaros 8ae7759964 feat(consumer): sync worker heartbeats from Redis to workers.last_seen_at 
Workers heartbeat into Redis every 90s as RFC3339 timestamp values with a
3-min TTL. The dashboard surfaces liveness based on workers.last_seen_at,
but until now nothing populated that column — the "Live" badge was always
red.

New 60s job in the consumer reads each active worker's Redis heartbeat
value, parses the timestamp, and writes it to workers.last_seen_at. Runs
on its own interval (separate from the 5-min dead-worker detection job,
which does heavier reassignment work) so the UI sees fresh data within a
minute.
 2026-05-18 14:55:34 +00:00
Matthew Meszaros 1ad8e7e995 feat: implement dead worker detection with heartbeat TTL and auto-reassignment 2026-04-12 11:14:24 +00:00
Matthew Meszaros 23b5c925e4 feat: add scheduled warmup health sweep, pool health summary endpoint, and admin overview 2026-04-09 15:57:04 +00:00
Matthew Meszaros 0799020dac feat: add metrics, warmup content variety, tz-aware scheduling, org budget, admin stubs, and bug fixes 2026-04-09 14:33:45 +00:00
Matthew Meszaros 19d20e72b4 feat: add warmup health throttled state, complaint/bounce metrics, A/B analysis, and rate limiting 2026-04-09 12:51:30 +00:00
Matthew Meszaros d8d88c7f69 feat: add warmup health tracking, migrate repos to postgres, and overhaul web UI 2026-04-03 06:08:52 +00:00
Matthew Meszaros 21ffb6a748 feat: add advanced outreach controls with A/B testing, deliverability dashboard, and DLQ 2026-02-20 08:59:17 +00:00
Matthew Meszaros c12fd93def Add local Sentry logging 2026-02-14 05:49:49 +01:00