diff --git a/docs/content/docs/api/endpoints.mdx b/docs/content/docs/api/endpoints.mdx index ac3907d24..eb256883d 100644 --- a/docs/content/docs/api/endpoints.mdx +++ b/docs/content/docs/api/endpoints.mdx @@ -373,6 +373,8 @@ A [placement batch](/guides/placement-tests/#testing-a-fleet-with-batches) runs Recommendations about deliverability, mailbox configuration, warmup, campaign performance, copy, and list quality. See the [Advisor guide](/guides/advisor/). +`GET /advisor/recommendations` returns open findings by default. Applied fixes leave that list immediately; pass an explicit `status=applied` to retrieve them, or `status=resolved` for findings cleared by a recheck. Repeat `status` to request several states. Campaign, step and mailbox changes trigger a background recheck that clears findings which no longer apply. + Reads are an analytics read of the workspace's sending posture. Applying or undoing a fix has no scope of its own: the change runs through the tool the fix actually uses and is refused if the caller lacks that tool's permission, so a key that can read recommendations cannot use them to make changes it could not make directly. | Method | Path | API Permission | diff --git a/docs/content/docs/guides/advisor.mdx b/docs/content/docs/guides/advisor.mdx index 91d30ab63..9b03463b8 100644 --- a/docs/content/docs/guides/advisor.mdx +++ b/docs/content/docs/guides/advisor.mdx @@ -37,11 +37,14 @@ Apply the fix. **Done** offers **Undo** where the change can be reverted. ## Good to know -- Detection is explicit rules with explicit thresholds, not AI, so it works with no AI configured and never spends AI credits. +- Core detection uses explicit rules and thresholds, works with no AI configured, and never spends AI credits. Optional TypeSafe copy reviews are automated judgments. - A check stays quiet until it has enough data, for example `100` sends in 30 days for complaint rate. - Nothing is applied until you have seen the **What changes** screen, and the change runs with your permissions. - Autopilot, under **Settings > Workspace**, is off by default and only takes bounded fixes that can be undone. - **Snooze** hides a finding for a week or a month. **Stop suggesting this** dismisses it until the problem clears and recurs. +- Applied fixes leave the suggestions list immediately. Campaign, step and mailbox changes trigger a background recheck; findings that no longer apply disappear when it finishes. Saved copy is reviewed again if it changed, using cached judgments for unchanged content. If the review service is unavailable, its existing findings stay until a successful review. + +Follow-up spacing advice applies only after the campaign's first email. A zero-day wait on the first email is valid: it can send when the campaign's schedule and sending limits allow. The first email still receives content checks. With TypeSafe configured, a separate copy check can flag wording that reads as bulk marketing; that is a judgment about the message, not a recommendation to delay it or proof that it will land in spam. @@ -75,7 +78,7 @@ If one check fires on many things, they collapse into a single card ("19 mailbox ## How findings are made -**Detection is not AI.** Every check is an explicit rule with an explicit threshold run over your own data, so the same numbers always produce the same result. AI only rewrites the explanation to read more naturally; it cannot create a finding, change its severity, or change what a fix does. The Advisor therefore works fully on a self-hosted install with no AI configured and never spends AI credits. The wording is just plainer. +Core checks use explicit rules and thresholds over your workspace data. Optional TypeSafe copy reviews supply model judgments to the copy checks described below. A separate AI narrator rewrites explanations; it cannot create a finding, change its severity, or change what a fix does. Self-hosts with no AI configured still receive the core checks, and Advisor evaluations never spend AI credits. ### Sample floors @@ -121,7 +124,9 @@ The advisor's placement sample covers today and the preceding six UTC dates, mat **Copy** (conservative, and drafts are skipped): a merge variable that will not resolve, shipping a literal `{{token}}`; two or more bulk-mail phrases (one is never flagged); over `200` words; a subject over `60` characters; more than two links in a first touch; a shouting subject. -Two more copy findings run only when the operator has configured TypeSafe (`TYPESAFE_API_KEY`), a calibrated reader model rather than a language model: a step that **reads as bulk mail** to its recipient, or makes a claim a spam filter objects to (guaranteed results, free money, a prize, pressure to act now); and a step with **no clear ask**, either asking for nothing or for several things. Each step's subject and body are sent to TypeSafe for this, once per version of the copy; the verdict is cached and an unchanged step is never sent again. At most `40` steps are judged per evaluation, and the rest wait for the next run. +Two more copy findings run only when the operator has configured TypeSafe (`TYPESAFE_API_KEY`), a calibrated reader model: an optional **wording review**, and a step with **no clear ask**, either asking for nothing or for several things. The wording review is low severity. Tone advice requires confident agreement that the copy sounds promotional and lacks relevance to the reader or their role. A business offer, a reusable template or a merge variable alone is not enough. Explicit guarantees, free-money or prize promises, and artificial urgency are checked separately. These are model assessments, not measurements of spam placement or reply likelihood. Dismiss a review that does not fit your message. + +Each step's subject and body are sent to TypeSafe for this, once per version of the copy and review policy. Verdicts are cached; saving unchanged content reuses its judgment. A policy update rechecks existing copy. At most `40` steps are judged per evaluation, and the rest wait for the next run. **List quality**: shared inboxes (`info@`, `sales@`) above `15%`; consumer domains above `40%` of a B2B list; suppressed contacts above `10%`; missing first names above `10%` when copy personalizes on them; unsubscribed contacts still enrolled. diff --git a/internal/api/handler/advisor_refresh_test.go b/internal/api/handler/advisor_refresh_test.go new file mode 100644 index 000000000..ee7037ef6 --- /dev/null +++ b/internal/api/handler/advisor_refresh_test.go @@ -0,0 +1,70 @@ +package handler + +import ( + "context" + "net/http" + "net/http/httptest" + "strings" + "testing" + "time" + + "github.com/gin-gonic/gin" + "github.com/google/uuid" + + "github.com/warmbly/warmbly/internal/api/middleware" + "github.com/warmbly/warmbly/internal/app/advisor" + "github.com/warmbly/warmbly/internal/app/audit" + "github.com/warmbly/warmbly/internal/app/sequence" + "github.com/warmbly/warmbly/internal/errx" + "github.com/warmbly/warmbly/internal/models" +) + +type editedSequenceService struct { + sequence.SequenceService + step models.Sequence +} + +func (s *editedSequenceService) Update(_ context.Context, _, _, _ string, _ *models.UpdateSequence) (*models.Sequence, *errx.Error) { + return &s.step, nil +} + +type editAdvisorEvaluator struct { + advisor.Service + refreshed chan uuid.UUID +} + +func (s *editAdvisorEvaluator) Evaluate(_ context.Context, orgID uuid.UUID, _ string) (*models.AdvisorSummary, error) { + s.refreshed <- orgID + return &models.AdvisorSummary{}, nil +} + +func TestSavingSequenceRefreshesAdvisorForItsOrganization(t *testing.T) { + gin.SetMode(gin.TestMode) + orgID, userID, campaignID, stepID := uuid.New(), uuid.New(), uuid.New(), uuid.New() + evaluator := &editAdvisorEvaluator{refreshed: make(chan uuid.UUID, 1)} + h := &Handler{ + SequenceService: &editedSequenceService{step: models.Sequence{ID: stepID}}, + AuditService: audit.NewNoOpService(), AdvisorService: evaluator, + } + r := gin.New() + r.Use(func(c *gin.Context) { + c.Set(middleware.OrganizationIDKey, orgID) + c.Set(middleware.UserIDKey, userID.String()) + }) + r.PATCH("/campaigns/:id/sequences/:sid", h.UpdateSequence) + w := httptest.NewRecorder() + req := httptest.NewRequest(http.MethodPatch, "/campaigns/"+campaignID.String()+"/sequences/"+stepID.String(), strings.NewReader(`{"wait_after":3}`)) + req.Header.Set("Content-Type", "application/json") + r.ServeHTTP(w, req) + if w.Code != http.StatusOK { + t.Fatalf("save returned %d: %s", w.Code, w.Body.String()) + } + select { + case got := <-evaluator.refreshed: + if got != orgID { + t.Fatalf("refreshed organization %s, want %s", got, orgID) + } + case <-time.After(2 * time.Second): + t.Fatal("saving a step did not refresh its advice") + } +} diff --git a/internal/api/handler/audit.go b/internal/api/handler/audit.go index 8c46de9b1..0f192a42b 100644 --- a/internal/api/handler/audit.go +++ b/internal/api/handler/audit.go @@ -37,23 +37,18 @@ func (h *Handler) auditInOrg(c *gin.Context, orgID uuid.UUID, action models.Audi return } h.AuditService.LogAction(c.Request.Context(), orgID, actorID, action, entityType, entityID, c.ClientIP(), c.Request.UserAgent(), changes, metadata) - h.refreshAdvisorAfter(orgID, action, entityType) + h.refreshAdvisorAfter(orgID, entityType) } -// refreshAdvisorAfter re-evaluates the advisor after a change it reads, so its -// advice follows the change instead of the next scheduled pass. Campaign and -// step edits are left to the schedule: they autosave, and each pass re-judges copy. -func (h *Handler) refreshAdvisorAfter(orgID uuid.UUID, action models.AuditAction, entityType models.AuditEntityType) { - switch { - case entityType == models.AuditEntityEmailAccount, - action == models.AuditActionDelete && (entityType == models.AuditEntityCampaign || entityType == models.AuditEntitySequence), - entityType == models.AuditEntityCampaign && (action == models.AuditActionStart || action == models.AuditActionStop): +// Campaign and step edits refresh findings; concurrent refreshes coalesce per org. +func (h *Handler) refreshAdvisorAfter(orgID uuid.UUID, entityType models.AuditEntityType) { + switch entityType { + case models.AuditEntityEmailAccount, models.AuditEntityCampaign, models.AuditEntitySequence: h.refreshAdvisorNow(orgID) } } -// refreshAdvisorNow re-evaluates the advisor now, for a change it reads that is -// not an autosave (a campaign's sender pool, its start or stop). +// refreshAdvisorNow requests a background evaluation of the organization's current state. func (h *Handler) refreshAdvisorNow(orgID uuid.UUID) { if h.AdvisorService == nil { return diff --git a/internal/app/advisor/detect_campaign.go b/internal/app/advisor/detect_campaign.go index 8b092b17e..d213eafce 100644 --- a/internal/app/advisor/detect_campaign.go +++ b/internal/app/advisor/detect_campaign.go @@ -4,6 +4,8 @@ import ( "fmt" "math" + "github.com/google/uuid" + "github.com/warmbly/warmbly/internal/models" "github.com/warmbly/warmbly/internal/repository" ) @@ -230,7 +232,7 @@ func detectStepDropoff(s *repository.AdvisorSnapshot) []Finding { label := st.Name if label == "" { - label = fmt.Sprintf("Step %d", st.Position+1) + label = fmt.Sprintf("Step %d", st.Position) } out = append(out, Finding{ Key: "campaign_step_dropoff", @@ -252,7 +254,7 @@ func detectStepDropoff(s *repository.AdvisorSnapshot) []Finding { Evidence: map[string]any{ "campaign": names[cid], "step": label, - "step_position": st.Position + 1, + "step_position": st.Position, "step_sends": st.Sent, "step_reply_rate_percent": band(r), "best_step_rate_percent": band(best), @@ -451,15 +453,24 @@ func noSenderCopyFor(camp repository.AdvisorCampaign) noSenderCopy { func detectFollowUpSpacing(s *repository.AdvisorSnapshot) []Finding { names := map[string]string{} statuses := map[string]string{} + firstEmail := map[uuid.UUID]int{} for _, camp := range s.Campaigns { names[camp.ID.String()] = camp.Name statuses[camp.ID.String()] = camp.Status } + for _, st := range s.Steps { + if st.Kind != "email" { + continue + } + if position, ok := firstEmail[st.CampaignID]; !ok || st.Position < position { + firstEmail[st.CampaignID] = st.Position + } + } out := []Finding{} for _, st := range s.Steps { - // Position 0 is the first email; its wait is a start delay, not spacing. - if st.Kind != "email" || st.Position == 0 { + // The first email's wait is a start delay, not follow-up spacing. + if st.Kind != "email" || st.Position == firstEmail[st.CampaignID] { continue } if statuses[st.CampaignID.String()] == "draft" { @@ -471,7 +482,7 @@ func detectFollowUpSpacing(s *repository.AdvisorSnapshot) []Finding { label := st.Name if label == "" { - label = fmt.Sprintf("Step %d", st.Position+1) + label = fmt.Sprintf("Step %d", st.Position) } var detail, remedy string @@ -510,7 +521,7 @@ func detectFollowUpSpacing(s *repository.AdvisorSnapshot) []Finding { "step": label, "wait_days": st.WaitAfter, "recommended": want, - "step_position": st.Position + 1, + "step_position": st.Position, }, Action: withUndo(toolAction("update_campaign_step", fmt.Sprintf("Wait %d days instead", want), diff --git a/internal/app/advisor/detect_copy.go b/internal/app/advisor/detect_copy.go index 64ad7a6ad..9ea8e3eb6 100644 --- a/internal/app/advisor/detect_copy.go +++ b/internal/app/advisor/detect_copy.go @@ -64,7 +64,7 @@ func copyDetectors() []Detector { { Key: "copy_reads_as_bulk", Category: models.AdvisorCategoryCopy, - About: "Copy that a calibrated reader model places at the bulk-mail end of a personal-to-bulk scale, or that makes a claim a spam filter objects to (guaranteed results, free money, prizes, pressure to act now). Unlike the phrase list this judges the whole email as its recipient would, so it catches copy that avoids every trigger word and still reads as a blast. Only runs when the operator has configured TypeSafe.", + About: "An optional automated copy review, not a measurement of spam placement or reply likelihood. It flags tone only when a confident reader model agrees the message sounds promotional and lacks relevance to a specific reader or role. A business offer, reusable template or merge variable alone does not qualify. Explicit guarantees, free-money or prize promises and urgency are checked separately. Present this as a possible wording issue to review, never a fact about how recipients or spam filters will react. Only runs with TypeSafe configured.", Reads: []repository.AdvisorSection{repository.AdvisorSectionCampaigns, repository.AdvisorSectionSteps}, Run: detectReadsAsBulk, }, @@ -127,7 +127,7 @@ func emailSteps(s *repository.AdvisorSnapshot) []stepContext { func stepLabel(sc stepContext) string { name := sc.step.Name if name == "" { - name = fmt.Sprintf("Step %d", sc.step.Position+1) + name = fmt.Sprintf("Step %d", sc.step.Position) } return name } @@ -487,19 +487,18 @@ func detectReadsAsBulk(s *repository.AdvisorSnapshot) []Finding { continue } - reason := "reads as a bulk marketing email rather than a note from one person to another" + reason := "may sound promotional and lack a clear connection to the recipient or their role" + title := fmt.Sprintf("Review the tone of %s", stepLabel(sc)) if v.SpamClaim >= copyjudge.SpamClaimAt { - reason = "makes the kind of claim a spam filter objects to: guaranteed results, free money, a prize, or pressure to act now" - if v.ReadsAs >= copyjudge.BulkAt && v.Confidence >= copyjudge.ConfFloor { - reason = "reads as a bulk marketing email and makes the kind of claim a spam filter objects to" - } + reason = "may contain a guarantee, a free-money or prize promise, or pressure to act immediately" + title = fmt.Sprintf("Review the claims in %s", stepLabel(sc)) } out = append(out, Finding{ Key: "copy_reads_as_bulk", - GroupTitle: "{count} steps read as bulk mail to their reader", + GroupTitle: "{count} steps have wording to review", Category: models.AdvisorCategoryCopy, - Severity: models.AdvisorMedium, + Severity: models.AdvisorLow, Surface: models.AdvisorSurfaceCampaigns, EntityType: "step", EntityID: ref(sc.step.ID), @@ -507,16 +506,16 @@ func detectReadsAsBulk(s *repository.AdvisorSnapshot) []Finding { ParentType: "campaign", ParentID: ref(sc.step.CampaignID), Impact: clampImpact(35 + int(v.ReadsAs*20) + sc.step.Sent/100), - Title: fmt.Sprintf("%s reads as bulk mail to its reader", stepLabel(sc)), + Title: title, Detail: fmt.Sprintf( - "Read as its recipient would read it, %s in %s %s. It passes the phrase list; the problem is the whole email, not a word in it. A person who takes a message for a blast does not reply to it, and a filter that does is right.", + "An automated copy review suggests %s in %s %s. This is a wording assessment, not evidence that recipients will ignore it or that a spam filter will reject it.", stepLabel(sc), sc.campaign, reason), - Remedy: "Rewrite it as you would write to this one person: open with the reason you are writing to them specifically, say what you do in a line, and ask one small question. Drop any promise you could not make face to face.", + Remedy: "Check whether the email explains why the offer is relevant to this recipient, describes what you do plainly, and asks one small question. Edit any unsupported promises or artificial urgency. If the wording already fits, dismiss this suggestion.", Steps: []string{ fmt.Sprintf("Open the campaign and go to %s.", stepLabel(sc)), "Read it aloud as if to the one person it is addressed to. Every line that would be strange to say to them is a line to cut.", "Replace claims (guaranteed, free, act now, limited time) with what is true for this reader: what you noticed, why it is relevant to them.", - "Run Analyze under the content check and confirm it now reads as a personal note.", + "Save any changes to refresh the review, or dismiss it if the assessment does not fit your message.", }, Evidence: map[string]any{ "campaign": sc.campaign, @@ -527,6 +526,7 @@ func detectReadsAsBulk(s *repository.AdvisorSnapshot) []Finding { "spam_claim": round2(v.SpamClaim), "confidence": round2(v.Confidence), "step_sends": sc.step.Sent, + "content_hash": copyjudge.ContentHash(sc.step.Subject, copyjudge.Body(sc.step.BodyPlain, sc.step.BodyHTML)), }, }) } diff --git a/internal/app/advisor/detect_test.go b/internal/app/advisor/detect_test.go index d37f7d458..b75d88b8f 100644 --- a/internal/app/advisor/detect_test.go +++ b/internal/app/advisor/detect_test.go @@ -73,6 +73,38 @@ func findingsByKey(findings []Finding) map[string]Finding { return out } +func TestFollowUpSpacingExcludesTheFirstEmail(t *testing.T) { + for _, firstPosition := range []int{1, 3} { + t.Run(fmt.Sprintf("opener_at_%d", firstPosition), func(t *testing.T) { + campaignID, otherID := uuid.New(), uuid.New() + openerID, followupID := uuid.New(), uuid.New() + s := &repository.AdvisorSnapshot{ + Campaigns: []repository.AdvisorCampaign{ + {ID: campaignID, Name: "Outbound", Status: "active"}, + {ID: otherID, Name: "Other", Status: "active"}, + }, + Steps: []repository.AdvisorStep{ + {ID: followupID, CampaignID: campaignID, Kind: "email", Position: firstPosition + 1}, + {ID: uuid.New(), CampaignID: campaignID, Kind: "action", Position: firstPosition - 1}, + {ID: openerID, CampaignID: campaignID, Kind: "email", Position: firstPosition}, + {ID: uuid.New(), CampaignID: otherID, Kind: "email", Position: 7}, + }, + } + got := detectFollowUpSpacing(s) + if len(got) != 1 || got[0].EntityID == nil || *got[0].EntityID != followupID { + t.Fatalf("only the follow-up should need spacing: %+v", got) + } + if got[0].Evidence["step_position"] != firstPosition+1 { + t.Fatalf("step position must match the stored position: %+v", got[0].Evidence) + } + s.Steps[0].WaitAfter = 3 + if got := detectFollowUpSpacing(s); len(got) != 0 { + t.Fatalf("a three-day follow-up should need no spacing advice: %+v", got) + } + }) + } +} + func TestHealthyMailboxProducesNoFindings(t *testing.T) { got := Detect(snapshotOf(healthyMailbox()), defaults()) if len(got) != 0 { @@ -562,7 +594,7 @@ func TestCopyFindingsAttachToTheirCampaign(t *testing.T) { snap := snapshotOf() snap.Campaigns = []repository.AdvisorCampaign{{ID: campaignID, Name: "Q3 outbound", Status: "active"}} snap.Steps = []repository.AdvisorStep{{ - ID: uuid.New(), CampaignID: campaignID, Kind: "email", Position: 0, + ID: uuid.New(), CampaignID: campaignID, Kind: "email", Position: 1, Name: "Opener", Subject: "quick question", BodyPlain: "Act now, this is a limited time offer with no obligation.", @@ -590,7 +622,7 @@ func TestSingleSpamPhraseIsNotAFinding(t *testing.T) { snap := snapshotOf() snap.Campaigns = []repository.AdvisorCampaign{{ID: campaignID, Name: "Outbound", Status: "active"}} snap.Steps = []repository.AdvisorStep{{ - ID: uuid.New(), CampaignID: campaignID, Kind: "email", Position: 0, + ID: uuid.New(), CampaignID: campaignID, Kind: "email", Position: 1, Subject: "quick question about your onboarding", BodyPlain: "We guarantee delivery within a day. Worth a chat?", }} @@ -693,7 +725,7 @@ func TestValidGoTemplatesAreNotFlagged(t *testing.T) { snap := snapshotOf() snap.Campaigns = []repository.AdvisorCampaign{{ID: campaignID, Name: "Outbound", Status: "active"}} snap.Steps = []repository.AdvisorStep{{ - ID: uuid.New(), CampaignID: campaignID, Kind: "email", Position: 0, + ID: uuid.New(), CampaignID: campaignID, Kind: "email", Position: 1, Subject: "quick question about {{.Company}}", BodyPlain: "Hi {{.FirstName}},\n\n" + `{{if .Company}}Saw {{.Company}} is hiring.{{else}}Saw your team is hiring.{{end}}` + @@ -712,7 +744,7 @@ func TestUnparseableTemplateIsFlagged(t *testing.T) { snap := snapshotOf() snap.Campaigns = []repository.AdvisorCampaign{{ID: campaignID, Name: "Outbound", Status: "active"}} snap.Steps = []repository.AdvisorStep{{ - ID: uuid.New(), CampaignID: campaignID, Kind: "email", Position: 0, + ID: uuid.New(), CampaignID: campaignID, Kind: "email", Position: 1, Name: "Opener", Subject: "quick question", BodyPlain: "Hi {{.FirstName}},\n\n{{if .Company}}Saw you are hiring.\n\nWorth a chat?", @@ -939,7 +971,7 @@ func judgedSnapshot(v *copyjudge.Verdict) (*repository.AdvisorSnapshot, uuid.UUI snap := snapshotOf() snap.Campaigns = []repository.AdvisorCampaign{{ID: campaignID, Name: "Outbound", Status: "active"}} snap.Steps = []repository.AdvisorStep{{ - ID: stepID, CampaignID: campaignID, Kind: "email", Position: 0, + ID: stepID, CampaignID: campaignID, Kind: "email", Position: 1, Name: "Opener", Subject: "quick question", BodyPlain: "Saw your talk on onboarding. Would a short note on how we cut it to a day be useful?", @@ -983,8 +1015,8 @@ func TestReadsAsBulkFiresOnAConfidentBulkVerdict(t *testing.T) { if !fired { t.Fatal("copy_reads_as_bulk did not fire on a confident bulk verdict") } - if f.Severity != models.AdvisorMedium { - t.Errorf("severity should be medium, got %s", f.Severity) + if f.Severity != models.AdvisorLow { + t.Errorf("copy review should be low severity, got %s", f.Severity) } if f.EntityType != "step" || f.EntityID == nil || *f.EntityID != stepID || f.ParentType != "campaign" { t.Error("finding is not attached to its step and campaign") @@ -1016,11 +1048,29 @@ func TestReadsAsBulkNeedsConfidenceUnlessAClaimIsMade(t *testing.T) { if !fired { t.Fatal("copy_reads_as_bulk did not fire on a spam claim") } - if !strings.Contains(f.Detail, "spam filter") { + if !strings.Contains(f.Detail, "may contain") || !strings.Contains(f.Detail, "not evidence") { t.Errorf("detail should say the claim is the problem: %q", f.Detail) } } +func TestCopyReviewRequiresAgreementAndTracksRewrites(t *testing.T) { + snap, id := judgedSnapshot(©judge.Verdict{ + ReadsAs: 1, Personalization: 0, Ask: copyjudge.AskOneClear, Confidence: 0.95, + }) + if got := detectReadsAsBulk(snap); len(got) != 0 { + t.Fatal("a message judged relevant should not get a bulk-tone suggestion") + } + v := snap.CopyJudgments[id] + v.Personalization = 1 + snap.CopyJudgments[id] = v + before := detectReadsAsBulk(snap)[0] + snap.Steps[0].BodyPlain = "A different email with the same review scores." + after := detectReadsAsBulk(snap)[0] + if before.Evidence["content_hash"] == after.Evidence["content_hash"] { + t.Fatal("rewritten copy must invalidate the old explanation even when scores match") + } +} + func TestNoClearAskSaysWhichWay(t *testing.T) { snap, _ := judgedSnapshot(©judge.Verdict{ReadsAs: 0, Ask: copyjudge.AskNone, Confidence: 0.9}) f, fired := findingsByKey(Detect(snap, defaults()))["copy_no_clear_ask"] @@ -1292,13 +1342,13 @@ func busySnapshot() *repository.AdvisorSnapshot { } snap.Steps = []repository.AdvisorStep{ { - ID: uuid.New(), CampaignID: campaignID, Kind: "email", Position: 0, Name: "Opener", WaitAfter: 1, + ID: uuid.New(), CampaignID: campaignID, Kind: "email", Position: 1, Name: "Opener", WaitAfter: 1, Subject: "ACT NOW!! LIMITED TIME OFFER, THIS IS NOT SPAM AND IT IS 100% FREE FOR YOU AND YOUR WHOLE TEAM TODAY", BodyPlain: "Hi {{.FirstName}},\n\n{{if .Company}}Act now, click here, buy now, limited time, no obligation, 100% free, risk free. " + strings.Repeat("More words here. ", 200), Sent: 2000, Replied: 40, }, { - ID: uuid.New(), CampaignID: campaignID, Kind: "email", Position: 1, Name: "Bump", + ID: uuid.New(), CampaignID: campaignID, Kind: "email", Position: 2, Name: "Bump", Subject: "re", BodyPlain: "Hi {{.FirstName}}, any thoughts?", Sent: 1000, Replied: 0, }, diff --git a/internal/app/advisor/service.go b/internal/app/advisor/service.go index 3e11c6101..c1cdffba9 100644 --- a/internal/app/advisor/service.go +++ b/internal/app/advisor/service.go @@ -239,10 +239,8 @@ func (s *service) Evaluate(ctx context.Context, orgID uuid.UUID, trigger string) } s.finishRun(ctx, runID, newCount, summary.Total, closed, narrated, "") - // One audit entry per run, only when something actually changed. The spine - // turns this into a live refresh of every teammate's advisor strips and - // nav badges without a bespoke realtime event. - if (newCount > 0 || closed > 0 || fixed > 0) && s.audit != nil { + // Publish after edits so every client receives the completed review, including updated wording. + if (newCount > 0 || closed > 0 || fixed > 0 || narrated > 0 || trigger == "change") && s.audit != nil { s.audit.LogAction(ctx, orgID, uuid.Nil, models.AuditActionUpdate, models.AuditEntityAdvisorFinding, nil, "", "", nil, map[string]string{ "new": fmt.Sprint(newCount), diff --git a/internal/app/copyjudge/judge.go b/internal/app/copyjudge/judge.go index fc73ea0aa..e74963670 100644 --- a/internal/app/copyjudge/judge.go +++ b/internal/app/copyjudge/judge.go @@ -66,8 +66,8 @@ var ( "A bulk marketing email", } personalizationLevels = []string{ - "Written for this specific reader", - "Could be sent to anyone", + "Relevant to this reader or their role", + "An indiscriminate promotion", } ) @@ -80,11 +80,10 @@ var askOptions = map[string]string{ // Questions is the entire question set, built once for one call. func Questions() map[string]typesafe.Question { return map[string]typesafe.Question{ - qReadsAs: typesafe.Score("How does this email read to the person receiving it?", readsAsLevels), - qPersonalization: typesafe.Score("Is this email written for one specific reader, or could it be sent to anyone?", personalizationLevels), + qReadsAs: typesafe.Score("How does this email read to the person receiving it? Judge the wording, not the fact that it is cold outreach. A relevant business offer or a short reusable template is not by itself bulk marketing. Treat merge variables as filled in for the recipient.", readsAsLevels), + qPersonalization: typesafe.Score("Is this email relevant to a specific reader or role, or is it an indiscriminate promotion? Role or company relevance counts; a name alone does not. Treat merge variables as filled in for the recipient.", personalizationLevels), qAsk: typesafe.Choice("What does this email ask the reader to do?", askOptions), - qSpamClaim: typesafe.Noul("The message makes a claim a spam filter would object to: " + - "guaranteed results, free money, prizes, or pressure to act now."), + qSpamClaim: typesafe.Noul("The message explicitly promises guaranteed results, free money or a prize, or pressures the recipient with urgency. An ordinary business offer, a reported past result, or a polite request for a reply does not qualify. Do not infer spam placement."), } } @@ -102,10 +101,9 @@ type Verdict struct { InputTokens int `json:"input_tokens"` } -// ReadsAsBulk is the Advisor's bulk-mail test: confidently past the bulk -// threshold, or carrying a claim a filter would object to whatever the tone. +// Tone and lack of relevance must agree; an explicit promotional claim is independent. func (v Verdict) ReadsAsBulk() bool { - return (v.ReadsAs >= BulkAt && v.Confidence >= ConfFloor) || v.SpamClaim >= SpamClaimAt + return (v.ReadsAs >= BulkAt && v.Personalization >= BulkAt && v.Confidence >= ConfFloor) || v.SpamClaim >= SpamClaimAt } // LacksClearAsk reports copy that asks for nothing or for several things, when @@ -181,7 +179,7 @@ func Text(body string) string { // cacheVersion is bumped whenever a question, rubric or threshold changes, so // unchanged copy is not served a verdict from the previous policy. -const cacheVersion = "copyjudge-v1" +const cacheVersion = "copyjudge-v2" // ContentHash identifies one piece of copy for caching. Trimmed, not // normalized further: a rewrite is a new judgment, a stray trailing newline is diff --git a/internal/app/copyjudge/judge_test.go b/internal/app/copyjudge/judge_test.go index 6caf13c58..37c3d7aeb 100644 --- a/internal/app/copyjudge/judge_test.go +++ b/internal/app/copyjudge/judge_test.go @@ -116,9 +116,12 @@ func TestReadsAsBulkNeedsConfidenceUnlessAClaimIsMade(t *testing.T) { v Verdict want bool }{ - {"confident bulk", Verdict{ReadsAs: 1, Confidence: 0.9}, true}, - {"at the threshold", Verdict{ReadsAs: BulkAt, Confidence: ConfFloor}, true}, - {"bulk but unsure", Verdict{ReadsAs: 1, Confidence: 0.5}, false}, + {"confident bulk and generic", Verdict{ReadsAs: 1, Personalization: 1, Confidence: 0.9}, true}, + {"at the threshold", Verdict{ReadsAs: BulkAt, Personalization: BulkAt, Confidence: ConfFloor}, true}, + {"promotional but relevant", Verdict{ReadsAs: 1, Personalization: 0, Confidence: 0.9}, false}, + {"generic but conversational", Verdict{ReadsAs: 0, Personalization: 1, Confidence: 0.9}, false}, + {"relevance borderline", Verdict{ReadsAs: 1, Personalization: BulkAt - 0.01, Confidence: 0.9}, false}, + {"bulk but unsure", Verdict{ReadsAs: 1, Personalization: 1, Confidence: 0.5}, false}, {"between", Verdict{ReadsAs: 0.5, Confidence: 0.95}, false}, {"personal with a spam claim", Verdict{ReadsAs: 0, Confidence: 0.2, SpamClaim: 0.8}, true}, {"spam claim below its line", Verdict{ReadsAs: 0, Confidence: 0.9, SpamClaim: 0.5}, false}, diff --git a/internal/repository/pg_advisor.go b/internal/repository/pg_advisor.go index 04f07e8a2..ffb4b8b7a 100644 --- a/internal/repository/pg_advisor.go +++ b/internal/repository/pg_advisor.go @@ -281,7 +281,7 @@ type AdvisorFindingFilter struct { Category models.AdvisorCategory EntityType string EntityID *uuid.UUID - // Statuses defaults to open + snoozed when empty. + // Statuses defaults to open when empty. Statuses []models.AdvisorStatus Limit int } @@ -497,7 +497,7 @@ func (r *advisorRepository) GetFinding(ctx context.Context, orgID, id uuid.UUID) func (r *advisorRepository) ListFindings(ctx context.Context, orgID uuid.UUID, f AdvisorFindingFilter) ([]models.AdvisorFinding, error) { statuses := f.Statuses if len(statuses) == 0 { - statuses = []models.AdvisorStatus{models.AdvisorStatusOpen, models.AdvisorStatusApplied} + statuses = []models.AdvisorStatus{models.AdvisorStatusOpen} } strs := make([]string, 0, len(statuses)) for _, s := range statuses { diff --git a/web/src/components/app/advisor/AdvisorCard.tsx b/web/src/components/app/advisor/AdvisorCard.tsx index 6e59a6547..7d43b79d0 100644 --- a/web/src/components/app/advisor/AdvisorCard.tsx +++ b/web/src/components/app/advisor/AdvisorCard.tsx @@ -53,9 +53,8 @@ interface Props { defaultOpen?: boolean; } -// Evidence keys that only exist to identify the subject, which the card header -// already shows. -const REDUNDANT_EVIDENCE = new Set(["mailbox", "campaign", "step"]); +// Subject identifiers and cache metadata do not help explain the finding. +const REDUNDANT_EVIDENCE = new Set(["mailbox", "campaign", "step", "content_hash"]); export default function AdvisorCard({ finding, onFix, compact = false, defaultOpen = false }: Props) { const [open, setOpen] = useState(defaultOpen); diff --git a/web/src/components/app/advisor/AdvisorFixDrawer.tsx b/web/src/components/app/advisor/AdvisorFixDrawer.tsx index c55724004..59a20545e 100644 --- a/web/src/components/app/advisor/AdvisorFixDrawer.tsx +++ b/web/src/components/app/advisor/AdvisorFixDrawer.tsx @@ -44,8 +44,8 @@ interface Props { onClose: () => void; } -// Evidence keys that only name the subject, which the header already shows. -const REDUNDANT_EVIDENCE = new Set(["mailbox", "campaign", "step"]); +// Subject identifiers and cache metadata do not help explain the finding. +const REDUNDANT_EVIDENCE = new Set(["mailbox", "campaign", "step", "content_hash"]); type Stage = "why" | "change" | "done"; diff --git a/web/src/components/app/advisor/AdvisorStrip.test.tsx b/web/src/components/app/advisor/AdvisorStrip.test.tsx new file mode 100644 index 000000000..8c25c33c5 --- /dev/null +++ b/web/src/components/app/advisor/AdvisorStrip.test.tsx @@ -0,0 +1,54 @@ +import type { ComponentProps, ReactNode } from "react"; +import { cleanup, fireEvent, render, screen } from "@testing-library/react"; +import { afterEach, expect, it, vi } from "vitest"; +import type { AdvisorFinding } from "@/lib/api/models/app/advisor/Advisor"; +import type AdvisorCard from "./AdvisorCard"; +import type AdvisorFixDrawer from "./AdvisorFixDrawer"; +import AdvisorStrip from "./AdvisorStrip"; + +const query = vi.hoisted((): { data: AdvisorFinding[] } => ({ data: [] })); + +vi.mock("@/lib/api/hooks/app/advisor/useAdvisor", () => ({ + SURFACE_FETCH_LIMIT: 200, + useAdvisorFindings: () => query, +})); + +vi.mock("./AdvisorCard", () => ({ + default: ({ finding, onFix }: ComponentProps) => ( + + ), +})); + +vi.mock("./AdvisorFixDrawer", () => ({ + default: ({ onClose }: ComponentProps) => ( +
+ +
+ ), +})); + +vi.mock("framer-motion", async (importOriginal) => ({ + ...(await importOriginal>()), + AnimatePresence: ({ children }: { children: ReactNode }) => <>{children}, +})); + +afterEach(cleanup); + +it("removes the last suggestion while keeping its fix result available", () => { + query.data = [{ + id: "finding", organization_id: "workspace", detector_key: "campaign_followup_spacing", + category: "campaign", severity: "low", surface: "campaigns", status: "open", impact: 15, + title: "Review follow-up spacing", detail: "Too soon", remedy: "Wait three days", + narrated: false, first_seen_at: new Date(), last_seen_at: new Date(), + }]; + const view = render(); + fireEvent.click(screen.getByRole("button", { name: "Review follow-up spacing" })); + + query.data = []; + view.rerender(); + + expect(screen.queryByRole("region", { name: "Advisor suggestions" })).toBeNull(); + expect(screen.getByRole("dialog", { name: "Fix result" })).toBeTruthy(); + fireEvent.click(screen.getByRole("button", { name: "Close result" })); + expect(screen.queryByRole("dialog")).toBeNull(); +}); diff --git a/web/src/components/app/advisor/AdvisorStrip.tsx b/web/src/components/app/advisor/AdvisorStrip.tsx index e6c86de20..49b1c3923 100644 --- a/web/src/components/app/advisor/AdvisorStrip.tsx +++ b/web/src/components/app/advisor/AdvisorStrip.tsx @@ -67,12 +67,6 @@ export default function AdvisorStrip({ const groups = useMemo(() => { const list = [...(data ?? [])].sort((a, b) => { - // Applied cards stay visible until the next evaluation confirms - // them, so "Fix" gives immediate, undoable feedback rather than a - // card that just disappears. They sort last. - if ((a.status === "applied") !== (b.status === "applied")) { - return a.status === "applied" ? 1 : -1; - } const rank = SEVERITY_RANK[b.severity] - SEVERITY_RANK[a.severity]; return rank !== 0 ? rank : b.impact - a.impact; }); @@ -84,12 +78,11 @@ export default function AdvisorStrip({ return groupFindings(list).slice(0, limit); }, [data, limit, entityId]); - if (groups.length === 0) return null; - const heading = title ?? "Suggestions"; return ( <> + {groups.length > 0 ? (
{heading ? (
@@ -164,6 +157,7 @@ export default function AdvisorStrip({ ) : null}
+ ) : null} {fixing ? (