warmbly/docs/content/docs/development/data-control.mdx

---
title: Data control
description: Where every store lives on disk, what each retention window governs, how backups work, and how to move an instance to another host.
---

Self-hosting Warmbly means holding mailbox credentials, message bodies and contact records on your own disk. This page is the whole answer to where that data is, how long it stays, and how you get it off this machine.

Three questions, and they have different answers:

| Question | Answer |
|---|---|
| Where is it? | [Six stores](#where-the-data-sits), each pointed at by one variable |
| How long does it stay? | [Retention windows](#what-is-kept-and-for-how-long), all editable in the admin panel |
| How do I move it? | [`warmblyctl backup`](#backups) for the instance, [workspace export](/guides/workspace-export-import/) for one workspace |

## Where the data sits

Every store's location is one variable in `.env`. Compose reads a source starting with `/` as a bind mount and anything else as a named volume, so the same variable covers both and there is no second compose file.

| Variable | Holds | Default |
|---|---|---|
| `WARMBLY_PG_DATA` | Postgres: organizations, users, mailboxes (credentials sealed), contacts, campaigns, the audit trail | `<data root>/postgres` |
| `WARMBLY_BLOBS` | Message bodies, attachments, avatars, logos and email body images | `<data root>/blobs` |
| `WARMBLY_NATS_DATA` | The event bus's JetStream state | `<data root>/nats` |
| `WARMBLY_REDIS_DATA` | Cache and rate-limit counters. Disposable | `<data root>/redis` |
| `WARMBLY_WORKER_STATE` | A worker's own id and sync cursors. Disposable | `<data root>/worker` |
| `WARMBLY_UPDATER_STATE` | The last update job's log. Disposable | `<data root>/updater` |

An [install from `install.sh`](/development/install/) points all six under one data root, so `/opt/warmbly/data` is the whole of it and you can `rsync` that path. A clone-and-build install keeps Docker named volumes unless you set the same variables.

Only the first two carry anything you cannot rebuild. The other four are state a fresh container reconstructs, which is why a backup does not include them.

### Encryption

Two keys, and they are not interchangeable.

| Key | Opens | Read by |
|---|---|---|
| `CREDENTIALS_ENCRYPTION_KEY` | Mailbox SMTP and IMAP credentials, and OAuth tokens | Backend and workers, without an organization context |
| `KMS_LOCAL_MASTER_KEY` | The per-organization data keys, which in turn open everything else | Backend and consumer |

Both are unrecoverable. A database backup without them restores an instance whose mailboxes authenticate against nothing, and there is no way back from that. They are in `.env`, and `install.sh` also writes them to `keys-backup.txt` next to the install, which is on the same disk as the database and therefore not a backup either. Copy them somewhere else.

### External stores

Postgres, Redis and blob storage each accept an external target, set at install time or by editing `.env`:

```bash
PRIMARY_DB=postgres://user:pass@db.internal:5432/warmbly?sslmode=require
REDIS=redis://cache.internal:6379
BLOB_PROVIDER=s3
BLOB_BUCKET=warmbly
AWS_ENDPOINT_URL_S3=https://<account>.r2.cloudflarestorage.com
```

<Callout type="warn" title="Filesystem blobs stop working when workers run off-host">
A remote worker writes bodies to its own disk, so the dashboard on the control-plane host finds nothing. Any deployment with workers on more than one machine needs S3-compatible blob storage. This is the one storage choice that is not just a preference.
</Callout>

## What is kept, and for how long

Every window below lives in the database, not the environment, and is edited under **Instance > Configuration > Settings** in the admin panel. A sweep runs a few times a day and reads the current value on every pass, so a change takes effect without a restart.

### Mailbox import and sync

| Setting | Default | Range | What it governs |
|---|---|---|---|
| `sync.backfill_days` | 90 | 1 to 730 | How far back the initial import reaches when a mailbox is connected, newest first |
| `sync.backfill_messages` | 5,000 | 1 to 100,000 | The most messages that import stores per mailbox |
| `sync.daily_messages_per_mailbox` | 2,000 | 1 to 100,000 | New mail one mailbox may store per UTC day |
| `sync.daily_messages_per_org` | 25,000 | 1 to 2,000,000 | New plus imported mail across one workspace per UTC day |

Mail over a daily budget is deferred, never dropped: the provider cursor is held and the mail is re-offered on the next pass. Replies to the mailbox's own outreach ride a separate budget of the same size and keep landing regardless.

### TypeSafe judgments

`TYPESAFE_API_KEY` turns on a family of typed judgments, each of which sends a bounded piece of text to TypeSafe and stores a verdict. Without the key none of them runs and no content leaves the instance. What each one sends:

| Judgment | On by | What is sent | Where the verdict lives |
|---|---|---|---|
| Automatic inbox tagging | `INBOX_TAGGING_ENABLED=true` | The inbound subject and bounded plain-text body, plus the previous outbound message in the thread when one exists | `inbox_tag_results`, with the raw probabilities and the actions taken. Completed results travel with the Inbox group in a workspace export; processing claims do not |
| Reply classification | The key alone | The subject and body of a reply to a campaign, only when no tagging verdict exists for it and the offline layers could not decide | `campaign_contact_progress.reply_class` and `reply_intent` |
| Copy judgment | The key alone | The subject and body of a campaign step, at Advisor evaluation and on the editor's re-check | `copy_judgments`, keyed by a hash of the copy and never exported |
| Warmup content lint | The key alone | A generated warmup thread before it enters the bank | Rejected threads are counted on the generation job; nothing else is kept |
| Bounce classification | The key alone | The bounce reason text, only when it does not name the recipient | `deliverability_events.metadata` |
| Form triage | A per-form switch | The submitted answers, labelled by field; never the visitor's address, agent or URL | `form_submissions.triage`, which travels with the Contacts group |

See [Automatic inbox tagging](/guides/inbox-tagging/), [Advisor](/guides/advisor/), [Warmup](/guides/warmup/), [Deliverability](/guides/deliverability/) and [Forms](/guides/forms/).

### Event history

| Setting | Default | What it holds |
|---|---|---|
| `retention.engagement_event_days` | 365 | Per-event open and click logs: client, device, approximate location |
| `retention.form_event_days` | 180 | Form funnel events: views, starts, field-level drop-off |
| `retention.audit_log_days` | 90 | The audit trail: actor, IP address, user agent, change payload |
| `retention.warmup_mail_days` | 30 | Warmup mail in the mailboxes themselves, and the stored copy of each message's body. Deleted from the warmup folder by the platform once older than this; a mailbox can set its own window |
| `retention.warmup_event_days` | 365 | Per-message warmup records: tokens, receipts, tampering events, spam reports. The daily sent and received counts behind the analytics are kept |

The first three are between 1 and 3,650 days, and they are the settings a retention or privacy policy applies to, because each window is also how long the personal data in that log is held. The warmup mail window starts at 3 days and the warmup records window at 30, the least the engagement legs and the pool health bands need.

None of them change a number anyone reads. Campaign progress keeps its own summary of opens and clicks that outlives the per-event log, so counts, filters and branching are unaffected by shortening any of these. What gets shorter is what a contact's timeline can show, how far a funnel report reaches, and how far back an admin can audit.

<Callout type="warn" title="Shortening a window deletes on the next sweep">
There is no grace period and no copy. Take a backup first if you are not sure.
</Callout>

The **minimal retention** preset in the admin panel and in the installer sets the three event logs to 30 days, warmup mail to 7 and warmup records to 30.

### Warmup mail in mailboxes

Warmup mail is real mail in a real mailbox, filed into the warmup folder in both directions, and it is the platform's job to clear it. Once a message is older than the mailbox's window it is deleted where it sits: moved to Trash on Gmail, which Gmail empties after 30 days, and removed outright on Outlook and IMAP. The platform's stored copy of the body goes with it. Nothing about a deletion the platform made is ever held against the mailbox, and a message deleted by its owner after the first day is treated the same way. See [Warmup](/guides/warmup/#retention) for what the mailbox owner sees.

### Warmup standing by address

Warmup keeps the standing of every penalised mailbox (score, state, and any block) against its address as well as its row, so removing the mailbox or dropping out of the pool and coming back does not start it clean. The copy is cleared as soon as the mailbox recovers, and once the mailbox is gone it is kept for 90 days after the block ends or after the removal, whichever is later; a block that requires review is kept until an operator lifts it. A mailbox in good standing has no such copy. The window is fixed rather than a setting, and the rows are purged by the same sweep.

### Pending inbox verification

Mail whose warmup verification is temporarily unavailable waits in the `unibox_pending_emails` Postgres table. It holds the sync event, including message metadata and bounded searchable body text, until verification succeeds or the mailbox is deleted. The consumer retries it automatically after restarts. Full bodies remain in blob storage. This pending queue is included in instance backups but excluded from workspace exports, whose destination re-syncs provider mail.

### Disconnecting a mailbox

Disconnecting a mailbox, or deleting the workspace holding it, removes its rows from Postgres straight away through the database's own cascades. Two things cannot be done in that instant and are queued instead, then worked off within about a minute:

| Owed | What happens |
|---|---|
| The OAuth grant | A Gmail mailbox's refresh token is sent to Google's revocation endpoint, which invalidates it and removes the instance from the customer's third-party access list. Microsoft publishes no per-application revocation endpoint, so an Outlook mailbox's tokens are destroyed locally and the customer removes the app themselves |
| The message bodies | Everything the mailbox wrote under `users/<user>/emails/<mailbox>/` in the blob store is deleted, whether that is a filesystem path or an S3 bucket |

While an erasure is outstanding its row holds a copy of the mailbox's OAuth refresh token, still sealed under `CREDENTIALS_ENCRYPTION_KEY`, because the row it was read from goes with the mailbox and the token is what the revocation sends. It is destroyed with the rest of the row when the erasure finishes.

The queue is the `mailbox_erasures` table, and it holds only work still owed: a finished erasure is removed rather than kept, because the row names the address the customer asked to have forgotten. A failure is retried with a widening backoff, capped at six hours, and never abandoned, so an unreachable provider or store delays the erasure instead of losing it. Anything still outstanding after a day is reported by the `mailbox_erasure` job, visible under scheduled jobs in the admin panel.

This runs on the control plane, which is the only place holding both the credential key and a credential for the blob store. A worker cannot do it: it signs one object at a time through the control plane and holds no bucket credential at all.

### Set them at install time

The installer writes the answers into `.env` as one document, applied on the first boot of a fresh database:

```bash
WARMBLY_SETTINGS_BOOTSTRAP={"sync":{"backfill_days":30},"retention":{"audit_log_days":30}}
```

It is read only while the settings row has never been written. From the first save in the admin panel onwards the panel is authoritative, so leaving the line in `.env` never undoes a later edit.

## Backups

`warmblyctl backup` writes one bundle holding the three things that only restore together:

- the database, as a `pg_dump`
- the blob root, when blobs are on the filesystem
- the encryption keys, unless you pass `--no-keys`

```bash
docker compose -p warmbly exec backend warmblyctl backup --out /data/blobs/warmbly.tar.gz
docker compose -p warmbly cp backend:/data/blobs/warmbly.tar.gz ./warmbly.tar.gz \
  && docker compose -p warmbly exec -T backend rm -f /data/blobs/warmbly.tar.gz
```

`/data/blobs` is a hand-off, not a destination: it is the one path the container and the host both see. The `&&` matters twice over. `backup` leaves its own output out of the archive, but a bundle left there is swept into the next run, so it has to be deleted; and a `cp` that failed must not be followed by deleting the only copy that exists.

The bundle is written 0600 and holds every mailbox credential on the instance plus the keys that open them. Treat the file as you would the instance itself.

An install from `install.sh --wizard` can schedule this for you: `backup.sh` next to the install, a systemd timer, a retention count, and an optional `aws s3 cp` to somewhere off the host. A backup that only exists on the machine it backs up is not one.

### Restore

On the destination host, with the same keys in place:

```bash
docker compose -p warmbly exec backend warmblyctl restore --file /data/blobs/warmbly.tar.gz
docker compose -p warmbly restart
```

The restore empties the schema and replays the dump, so it replaces everything currently on that instance and asks you to type `restore` first.

Before it does anything it compares the bundle's `CREDENTIALS_ENCRYPTION_KEY` and `KMS_LOCAL_MASTER_KEY` against the destination's and refuses to continue when they differ, printing the two lines to put in `.env`. That check is the point of the command: without it a restore looks like it worked and every mailbox fails to authenticate days later, with no error that names the cause.

## Moving an instance

Two ways, and they answer different questions.

### The whole instance, to a new host

Every workspace, every user, the platform admins, the API keys.

<Steps>

<Step>

### Install Warmbly on the new host

```bash
curl -fsSL https://warmbly.com/install.sh | sh -s -- --host <new-hostname>
```

</Step>

<Step>

### Put the old keys in the new `.env`

Copy `CREDENTIALS_ENCRYPTION_KEY` and `KMS_LOCAL_MASTER_KEY` from the old install, then recreate the containers so they take:

```bash
docker compose -p warmbly up -d
```

</Step>

<Step>

### Restore the bundle

```bash
docker compose -p warmbly exec backend warmblyctl restore --file /data/blobs/warmbly.tar.gz
docker compose -p warmbly restart
```

</Step>

<Step>

### Check it

```bash
docker compose -p warmbly exec backend warmblyctl status
```

Mailboxes should be connected, not needing a reconnect. If they need one, the keys did not match.

</Step>

</Steps>

The `rsync` alternative works too and is sometimes simpler, as long as the stack is **stopped** first: a running Postgres data directory copied file by file is not a consistent snapshot and can restore as a corrupt cluster. Stop it, copy the data root and the `.env` to the new host, start it there. It moves the same bytes; the bundle exists because it is the version that survives a different host layout, a different Postgres, a live instance, and a partial copy.

### One workspace, to another instance

[Workspace export and import](/guides/workspace-export-import/) moves a single organization's data between two running instances, re-sealing its secrets for the destination's keys. That is the per-customer tool; the bundle here is the instance-level one. They are not interchangeable: a bundle cannot be applied to one workspace, and a workspace archive cannot restore an instance.

## Outbound calls

A self-hosted Warmbly makes no outbound call of its own except one, and it is off with a single setting.

| Call | When | Turn it off |
|---|---|---|
| GitHub releases API | Every 30 minutes, to tell the admin panel a newer version exists | `UPDATE_CHECK_ENABLED=false` |

Everything else is you: mail through the mailboxes you connect, DNS lookups for the domains you check, and whatever integrations you configure. There is no telemetry, no phone-home, and no license check.

### Error reporting

An instance reports errors nowhere unless you point it somewhere. Every service reads its own credentials and none of them ship with any, so a default install sends no crash, no stack trace and no browser error to anybody, including us.

Two backends are supported and you choose either, both, or neither. PostHog is the default one; Sentry works exactly as it always has.

| Service | PostHog | Sentry |
|---|---|---|
| Backend, consumer, worker | `POSTHOG_KEY` | `SENTRY_DSN` |
| Forms service | `POSTHOG_KEY` | `SENTRY_DSN` |
| Tracking service | `POSTHOG_KEY` | `SENTRY_DSN` |
| Realtime service | `POSTHOG_KEY` | `SENTRY_DSN` |
| Dashboard and admin containers | `WARMBLY_POSTHOG_KEY` | `WARMBLY_SENTRY_DSN` |
| Public form pages | `WARMBLY_POSTHOG_KEY` on the forms service | `WARMBLY_SENTRY_DSN` on the forms service |

`POSTHOG_HOST` decides which PostHog receives them, including one you run yourself, and `POSTHOG_ERROR_TRACKING=false` turns error reporting off while leaving the key configured for the product analytics below. A Sentry DSN names whatever Sentry Cloud project, self-hosted Sentry or Sentry-compatible server you like.

Leave them unset, which is the default the installer writes, and nothing leaves the process: there is no host to contact and nothing to opt out of. Errors still reach that service's own log, as they always did. The dashboard, the admin panel and form pages go further and load no reporting code at all, so there is not even a script to block.

What a reported error carries is the exception, its stack trace, the service, the environment and the build, plus enough context to answer it:

- **Which workspace and user.** A browser exception carries the organization id and user id of whoever hit it, and a server-side one carries them for the request that panicked, along with its route and request id. That is what makes "this customer says campaigns are broken" a search rather than a guess. On the server side that is a property and nothing more: no person is identified. In the dashboard and the admin panel, with a browser key set, the signed-in user is identified to PostHog (a person profile with their email and name, and an identifier in local storage and a cookie), so the exception belongs to that person; see usage analytics below.
- **What led to it.** The last routes visited and the last API calls that failed, with their method, path, status and request id.
- **Nothing else about the request.** No headers, no bodies, no query strings. Session replay exists only in the dashboard and the admin panel, only with a browser key set, and `WARMBLY_POSTHOG_SESSION_REPLAY=false` turns it off; the marketing site and public form pages are never recorded.

Server-side reports that belong to no request, a scheduled job or a worker, are attributed to the process that raised them and to nobody.

The `warmbly` CLI runs on your own machine and reports nowhere, ever. It has nothing to set.

### Usage analytics

There are none. A self-hosted instance loads no analytics script and sends no usage data, not aggregated, not anonymised, not "to help us improve the product".

The hosted service at warmbly.com does measure its own marketing site, dashboard and admin panel, including session replay in the two apps, and the code for that ships in the same images you run. It is inert without a key: the installer never asks for one, the `.env` template does not mention it, and a build without one contains no analytics script for you to block. If you want to check, `grep posthog` the served assets of your own dashboard.

If you do want product analytics on your own instance, the variables are on [configuration](/development/configuration/#product-analytics-and-session-replay) and they accept a self-hosted PostHog. The server-side `POSTHOG_KEY` carries the backend's error tracking and its product events; the browser-side `WARMBLY_POSTHOG_KEY` is what turns on the dashboard's and the admin panel's analytics, session replay and browser errors, and `WARMBLY_POSTHOG_ERROR_TRACKING` and `WARMBLY_POSTHOG_SESSION_REPLAY` switch those last two off separately. That is your decision to make, and nothing about it points at us.

## See also

- [Install](/development/install/): the wizard that asks all of this up front
- [Configuration](/development/configuration/): every variable and every database-backed setting
- [warmblyctl](/development/warmblyctl/): `backup`, `restore`, and the operator commands
- [Workspace export and import](/guides/workspace-export-import/): the per-workspace story