mirror of https://github.com/mailscope/kumomta.git synced 2026-08-18 18:38:18 +00:00

Files

T

Wez Furlong 9987248313 spool: surface and handle rocksdb background errors

We recently investigated an issue where a rocksdb had been damaged by
corrupting/removing SST files (it sounded like this was accidentally
self-inflicted by some backup/orchestration infrastructure) leaving the
system in a silently-broken state: writes just wouldn't make progress
and there were no error messages.

Inspecting the `/var/spool/kumomta/data/LOG` log file (which is a
readable text file) revealed messages like:

```
2026/06/12-14:55:31.884227 2875746 [ERROR] [db/compaction/compaction.cc:262] Unable to load table properties for file 29704 --- IO error: No such file or directory: While open a file for random read: /var/spool/kumomta/data/029704.sst: No such file or directory
2026/06/12-14:55:31.884311 2875746 [ERROR] [db/db_impl/db_impl_compaction_flush.cc:3385] Waiting after background compaction error: IO error: No such file or directory: While open a file for random read: /var/spool/kumomta/data/029704.sst: No such file or directory, Accumulated background error counts: 6363
```

This commit improves the observability in this situation by proactively
checking for error conditions:

1. The store() and remove() operations now use our own polling within
   a deadline loop rather than spawning a blocking task and delegating
   to rocksdb's blocking interface.  This allows us to inspect the
   background error count and be cancellable, safely respecting and
   caller provided smtp max transaction duration.

2. All read and write operations check for IO and Corruption errors
   and immediately latch an error state

3. The metrics monitoring task inspects and track background error
   counts and latch us into an unhealthy state when the background
   error state appears unhealthy and persistent.

4. Additional metrics are exposed to help monitoring and alerting

While adding integration test coverage for this, I found a typo that
meant that spool errors were ignored in the message crate; they got
silently converted to `true` in all cases rather than just mapping
the success case to a `true`.

Integration tests handle the case where an SST file is corrupted
(truncated) during runtime, as well as starting up when an SST file
is missing.   These excercise both the foreground and background
error detection paths.

2026-06-23 09:11:48 +01:00

658 B

Raw Permalink Blame History

kcli spool-compact

Forces a flush and full compaction of the named spool.

Primarily a diagnostic and test helper. For rocksdb-backed spools this calls flush() followed by a full-keyspace compact_range(). For other spool kinds it is a no-op.

If the underlying storage reports an error during the operation (for example, a missing or corrupt SST file in a rocksdb spool), the error is reported to the caller and the command exits non-zero.

Usage: kcli spool-compact --name <NAME>

Options

--name <NAME> — Name of the spool to compact, matching a name passed to kumo.define_spool in the policy

658 B Raw Permalink Blame History

kcli spool-compact

Options

658 B

Raw Permalink Blame History