Wez Furlong
9987248313
We recently investigated an issue where a rocksdb had been damaged by corrupting/removing SST files (it sounded like this was accidentally self-inflicted by some backup/orchestration infrastructure) leaving the system in a silently-broken state: writes just wouldn't make progress and there were no error messages. Inspecting the `/var/spool/kumomta/data/LOG` log file (which is a readable text file) revealed messages like: ``` 2026/06/12-14:55:31.884227 2875746 [ERROR] [db/compaction/compaction.cc:262] Unable to load table properties for file 29704 --- IO error: No such file or directory: While open a file for random read: /var/spool/kumomta/data/029704.sst: No such file or directory 2026/06/12-14:55:31.884311 2875746 [ERROR] [db/db_impl/db_impl_compaction_flush.cc:3385] Waiting after background compaction error: IO error: No such file or directory: While open a file for random read: /var/spool/kumomta/data/029704.sst: No such file or directory, Accumulated background error counts: 6363 ``` This commit improves the observability in this situation by proactively checking for error conditions: 1. The store() and remove() operations now use our own polling within a deadline loop rather than spawning a blocking task and delegating to rocksdb's blocking interface. This allows us to inspect the background error count and be cancellable, safely respecting and caller provided smtp max transaction duration. 2. All read and write operations check for IO and Corruption errors and immediately latch an error state 3. The metrics monitoring task inspects and track background error counts and latch us into an unhealthy state when the background error state appears unhealthy and persistent. 4. Additional metrics are exposed to help monitoring and alerting While adding integration test coverage for this, I found a typo that meant that spool errors were ignored in the message crate; they got silently converted to `true` in all cases rather than just mapping the success case to a `true`. Integration tests handle the case where an SST file is corrupted (truncated) during runtime, as well as starting up when an SST file is missing. These excercise both the foreground and background error detection paths.
2.6 KiB
Command-Line Help for kcli
This document contains the help content for the kcli command-line program.
kcli
KumoMTA CLI.
Interacts with a KumoMTA instance via its HTTP API endpoint. To use it, you must be running an HTTP listener.
The default is to assume that KumoMTA is running a listener at http://127.0.0.1:8000 (which is in the default configuration), but otherwise you can override this either via the --endpoint parameter or KUMO_KCLI_ENDPOINT environment variable.
Full docs available at: https://docs.kumomta.com
Usage: kcli [OPTIONS] <COMMAND>
Subcommands:
-
bounce— Administratively bounce messages in matching queues -
bounce-list— Returns list of current administrative bounce rules -
bounce-cancel— Cancels an admin bounce entry -
rebind— Rebind messages from matching queues into different queue(s) -
spool-compact— Forces a flush and full compaction of the named spool -
suspend— Administratively suspend messages in matching queues -
suspend-list— Returns list of current administrative suspend rules -
suspend-cancel— Cancels an admin suspend entry -
suspend-ready-q— Administratively suspend the ready queue for an egress path -
suspend-ready-q-list— Returns list of current ready queue/egress path suspend rules -
suspend-ready-q-cancel— Cancels an admin suspend entry for a ready queue/egress path -
set-log-filter— Changes the diagnostic log filter -
inspect-message— Returns information about a message in the spool -
inspect-sched-q— Returns information about a scheduled queue -
provider-summary— Prints a summary of the aggregate state of the queues from the perspective of the provider or destination site -
queue-summary— Prints a summary of the state of the queues, for a human to read -
trace-smtp-client— Trace outgoing sessions made by the SMTP service -
trace-smtp-server— Trace incoming connections made to the SMTP service -
top— Continually update and show what's happening in kumod -
xfer— Transfer messages from matching queues to an alternative kumomta node -
xfer-cancel— Cancels a message transfer that was initiated via the xfer subcommand. You specify the name of the xfer queue associated with the transfer and matching messages will be taken out of that queue and returned to their originating queue
Options
--endpoint <ENDPOINT>— URL to reach the KumoMTA HTTP API. You may set KUMO_KCLI_ENDPOINT in the environment to specify this without explicitly using --endpoint. If not specified, http://127.0.0.1:8000 will be assumed