Commit Graph
19 Commits
Author SHA1 Message Date
Wez Furlong 95d80d96d3 metrics: export per-thread-pool runtime metrics and cache populate duration
Per-thread-pool tokio runtime metrics are now exported for every named
runtime, not just the main runtime. An lruttl_populate_duration histogram
records cache populate time. An integration test verifies both metrics are
exported.
2026-10-21 18:00:06 +01:00
Wez Furlong 7c99973dfc http-inject: Add resizable concurrency limit with saturation metrics
Matches the existing max_connections behavior from SMTP and proxy listeners.
Operators can monitor and alert on endpoint saturation.

refs: https://github.com/KumoCorp/kumomta/issues/631
2026-10-14 14:01:32 +01:00
Wez Furlong 4170955400 mailexchanger: Distinguish MX lookup timeout causes in logs and metrics
Lets operators tell local concurrency permit wait timeouts apart from real
remote resolver query timeouts, to avoid misdiagnosing overload as remote
deliverability failure.

Adds metrics dns_mx_resolve_permit_timeout, dns_mx_resolve_query_timeout
and dns_mx_resolve_permit_waiters.
2026-10-08 14:58:52 +01:00
Wez Furlong 8faf16a8a7 listeners: don't die on transient accept errors
Prevent transient accept() errors from permanently stopping listeners
while the process stays alive and appears healthy. Trivial peer resets
that happen between kernel queue and accept are logged only at debug
level to close remote log spam vectors.
2026-10-06 15:24:59 +01:00
Wez Furlong 004b2aabd5 logging: expose log consumer backlog as metrics
Publish metrics showing how far each out-of-process log consumer trails
the newest data in the directories this node logs to, so that operators
can tell whether a consumer is keeping up. Covers both the
configure_local_logs and kumo.jsonl.new_writer directories.
2026-10-05 13:22:27 +01:00
Wez Furlong 41291a4afb docs: mark generated pages with an @generated marker
The documentation generators now stamp each page they emit with an
@generated comment so that tooling can tell generated pages apart from
hand-written ones. Update the table-of-contents generator to skip YAML
front matter when picking a page title, since the marker is now stored
there.
2026-10-05 13:22:26 +01:00
Wez Furlong b2e94ac370 docs: prep for release
```
./assets/find-since-dev.sh 2026.09.29-b90d8bc1
./docs/build.sh
```
2026-09-29 10:28:24 +01:00
Wez Furlong d9851e3e02 update dev -> 2026.09.22-a276d4a8
Update docs for new stable tag

```
./assets/find-since-dev.sh 2026.09.22-a276d4a8
./docs/build.sh
```
2026-09-23 15:26:24 +01:00
Wez Furlong 3a88e3c883 spool: fix rocksdb load-shedding gate never reopening after a single-signal error
The gate could latch permanently when only one of the two error signals
(foreground or background) ever moved, requiring an operator restart. It now
auto-reopens for a retry after `error_unlatch_duration`, regardless of which
signal latched it.

Also serializes foreground error reporting with gate transitions so a
concurrent fatal error cannot be undone by a reopen, and rejects a zero
`error_unlatch_duration` when automatic reopening is enabled. The underlying
cause is always logged. Adds a reusable LD_PRELOAD fault-injection crate and
integration tests that reproduce a full disk and a slow disk to drive the
latch-and-reopen cycle end to end.

Fixes #597
2026-09-21 20:16:32 +01:00
Wez Furlong a4c4b40fbe add dane_result_count metric
This surfaces aggregate dane results
2026-07-06 11:46:50 +01:00
Wez Furlong 423abc4704 add per-dispatcher state tracking and watchdog
The ready queue maintainer now tracks for each dispatcher whether it
is making progress.  If it exceeds dispatcher_progress_watchdog_timeout
then the task will be aborted and any message(s) it held onto will
be returned to the scheduled queue.  This will free up the connection
slot and allow another connection to be attempted, potentially
making further progress.

refs: https://github.com/KumoCorp/kumomta/issues/539
2026-06-23 09:56:45 +01:00
Wez Furlong 002d6593f0 new: suspend_when_unplumbed and suspend_when_proxy_unhealthy source options
Egress sources can now be configured to auto-suspend when their local
bind address appears unplumbed or when their configured proxy server
appears unreachable.  A suspended source is skipped during pool selection
until the configured duration elapses.
2026-06-23 09:22:27 +01:00
Wez Furlong 9987248313 spool: surface and handle rocksdb background errors
We recently investigated an issue where a rocksdb had been damaged by
corrupting/removing SST files (it sounded like this was accidentally
self-inflicted by some backup/orchestration infrastructure) leaving the
system in a silently-broken state: writes just wouldn't make progress
and there were no error messages.

Inspecting the `/var/spool/kumomta/data/LOG` log file (which is a
readable text file) revealed messages like:

```
2026/06/12-14:55:31.884227 2875746 [ERROR] [db/compaction/compaction.cc:262] Unable to load table properties for file 29704 --- IO error: No such file or directory: While open a file for random read: /var/spool/kumomta/data/029704.sst: No such file or directory
2026/06/12-14:55:31.884311 2875746 [ERROR] [db/db_impl/db_impl_compaction_flush.cc:3385] Waiting after background compaction error: IO error: No such file or directory: While open a file for random read: /var/spool/kumomta/data/029704.sst: No such file or directory, Accumulated background error counts: 6363
```

This commit improves the observability in this situation by proactively
checking for error conditions:

1. The store() and remove() operations now use our own polling within
   a deadline loop rather than spawning a blocking task and delegating
   to rocksdb's blocking interface.  This allows us to inspect the
   background error count and be cancellable, safely respecting and
   caller provided smtp max transaction duration.

2. All read and write operations check for IO and Corruption errors
   and immediately latch an error state

3. The metrics monitoring task inspects and track background error
   counts and latch us into an unhealthy state when the background
   error state appears unhealthy and persistent.

4. Additional metrics are exposed to help monitoring and alerting

While adding integration test coverage for this, I found a typo that
meant that spool errors were ignored in the message crate; they got
silently converted to `true` in all cases rather than just mapping
the success case to a `true`.

Integration tests handle the case where an SST file is corrupted
(truncated) during runtime, as well as starting up when an SST file
is missing.   These excercise both the foreground and background
error detection paths.
2026-06-23 09:11:48 +01:00
Wez Furlong 93e8e7dd01 docs: update since('dev') macros for stable release 2026-03-04 08:03:21 +00:00
Wez Furlong d8a7b0170e add new cpu usage metrics
refs: https://github.com/KumoCorp/kumomta/issues/186
2026-02-24 12:19:54 +00:00
Wez Furlong 80decbc08e redis: add redis_operation_latency histogram
This enables tracking the latency and status of redis operations.
2026-02-20 08:15:09 +00:00
Wez Furlong 37111c8036 add auto-gen docs for proxy-server http api and metrics
refs: https://github.com/KumoCorp/kumomta/pull/472
2026-02-06 07:11:56 +00:00
Wez Furlong 87b64c8d2b docs: flesh out docs for a number of metrics
Give some suggestions on how to interpret a number of the metrics,
as well as cross linking to relevant documentation for them.
2026-02-04 11:45:11 +00:00
Wez Furlong 66e5aa5508 docs: generate a page for each exported metric 2026-02-04 09:10:25 +00:00