Commit Graph
618 Commits
Author SHA1 Message Date
Wez Furlong b1e1262e2b spool: deprecate kind = LocalDisk
Nobody runs it and it is not as good as RocksDB.  I want to remove it
in the future so that users don't unwittingly deploy a poor configuration.

We now log errors on startup to help surface configuration that will
need attention, but we don't block startup.
2026-10-09 17:47:51 +01:00
Wez Furlong 04d7acd892 kumo.dns.configure_unbound_resolver: deprecated!
Since hickory can now be used to enable DANE, the only remaining
use case for the unbound embedded resolver is disolved.

I want to remove it from the tree because:

 * I don't want to own the rust crate wrapping libunbound
 * Reduces complexity
 * Users have reported issues with unbound that simply disappear
   by switching back to hickory
 * Essentially nobody is running kumomta with unbound enabled,
   so it is has no real production coverage: you're a snowflake
   if you're running it

Removing it will reduce the potential for a user unwittingly enabling
something that isn't known to work as well as the golden path.

The deprecation manifests as an error printed on startup when
enabling unbound, and adjustments in the docs.

It will continue to work as it does today.
2026-10-09 17:47:42 +01:00
Wez Furlong c80cb8fabe memoize: prevent cancelled callers from aborting shared cache population
Cache population now runs detached by default for top-level memoized
functions. Other callers waiting on the same key now receive the result
even if the original triggering caller is cancelled mid-populate;
previously that case could impact the other yet-to-be-timed-out
waiters and cause cache population to restart or otherwise fail.

refs: #602
2026-10-09 10:35:44 +01:00
Wez Furlong 4170955400 mailexchanger: Distinguish MX lookup timeout causes in logs and metrics
Lets operators tell local concurrency permit wait timeouts apart from real
remote resolver query timeouts, to avoid misdiagnosing overload as remote
deliverability failure.

Adds metrics dns_mx_resolve_permit_timeout, dns_mx_resolve_query_timeout
and dns_mx_resolve_permit_waiters.
2026-10-08 14:58:52 +01:00
Wez Furlong 3e1932a86a dns: avoid long caching of transient MX lookup failures
Indeterminate MX lookup failures now use a short 30s transient negative TTL,
while NXDOMAIN and policy rejections retain the 5 minute TTL.  Lookups blocked
by the concurrency limit are not cached at all.  You can configure the transient
TTL with kumo.dns.set_mx_transient_negative_cache_ttl.

This prevents transient failures from causing domains to drop out of shaping
rollups and incorrectly flipping the shaping hash.
2026-10-08 14:27:34 +01:00
Wez Furlong 9448f92e11 log_hooks: Add log_parameters pass-through to new_disposition_hook
You can now set per_record and other log options directly via the
new_disposition_hook helper.

Closes: #499
Closes: #500
Closes: #519
Closes: #520
2026-10-08 06:47:54 +01:00
Wez Furlong d64eaf9ba6 docs: add kumo.configure_log_disposition_hook
closes: https://github.com/KumoCorp/kumomta/pull/520
closes: https://github.com/KumoCorp/kumomta/issues/499
2026-10-08 06:47:54 +01:00
Michael Renner 907abb52e3 mx: clarify site-name factoring and upgrade behavior
Explain the factoring phases and input edge cases, and make the DNS
fixture and session-reuse timeout intent easier to read.

Move site-name guidance to the queues reference with an inline release
marker, and link to it from lookup_mx and the traffic-shaping guide.
Add a breaking-change note with an observed collision pair.
2026-10-07 14:05:32 +01:00
Michael Renner 746e89a735 mx: make site-name factoring lossless and canonical
Normalize, deduplicate and sort MX destinations before deriving site
names. Use per-label alternatives only when their expansion matches
the destination set; otherwise factor complete hostname branches.

Keep MX preferences in connection plans rather than site identity.
Keep asterisks literal and percent-encode other special label bytes
so names remain distinct without introducing metric-label escapes.

Cover collisions, normalization, ports, shaping and queue/session
sharing, and document the naming and preference semantics.
2026-10-07 11:04:46 +01:00
Wez Furlong 238bf2bc4b smtp: bound maximum size of SMTP connection plans
closes: https://github.com/KumoCorp/kumomta/issues/622
2026-10-07 10:51:05 +01:00
Wez Furlong 7c569bbf8d mail_auth: limit PTR names followed during iprev checks
Cap iprev at 10 PTR names for forward lookups, as required by RFC 8601.
Prevents malicious reverse DNS zones from causing unbounded DNS lookups.

closes: https://github.com/KumoCorp/kumomta/issues/623
2026-10-07 09:05:32 +01:00
Wez Furlong 9273ec1351 kumo-log-types: generate a valid bounce when the original message has 8-bit content
A message with 8-bit headers or body could not be bounced at all: building the
returned-content part failed and aborted the whole report. It now produces a
7-bit clean report, returning as much of the original as can be represented and
omitting the rest.
2026-10-07 08:19:38 +01:00
Wez Furlong 8faf16a8a7 listeners: don't die on transient accept errors
Prevent transient accept() errors from permanently stopping listeners
while the process stays alive and appears healthy. Trivial peer resets
that happen between kernel queue and accept are logged only at debug
level to close remote log spam vectors.
2026-10-06 15:24:59 +01:00
Wez Furlong 004b2aabd5 logging: expose log consumer backlog as metrics
Publish metrics showing how far each out-of-process log consumer trails
the newest data in the directories this node logs to, so that operators
can tell whether a consumer is keeping up. Covers both the
configure_local_logs and kumo.jsonl.new_writer directories.
2026-10-05 13:22:27 +01:00
Wez Furlong 41291a4afb docs: mark generated pages with an @generated marker
The documentation generators now stamp each page they emit with an
@generated comment so that tooling can tell generated pages apart from
hand-written ones. Update the table-of-contents generator to skip YAML
front matter when picking a page title, since the marker is now stored
there.
2026-10-05 13:22:26 +01:00
Wez Furlong b2e94ac370 docs: prep for release
```
./assets/find-since-dev.sh 2026.09.29-b90d8bc1
./docs/build.sh
```
2026-09-29 10:28:24 +01:00
Wez Furlong b90d8bc1f3 mailparsing: Handle invalid multipart boundaries gracefully
The prior commit focused on avoiding a panic when we encountered an invalid
boundary string.  This commit makes thing degrade more gracefully: we'll
treat the bad MIME part in the same way that we deal with overly deep nesting
and handle it as an opqaque leaf part.

The new_multipart constructor now rejects an invalid boundary rather than
building an unserializable part, so a flawed policy can't accidentally create
broken messages.
2026-09-29 10:22:18 +01:00
Wez Furlong c1daddcbe3 mailparsing: bound MIME nesting depth to prevent stack overflow
Deeply nested multipart content is now retained as an opaque part with
its headers parsed and raw body preserved once nesting passes 100
levels, instead of recursing until the process stack overflows.

Those messages are tagged with the new MIME_NESTING_LIMIT_EXCEEDED
conformance flag.
2026-09-28 14:04:31 +01:00
Wez Furlong d2f31b4fdf mailparsing: emit MIME-Version with its RFC 2045 canonical spelling
Headers were built and rebuilt as "Mime-Version" rather than
"MIME-Version". Both are valid per RFC 2045, but some spam filters,
such as rspamd, score the mixed-case form as a deliverability signal.
Fix the header name in the accessor macro and in ParsedHeader's
grammar table, the places that spell it, and update the snapshot
tests across mailparsing, message, kumo-log-types, mod-mimepart, and
kumod that captured the old spelling, along with the reference docs
that showed the old casing in their examples. #564
2026-09-28 13:21:16 +01:00
Wez Furlong bb74f3857c kumo-jsonl: skip only oversized records, not whole segments
When a record exceeds the configured max_line_size, discard just that record
and continue reading the rest of the segment instead of aborting the entire
segment.
2026-09-28 09:57:52 +01:00
Wez Furlong d56fd68ef9 kumo-jsonl: Fix silent data loss for records larger than 128KiB
The zstd decompression buffer now grows on demand to read records up to
a configurable 128 MiB cap.  Oversized records are rejected explicitly
both on read and write, with matching limits so all written records are
guaranteed readable.
2026-09-28 09:57:52 +01:00
Wez Furlong d9851e3e02 update dev -> 2026.09.22-a276d4a8
Update docs for new stable tag

```
./assets/find-since-dev.sh 2026.09.22-a276d4a8
./docs/build.sh
```
2026-09-23 15:26:24 +01:00
Wez Furlong 3a88e3c883 spool: fix rocksdb load-shedding gate never reopening after a single-signal error
The gate could latch permanently when only one of the two error signals
(foreground or background) ever moved, requiring an operator restart. It now
auto-reopens for a retry after `error_unlatch_duration`, regardless of which
signal latched it.

Also serializes foreground error reporting with gate transitions so a
concurrent fatal error cannot be undone by a reopen, and rejects a zero
`error_unlatch_duration` when automatic reopening is enabled. The underlying
cause is always logged. Adds a reusable LD_PRELOAD fault-injection crate and
integration tests that reproduce a full disk and a slow disk to drive the
latch-and-reopen cycle end to end.

Fixes #597
2026-09-21 20:16:32 +01:00
Wez Furlong 7d6e6d558d Tidy up Option<PolicyMode> -> PolicyMode
The latter already has a None so it felt cumbersome to layer it up in an
Option.
2026-07-27 08:02:38 +01:00
Wez Furlong 9efc2a031c allow mocking mta-sts end-to-end, add integration test coverage
This sets up plumbing to allow testing the broken mta-sts aliasing
issue, and enables feeding an optional resolver through the mx lookups
as well.

closes: https://github.com/KumoCorp/kumomta/pull/524
2026-07-27 08:02:37 +01:00
Wez Furlong 41be60dfce Evaluate MTA-STS during MX resolution to fix aliasing issue
refs: https://github.com/KumoCorp/kumomta/issues/484

Briefly, the issue is that if some random domain that shares MX records
with another (eg: someone is using google apps or icloud for their
vanity domain) publishes a broken MTA-STS policy that requires eg:
cloudflare MX hosts then because we roll up by site name, that broken
MTA-STS policy bleeds into all the other domains that share those MX
records.

The resolution is simple, but is technically a breaking change.

Moving the policy resolution to happen during site_name resolution
allows us to resolve both per-domain things at the same and have the
MTA-STS policy amend the effective set of MX hosts.  The output of that
is then used for site_name aggregation/rollup.

The consequence of this is quite nice: an MTA-STS policy that is more
restrictive than the full set of MX hosts now prevents delivering to
any of the excluded hosts, and a totally broken policy that prevents all
of its MX hosts is now completely undeliverable and will produce
transient failures.

The downside is that for users that had previously disabled mta-sts in
their default shaping block, they will need to change a different config
option to continue to prevent MTA-STS from being consulted.  One example
of this that I recall is that one user's network posture prevented
MTA-STS from making HTTPS requests to fetch the policy.  Another user
just wanted to cut out the additional DNS traffic.  Those use cases
require altering the new kumo.dns.set_mta_sts_enabled enabled to false
during `init`.
2026-07-27 08:02:37 +01:00
Daniel Schaaff c17c123402 improve cgroup memory usage calculation to avoid counting purgeable cache
When running under a cgroup limit, get_cgroup() reported
stat.usage_in_bytes (memory.current on v2, memory.usage_in_bytes on v1)
as the process memory usage. That counter includes all page
cache, including cold reclaimable cache (inactive_file). For a
workload that writes logs and spool to disk, cold cache can dominate:
on a real pod memory.current was ~51GB while inactive_file was ~46GB
and the true non-cache footprint was ~1.8GB. kumod saw usage near its
limit, hit get_headroom() == 0, and ran
shrink_ready_queue_due_to_low_mem
and other reductions against pressure that was almost entirely
reclaimable cache the kernel would drop before any OOM.

Report the working set instead:
```
working_set = max(memory.current - inactive_file, anon)
```
This matches container_memory_working_set_bytes (kubelet/cAdvisor) and
the kernel's own reclaimability accounting. It still counts everything
that can drive an OOM: anonymous memory, dirty page cache, active_file,
tmpfs/shm, and slab.

Signed-off-by: Daniel Schaaff <daniel@danielschaaff.com>
closes: https://github.com/KumoCorp/kumomta/pull/549
2026-07-07 10:02:02 +01:00
Wez Furlong dfb17abd0d dane: engage for secure CNAME into unsigned zone
Per RFC 7672 section 2.2.2, an MX host that is a securely published
CNAME remains DANE-eligible at its original name even when the alias
target lands in an unsigned zone: it is the secure TLSA RRset, not the
address records, that authenticates the peer. When the address chain is
insecure but MX selection was secure, an explicit CNAME query isolates
the alias's own DNSSEC status; a secure alias engages DANE, an
indeterminate status defers for downgrade resistance.

refs: https://github.com/KumoCorp/kumomta/pull/545#discussion_r3472353021
2026-07-06 11:46:51 +01:00
Wez Furlong 4fc3cbca92 add treat_mx_list_as_secure to SmtpProtocol
This allows you to explicit indicate that an mx_list should
be treated as DNSSEC secure.

We cannot simply assume that lua code setting mx_list be treated
as secure, because the lua code may be simply passing through
the result of some other DNS lookup--we need an explicit way
to thread through the overall secure flag in order for the
secure property to be correctly upheld.
2026-07-06 11:46:51 +01:00
Wez Furlong e13734d186 address review feedback on https://github.com/KumoCorp/kumomta/pull/545
Address most of the easy points from the review, others will land
separately.
2026-07-06 11:46:51 +01:00
Wez Furlong 3b61117145 dns: add RFC 5011 managed trust anchor option for unbound
trust_anchor_file now accepts `{ managed = "<path>" }` for an RFC 5011
auto-maintained anchor file (unbound only; hickory errors). Unbound seeds
the file from the bundled root anchors when the file is absent.
2026-07-06 11:46:51 +01:00
Wez Furlong 1874dc09f7 enable DNSSEC when using hickory, allows enable_dane w/o unbound
When we first built out DANE, the hickory implementation was
all or nothing when DNSSEC validation was enabled, making it unsuitable
for use in an MTA.

Since that time hickory has matured; this commit wires up the
DNSSEC signals and that enables its use with DANE.

We do still require openssl for establishing the connection when
DANE is enabled; that's a different constraint.
2026-07-06 11:46:50 +01:00
Wez Furlong 7331cb4ec7 new allow_smtp_auth_plain_without_valid_certificate option
This is technically a breaking change, but in a good way.
We will no longer send SMTP AUTH PLAIN creds when we didn't
validate the peer certificate.  This is what you want in
all production deployments.  The new option facilitates
test setups or other unusual configurations where the certs
cannot be verified but you have contrived to trust the
remote system through some other means.
2026-07-06 11:46:50 +01:00
Wez Furlong a4c4b40fbe add dane_result_count metric
This surfaces aggregate dane results
2026-07-06 11:46:50 +01:00
Wez Furlong d920ffb5c4 fix: DANE downgrade resistance
We weren't checking the DNSSEC status for A/AAAA when considering
DANE, and were overly permissive in a couple of related cases.

This diff ended up fairly large because the bulk of it is to
facilitate testing.

refs: https://github.com/KumoCorp/kumomta/issues/543
refs: https://github.com/KumoCorp/kumomta/pull/517
2026-07-06 11:46:49 +01:00
Wez Furlong 3f11963acd docs: fix broken link 2026-06-27 18:58:12 +01:00
Wez Furlong 407379dd36 track sched->readyq fan in, clean up some internals
This commit removes the queue_name_for_config_change_purposes_only wart
from inside the readyq machinery.  This field had a bunch of caveats
and could in some cases become stale if the originating scheduled
queue had aged out.

It's been replaced by a HashSet of Weak refs back to the associated
scheduled queues so that it is now possible to walk the reverse of
the fan-in graph when we need to.
2026-06-25 13:41:01 +01:00
Wez Furlong cf5db1a796 fixup doc build 2026-06-23 15:54:06 +01:00
Wez Furlong 4d576922f8 deps: hickory 0.25 -> 0.26
This is fairly big diff because a number of hickory types moved around
or changed signature.

In particular, the config structs changed in a way that would fan out
and require attention by our users.  So the approach I opted to take
here was to define our own types that look like the hickory 0.25
shape and then we have explicit logic to map those to hickory and
also to our unbound resolver crate.

This commit also bumps up the unbound resolver crate because it also
uses hickory's types.  I took the opportunity to upgrade the version
of the embedded unbound resolver as part of that work.
2026-06-23 15:47:08 +01:00
Wez Furlong a1a7e16759 docs: fixup links 2026-06-23 10:13:11 +01:00
Wez Furlong 2a657f68fb included scheduled queue config in resolve-egress-path output
and also in resolve-shaping-domain, for consistency.

We can use the message rate from the scheduled queue config to refine
the computed ceilings/constraints too, which is nice.
2026-06-23 10:00:07 +01:00
Wez Furlong 49e1a62733 add resolve-egress-path API endpoint
This is conceptually equivalent to resolve-shaping-domain but operates
on a live instance, rather than loading up the policy offline.
2026-06-23 10:00:05 +01:00
Wez Furlong 61bb95a558 resolve-shaping-domain: expand output
This changes the output to be a bit more human friendly by default
by rendering as toml.

It also includes the rate ceilings added in the prior commit.
2026-06-23 09:59:32 +01:00
Wez Furlong 0a72ce7f02 introduce EgressPathConfigConstraints concept
This can be computed for an EgressPathConfig and is useful to indicate
what the effective ceilings are for this egress path.

We return this from inspect-ready-q
2026-06-23 09:58:38 +01:00
Wez Furlong b9e677dd16 add abort-ready-q-conn and inspect-readyq endpoints/commands
These build on the infrastructure from the prior commit and improve
the observability of the system.

refs: https://github.com/KumoCorp/kumomta/issues/539
2026-06-23 09:58:36 +01:00
Wez Furlong 423abc4704 add per-dispatcher state tracking and watchdog
The ready queue maintainer now tracks for each dispatcher whether it
is making progress.  If it exceeds dispatcher_progress_watchdog_timeout
then the task will be aborted and any message(s) it held onto will
be returned to the scheduled queue.  This will free up the connection
slot and allow another connection to be attempted, potentially
making further progress.

refs: https://github.com/KumoCorp/kumomta/issues/539
2026-06-23 09:56:45 +01:00
Wez Furlong 002d6593f0 new: suspend_when_unplumbed and suspend_when_proxy_unhealthy source options
Egress sources can now be configured to auto-suspend when their local
bind address appears unplumbed or when their configured proxy server
appears unreachable.  A suspended source is skipped during pool selection
until the configured duration elapses.
2026-06-23 09:22:27 +01:00
Wez Furlong e21bb1c672 ha_proxy_server and socks5_proxy_server now also resolve host names
We'll expand these to a list of candidate addresses, trying each in
turn.  We use the same overall connect timeout regardless of how
many candidates are present, to respect that timeout bound.

closes: https://github.com/KumoCorp/kumomta/pull/402
2026-06-23 09:11:53 +01:00
Wez Furlong 9987248313 spool: surface and handle rocksdb background errors
We recently investigated an issue where a rocksdb had been damaged by
corrupting/removing SST files (it sounded like this was accidentally
self-inflicted by some backup/orchestration infrastructure) leaving the
system in a silently-broken state: writes just wouldn't make progress
and there were no error messages.

Inspecting the `/var/spool/kumomta/data/LOG` log file (which is a
readable text file) revealed messages like:

```
2026/06/12-14:55:31.884227 2875746 [ERROR] [db/compaction/compaction.cc:262] Unable to load table properties for file 29704 --- IO error: No such file or directory: While open a file for random read: /var/spool/kumomta/data/029704.sst: No such file or directory
2026/06/12-14:55:31.884311 2875746 [ERROR] [db/db_impl/db_impl_compaction_flush.cc:3385] Waiting after background compaction error: IO error: No such file or directory: While open a file for random read: /var/spool/kumomta/data/029704.sst: No such file or directory, Accumulated background error counts: 6363
```

This commit improves the observability in this situation by proactively
checking for error conditions:

1. The store() and remove() operations now use our own polling within
   a deadline loop rather than spawning a blocking task and delegating
   to rocksdb's blocking interface.  This allows us to inspect the
   background error count and be cancellable, safely respecting and
   caller provided smtp max transaction duration.

2. All read and write operations check for IO and Corruption errors
   and immediately latch an error state

3. The metrics monitoring task inspects and track background error
   counts and latch us into an unhealthy state when the background
   error state appears unhealthy and persistent.

4. Additional metrics are exposed to help monitoring and alerting

While adding integration test coverage for this, I found a typo that
meant that spool errors were ignored in the message crate; they got
silently converted to `true` in all cases rather than just mapping
the success case to a `true`.

Integration tests handle the case where an SST file is corrupted
(truncated) during runtime, as well as starting up when an SST file
is missing.   These excercise both the foreground and background
error detection paths.
2026-06-23 09:11:48 +01:00
Wez Furlong d88fee2e0d docs: manually run cargo run -p jsonschematodocs
This somehow got missed in an earlier commit
2026-06-17 22:52:45 +01:00