Base64-encoded supplemental trace headers (X-KumoRef) can exceed the
998-octet SMTP line length limit when they include sizeable metadata,
causing a strict receiver (eg: us, when an ARF comes back to us with
that header) to reject the message at DATA with "line too long".
Fold the encoded value across continuation lines so each physical line
stays within the limit, and strip the folding whitespace before decoding
it back into a feedback report.
Running down a test flake and diagnosed this one.
This commit closes a narrow race when attaching to the SMTP tracing
(trace-smtp-server, trace-smtp-client) and TSA subscription
(subscribe_suspension_v1, subscribe_event_v1) websocket endpoints. The
server did not finish registering the new subscriber until just after
the connection handshake completed, so any event produced in the brief
window between those two points was not delivered to that client.
In practice this could cause a freshly-attached SMTP trace to miss the
first event or two of a session that happened to start at the same
instant; it did not affect mail flow. The endpoints now register the
subscriber before completing the handshake
RFC 2822 date parsing now tolerates an obsolete alphabetic time zone
such as the `UTC` that Amazon SES emits in its bounce reports, which
strict parsing would otherwise reject. A recognized abbreviation
resolves to its offset (derived from the IANA time zone database), and
any other alphabetic zone falls back to the `-0000` unknown offset per
RFC 5322 section 4.3 rather than failing the parse.
closes: https://github.com/KumoCorp/kumomta/pull/551
While testing message parsing code, there's a chance that the sender
isn't fully compliant with base64. This leads to failure such as below
base64 decode: non-zero trailing bits at 20211 b='i' in HRtbD4NCi==
As long as it's not destructive, its more convenient to be able to
support non-RFC messages so we can extract message artifacts for its
decision making.
closes: https://github.com/KumoCorp/kumomta/pull/558
RFC 5321 does not permit a space between the colon and the reverse/forward
path in MAIL FROM: and RCPT TO:, but a number of legacy clients emit one
(e.g. "MAIL FROM: <addr>"), which kumod rejected with 501 5.1.7.
Relax the grammar to accept and discard an optional run of spaces/tabs after
the colon in both the success and "valid-address + trailing junk" arms for
MAIL FROM and RCPT TO. This is a grammar-level change per review feedback,
replacing the earlier opt-in allow_space_before_path listener option.
Adds parser tests covering the tolerated space for both verbs, including the
null sender, postmaster, and ESMTP parameter cases.
closes: https://github.com/KumoCorp/kumomta/pull/559
This sets up plumbing to allow testing the broken mta-sts aliasing
issue, and enables feeding an optional resolver through the mx lookups
as well.
closes: https://github.com/KumoCorp/kumomta/pull/524
refs: https://github.com/KumoCorp/kumomta/issues/484
Briefly, the issue is that if some random domain that shares MX records
with another (eg: someone is using google apps or icloud for their
vanity domain) publishes a broken MTA-STS policy that requires eg:
cloudflare MX hosts then because we roll up by site name, that broken
MTA-STS policy bleeds into all the other domains that share those MX
records.
The resolution is simple, but is technically a breaking change.
Moving the policy resolution to happen during site_name resolution
allows us to resolve both per-domain things at the same and have the
MTA-STS policy amend the effective set of MX hosts. The output of that
is then used for site_name aggregation/rollup.
The consequence of this is quite nice: an MTA-STS policy that is more
restrictive than the full set of MX hosts now prevents delivering to
any of the excluded hosts, and a totally broken policy that prevents all
of its MX hosts is now completely undeliverable and will produce
transient failures.
The downside is that for users that had previously disabled mta-sts in
their default shaping block, they will need to change a different config
option to continue to prevent MTA-STS from being consulted. One example
of this that I recall is that one user's network posture prevented
MTA-STS from making HTTPS requests to fetch the policy. Another user
just wanted to cut out the additional DNS traffic. Those use cases
require altering the new kumo.dns.set_mta_sts_enabled enabled to false
during `init`.
We spent some time running down an issue where the combination
of the root-user umask and some ad-hoc testing to recreate the
spool directory led to a very difficult to understand situation
where the spool seemed to corrupt itself on the next restart.
The root cause was that the spool directories had been re-created
by the operator with permissions that led to an asymmetric
filesystem view for the dropped privs scenario (eg: the default
when running via systemd).
The crux of it is that the explicit existence check used by rocksdb to
determine if the database was new or pre-existing was inconsistent with
what the kernel actually allowed for the permissions on the directory,
causing the second start to believe that it was a new database, then
surprise itself when it found other evidence that it wasn't new at all.
This commit adds a dir-probe crate that tests for this misconfiguration
and raises an error before rocksdb can try to open and corrupt itself.
We can't reasonably exercise this in CI because of the priv dropping
component, but the included example probe utility confirmed it for
me outside of the harness:
```
bdad88ed680e# install -d -o wez -g wez -m 2700 /tmp/probe-demo
bdad88ed680e# ./target/debug/examples/probe /tmp/probe-demo --user wez
probing /tmp/probe-demo as ruid=0 euid=1000 rgid=1000 egid=1000
FAIL: /tmp/probe-demo (directory owner uid=1000 gid=1000 mode=2700); process ruid=0 euid=1000 rgid=1000 egid=1000: inconsistent
view of /tmp/probe-demo/.kumo-dir-probe9nMaUj.renamed: access(2) reports the file present=false but open(2) reports it present
=true. This means the process real and effective user ids differ (a privilege drop) and the directory permissions are too restr
ictive for one of those identities. A database opened here would decide to create a fresh instance yet write over the existing
files, corrupting itself on the next startup. Ensure the directory is owned by, and grants rwx to, the identity the service run
s as.
bdad88ed680e# chmod 2755 /tmp/probe-demo
bdad88ed680e# ./target/debug/examples/probe /tmp/probe-demo --user wez
probing /tmp/probe-demo as ruid=0 euid=1000 rgid=1000 egid=1000
PASS: /tmp/probe-demo is usable by this identity
bdad88ed680e# rm -rf /tmp/probe-demo
```
When running under a cgroup limit, get_cgroup() reported
stat.usage_in_bytes (memory.current on v2, memory.usage_in_bytes on v1)
as the process memory usage. That counter includes all page
cache, including cold reclaimable cache (inactive_file). For a
workload that writes logs and spool to disk, cold cache can dominate:
on a real pod memory.current was ~51GB while inactive_file was ~46GB
and the true non-cache footprint was ~1.8GB. kumod saw usage near its
limit, hit get_headroom() == 0, and ran
shrink_ready_queue_due_to_low_mem
and other reductions against pressure that was almost entirely
reclaimable cache the kernel would drop before any OOM.
Report the working set instead:
```
working_set = max(memory.current - inactive_file, anon)
```
This matches container_memory_working_set_bytes (kubelet/cAdvisor) and
the kernel's own reclaimability accounting. It still counts everything
that can drive an OOM: anonymous memory, dirty page cache, active_file,
tmpfs/shm, and slab.
Signed-off-by: Daniel Schaaff <daniel@danielschaaff.com>
closes: https://github.com/KumoCorp/kumomta/pull/549
A command line that decoded to invalid UTF-8 propagated the decode
error out of read_line, which the session handler turned into a
421 technical difficulties response that tore down the connection.
Model it as a ReadLine variant so both the command loop and the AUTH
exchange emit a 501 syntax error and keep the session alive.
Closes#550
Per RFC 7672 section 2.2.2, an MX host that is a securely published
CNAME remains DANE-eligible at its original name even when the alias
target lands in an unsigned zone: it is the secure TLSA RRset, not the
address records, that authenticates the peer. When the address chain is
insecure but MX selection was secure, an explicit CNAME query isolates
the alias's own DNSSEC status; a secure alias engages DANE, an
indeterminate status defers for downgrade resistance.
refs: https://github.com/KumoCorp/kumomta/pull/545#discussion_r3472353021
This allows you to explicit indicate that an mx_list should
be treated as DNSSEC secure.
We cannot simply assume that lua code setting mx_list be treated
as secure, because the lua code may be simply passing through
the result of some other DNS lookup--we need an explicit way
to thread through the overall secure flag in order for the
secure property to be correctly upheld.
trust_anchor_file now accepts `{ managed = "<path>" }` for an RFC 5011
auto-maintained anchor file (unbound only; hickory errors). Unbound seeds
the file from the bundled root anchors when the file is absent.
When we first built out DANE, the hickory implementation was
all or nothing when DNSSEC validation was enabled, making it unsuitable
for use in an MTA.
Since that time hickory has matured; this commit wires up the
DNSSEC signals and that enables its use with DANE.
We do still require openssl for establishing the connection when
DANE is enabled; that's a different constraint.
This is technically a breaking change, but in a good way.
We will no longer send SMTP AUTH PLAIN creds when we didn't
validate the peer certificate. This is what you want in
all production deployments. The new option facilitates
test setups or other unusual configurations where the certs
cannot be verified but you have contrived to trust the
remote system through some other means.
This commit removes the queue_name_for_config_change_purposes_only wart
from inside the readyq machinery. This field had a bunch of caveats
and could in some cases become stale if the originating scheduled
queue had aged out.
It's been replaced by a HashSet of Weak refs back to the associated
scheduled queues so that it is now possible to walk the reverse of
the fan-in graph when we need to.
This is fairly big diff because a number of hickory types moved around
or changed signature.
In particular, the config structs changed in a way that would fan out
and require attention by our users. So the approach I opted to take
here was to define our own types that look like the hickory 0.25
shape and then we have explicit logic to map those to hickory and
also to our unbound resolver crate.
This commit also bumps up the unbound resolver crate because it also
uses hickory's types. I took the opportunity to upgrade the version
of the embedded unbound resolver as part of that work.
and also in resolve-shaping-domain, for consistency.
We can use the message rate from the scheduled queue config to refine
the computed ceilings/constraints too, which is nice.
This can be computed for an EgressPathConfig and is useful to indicate
what the effective ceilings are for this egress path.
We return this from inspect-ready-q
The ready queue maintainer now tracks for each dispatcher whether it
is making progress. If it exceeds dispatcher_progress_watchdog_timeout
then the task will be aborted and any message(s) it held onto will
be returned to the scheduled queue. This will free up the connection
slot and allow another connection to be attempted, potentially
making further progress.
refs: https://github.com/KumoCorp/kumomta/issues/539
Egress sources can now be configured to auto-suspend when their local
bind address appears unplumbed or when their configured proxy server
appears unreachable. A suspended source is skipped during pool selection
until the configured duration elapses.
We'll expand these to a list of candidate addresses, trying each in
turn. We use the same overall connect timeout regardless of how
many candidates are present, to respect that timeout bound.
closes: https://github.com/KumoCorp/kumomta/pull/402
We recently investigated an issue where a rocksdb had been damaged by
corrupting/removing SST files (it sounded like this was accidentally
self-inflicted by some backup/orchestration infrastructure) leaving the
system in a silently-broken state: writes just wouldn't make progress
and there were no error messages.
Inspecting the `/var/spool/kumomta/data/LOG` log file (which is a
readable text file) revealed messages like:
```
2026/06/12-14:55:31.884227 2875746 [ERROR] [db/compaction/compaction.cc:262] Unable to load table properties for file 29704 --- IO error: No such file or directory: While open a file for random read: /var/spool/kumomta/data/029704.sst: No such file or directory
2026/06/12-14:55:31.884311 2875746 [ERROR] [db/db_impl/db_impl_compaction_flush.cc:3385] Waiting after background compaction error: IO error: No such file or directory: While open a file for random read: /var/spool/kumomta/data/029704.sst: No such file or directory, Accumulated background error counts: 6363
```
This commit improves the observability in this situation by proactively
checking for error conditions:
1. The store() and remove() operations now use our own polling within
a deadline loop rather than spawning a blocking task and delegating
to rocksdb's blocking interface. This allows us to inspect the
background error count and be cancellable, safely respecting and
caller provided smtp max transaction duration.
2. All read and write operations check for IO and Corruption errors
and immediately latch an error state
3. The metrics monitoring task inspects and track background error
counts and latch us into an unhealthy state when the background
error state appears unhealthy and persistent.
4. Additional metrics are exposed to help monitoring and alerting
While adding integration test coverage for this, I found a typo that
meant that spool errors were ignored in the message crate; they got
silently converted to `true` in all cases rather than just mapping
the success case to a `true`.
Integration tests handle the case where an SST file is corrupted
(truncated) during runtime, as well as starting up when an SST file
is missing. These excercise both the foreground and background
error detection paths.
Wez Furlong
kay ozaki
Alex Burch
Mike Hillyer
Daniel Schaaff