Add some internal functions that return the set of valid field
names for the various make_XXX config building lua functions
so that we can change the cost of validating the helper config
types via typing.lua from O(N) relatively expensive individual
calls through lua/mlua/rust into O(1) lua table lookups.
This takes the cost of loading a 2000 entry queues.toml down
from an uncomfortable 300ms to below 50ms and the cost of
the get_queue_config event handler down from 240us to 10us.
The same pattern is also applied to listener_domains.lua
and sources.lua.
Add a `StickyWeightedRoundRobin` selection strategy for egress pools that
retains the same assigned source for a message across all retries, while
still using weighted round robin for the first attempt.
Implements #670
Provider-scoped automation rules using SetConfig now apply their override
to the full MX cluster for the triggering domain, rather than just that
single domain.
Avoid client connection stalls and timeouts during connection bursts by
allowing configuration of the listen backlog, replacing the
rust-internal hardcoded 128 entry limit.
Add a centralized LRU cache for compiled templates, avoiding repeated
compilation per HTTP injection request. This yields large performance
gains for Handlebars workloads.
Closes: https://github.com/KumoCorp/kumomta/pull/638
Fixes the timeout deadlock where calling a memoized function while loading
the policy would hang waiting for cache population. This also fixes the
same deadlock when cache population triggers a policy reload, and retries
population correctly when an in-progress attempt exits early.
When enabled (default is disabled), domains with insecure MX selection
but valid DANE TLSA records use a distinct site name. This prevents an
insecure domain from denying DANE authentication to DNSSEC secure
sibling domains on the same MX hosts.
MTA-STS and DANE policies can only raise, never lower, the configured
enable_tls level for a destination.
Integration tests and documentation adapted from @terrorobe's PR #616.
closes: https://github.com/KumoCorp/kumomta/pull/616
Per-thread-pool tokio runtime metrics are now exported for every named
runtime, not just the main runtime. An lruttl_populate_duration histogram
records cache populate time. An integration test verifies both metrics are
exported.
Add kumo.serde.load_files which loads and parses a list of JSON and
TOML files with only one await point, which is more efficient than
the individual calls for folks using the sources helper with large
numbers of source toml files.
Add kumo.string.psl_suffix_is_known and kumo.string.psl_domain_is_known,
which report whether the resolved public suffix is explicitly listed in
Mozilla's Public Suffix List.
The psl crate applies a "fallback rule": a domain whose public suffix is
not in the list (e.g. `domain.invalid`) still resolves to a suffix, but
that suffix is not explicitly known. Existing kumo.string.psl_suffix /
psl_domain discard this information, so fallback-resolved suffixes are
indistinguishable from explicitly listed ones.
The new functions return `true` when the suffix is explicitly listed,
or `false` otherwise (fallback-resolved, or no public suffix at all).
closes: https://github.com/KumoCorp/kumomta/pull/592
Matches the existing max_connections behavior from SMTP and proxy listeners.
Operators can monitor and alert on endpoint saturation.
refs: https://github.com/KumoCorp/kumomta/issues/631
Nobody runs it and it is not as good as RocksDB. I want to remove it
in the future so that users don't unwittingly deploy a poor configuration.
We now log errors on startup to help surface configuration that will
need attention, but we don't block startup.
Since hickory can now be used to enable DANE, the only remaining
use case for the unbound embedded resolver is disolved.
I want to remove it from the tree because:
* I don't want to own the rust crate wrapping libunbound
* Reduces complexity
* Users have reported issues with unbound that simply disappear
by switching back to hickory
* Essentially nobody is running kumomta with unbound enabled,
so it is has no real production coverage: you're a snowflake
if you're running it
Removing it will reduce the potential for a user unwittingly enabling
something that isn't known to work as well as the golden path.
The deprecation manifests as an error printed on startup when
enabling unbound, and adjustments in the docs.
It will continue to work as it does today.
Cache population now runs detached by default for top-level memoized
functions. Other callers waiting on the same key now receive the result
even if the original triggering caller is cancelled mid-populate;
previously that case could impact the other yet-to-be-timed-out
waiters and cause cache population to restart or otherwise fail.
refs: #602
Lets operators tell local concurrency permit wait timeouts apart from real
remote resolver query timeouts, to avoid misdiagnosing overload as remote
deliverability failure.
Adds metrics dns_mx_resolve_permit_timeout, dns_mx_resolve_query_timeout
and dns_mx_resolve_permit_waiters.
Indeterminate MX lookup failures now use a short 30s transient negative TTL,
while NXDOMAIN and policy rejections retain the 5 minute TTL. Lookups blocked
by the concurrency limit are not cached at all. You can configure the transient
TTL with kumo.dns.set_mx_transient_negative_cache_ttl.
This prevents transient failures from causing domains to drop out of shaping
rollups and incorrectly flipping the shaping hash.
Explain the factoring phases and input edge cases, and make the DNS
fixture and session-reuse timeout intent easier to read.
Move site-name guidance to the queues reference with an inline release
marker, and link to it from lookup_mx and the traffic-shaping guide.
Add a breaking-change note with an observed collision pair.
Normalize, deduplicate and sort MX destinations before deriving site
names. Use per-label alternatives only when their expansion matches
the destination set; otherwise factor complete hostname branches.
Keep MX preferences in connection plans rather than site identity.
Keep asterisks literal and percent-encode other special label bytes
so names remain distinct without introducing metric-label escapes.
Cover collisions, normalization, ports, shaping and queue/session
sharing, and document the naming and preference semantics.
Cap iprev at 10 PTR names for forward lookups, as required by RFC 8601.
Prevents malicious reverse DNS zones from causing unbounded DNS lookups.
closes: https://github.com/KumoCorp/kumomta/issues/623
A message with 8-bit headers or body could not be bounced at all: building the
returned-content part failed and aborted the whole report. It now produces a
7-bit clean report, returning as much of the original as can be represented and
omitting the rest.
Prevent transient accept() errors from permanently stopping listeners
while the process stays alive and appears healthy. Trivial peer resets
that happen between kernel queue and accept are logged only at debug
level to close remote log spam vectors.
Publish metrics showing how far each out-of-process log consumer trails
the newest data in the directories this node logs to, so that operators
can tell whether a consumer is keeping up. Covers both the
configure_local_logs and kumo.jsonl.new_writer directories.
The documentation generators now stamp each page they emit with an
@generated comment so that tooling can tell generated pages apart from
hand-written ones. Update the table-of-contents generator to skip YAML
front matter when picking a page title, since the marker is now stored
there.
The prior commit focused on avoiding a panic when we encountered an invalid
boundary string. This commit makes thing degrade more gracefully: we'll
treat the bad MIME part in the same way that we deal with overly deep nesting
and handle it as an opqaque leaf part.
The new_multipart constructor now rejects an invalid boundary rather than
building an unserializable part, so a flawed policy can't accidentally create
broken messages.
Deeply nested multipart content is now retained as an opaque part with
its headers parsed and raw body preserved once nesting passes 100
levels, instead of recursing until the process stack overflows.
Those messages are tagged with the new MIME_NESTING_LIMIT_EXCEEDED
conformance flag.
Headers were built and rebuilt as "Mime-Version" rather than
"MIME-Version". Both are valid per RFC 2045, but some spam filters,
such as rspamd, score the mixed-case form as a deliverability signal.
Fix the header name in the accessor macro and in ParsedHeader's
grammar table, the places that spell it, and update the snapshot
tests across mailparsing, message, kumo-log-types, mod-mimepart, and
kumod that captured the old spelling, along with the reference docs
that showed the old casing in their examples. #564
When a record exceeds the configured max_line_size, discard just that record
and continue reading the rest of the segment instead of aborting the entire
segment.
The zstd decompression buffer now grows on demand to read records up to
a configurable 128 MiB cap. Oversized records are rejected explicitly
both on read and write, with matching limits so all written records are
guaranteed readable.
The gate could latch permanently when only one of the two error signals
(foreground or background) ever moved, requiring an operator restart. It now
auto-reopens for a retry after `error_unlatch_duration`, regardless of which
signal latched it.
Also serializes foreground error reporting with gate transitions so a
concurrent fatal error cannot be undone by a reopen, and rejects a zero
`error_unlatch_duration` when automatic reopening is enabled. The underlying
cause is always logged. Adds a reusable LD_PRELOAD fault-injection crate and
integration tests that reproduce a full disk and a slow disk to drive the
latch-and-reopen cycle end to end.
Fixes#597
This sets up plumbing to allow testing the broken mta-sts aliasing
issue, and enables feeding an optional resolver through the mx lookups
as well.
closes: https://github.com/KumoCorp/kumomta/pull/524
refs: https://github.com/KumoCorp/kumomta/issues/484
Briefly, the issue is that if some random domain that shares MX records
with another (eg: someone is using google apps or icloud for their
vanity domain) publishes a broken MTA-STS policy that requires eg:
cloudflare MX hosts then because we roll up by site name, that broken
MTA-STS policy bleeds into all the other domains that share those MX
records.
The resolution is simple, but is technically a breaking change.
Moving the policy resolution to happen during site_name resolution
allows us to resolve both per-domain things at the same and have the
MTA-STS policy amend the effective set of MX hosts. The output of that
is then used for site_name aggregation/rollup.
The consequence of this is quite nice: an MTA-STS policy that is more
restrictive than the full set of MX hosts now prevents delivering to
any of the excluded hosts, and a totally broken policy that prevents all
of its MX hosts is now completely undeliverable and will produce
transient failures.
The downside is that for users that had previously disabled mta-sts in
their default shaping block, they will need to change a different config
option to continue to prevent MTA-STS from being consulted. One example
of this that I recall is that one user's network posture prevented
MTA-STS from making HTTPS requests to fetch the policy. Another user
just wanted to cut out the additional DNS traffic. Those use cases
require altering the new kumo.dns.set_mta_sts_enabled enabled to false
during `init`.
When running under a cgroup limit, get_cgroup() reported
stat.usage_in_bytes (memory.current on v2, memory.usage_in_bytes on v1)
as the process memory usage. That counter includes all page
cache, including cold reclaimable cache (inactive_file). For a
workload that writes logs and spool to disk, cold cache can dominate:
on a real pod memory.current was ~51GB while inactive_file was ~46GB
and the true non-cache footprint was ~1.8GB. kumod saw usage near its
limit, hit get_headroom() == 0, and ran
shrink_ready_queue_due_to_low_mem
and other reductions against pressure that was almost entirely
reclaimable cache the kernel would drop before any OOM.
Report the working set instead:
```
working_set = max(memory.current - inactive_file, anon)
```
This matches container_memory_working_set_bytes (kubelet/cAdvisor) and
the kernel's own reclaimability accounting. It still counts everything
that can drive an OOM: anonymous memory, dirty page cache, active_file,
tmpfs/shm, and slab.
Signed-off-by: Daniel Schaaff <daniel@danielschaaff.com>
closes: https://github.com/KumoCorp/kumomta/pull/549
Per RFC 7672 section 2.2.2, an MX host that is a securely published
CNAME remains DANE-eligible at its original name even when the alias
target lands in an unsigned zone: it is the secure TLSA RRset, not the
address records, that authenticates the peer. When the address chain is
insecure but MX selection was secure, an explicit CNAME query isolates
the alias's own DNSSEC status; a secure alias engages DANE, an
indeterminate status defers for downgrade resistance.
refs: https://github.com/KumoCorp/kumomta/pull/545#discussion_r3472353021
This allows you to explicit indicate that an mx_list should
be treated as DNSSEC secure.
We cannot simply assume that lua code setting mx_list be treated
as secure, because the lua code may be simply passing through
the result of some other DNS lookup--we need an explicit way
to thread through the overall secure flag in order for the
secure property to be correctly upheld.
trust_anchor_file now accepts `{ managed = "<path>" }` for an RFC 5011
auto-maintained anchor file (unbound only; hickory errors). Unbound seeds
the file from the bundled root anchors when the file is absent.
When we first built out DANE, the hickory implementation was
all or nothing when DNSSEC validation was enabled, making it unsuitable
for use in an MTA.
Since that time hickory has matured; this commit wires up the
DNSSEC signals and that enables its use with DANE.
We do still require openssl for establishing the connection when
DANE is enabled; that's a different constraint.