Add some internal functions that return the set of valid field
names for the various make_XXX config building lua functions
so that we can change the cost of validating the helper config
types via typing.lua from O(N) relatively expensive individual
calls through lua/mlua/rust into O(1) lua table lookups.
This takes the cost of loading a 2000 entry queues.toml down
from an uncomfortable 300ms to below 50ms and the cost of
the get_queue_config event handler down from 240us to 10us.
The same pattern is also applied to listener_domains.lua
and sources.lua.
These types now use Arc shared storage with copy-on-write for inserts.
Clone is now a cheap pointer increment instead of a full deep copy,
removing the per-message cost that scaled linearly with queues.toml size.
Add a `StickyWeightedRoundRobin` selection strategy for egress pools that
retains the same assigned source for a message across all retries, while
still using weighted round robin for the first attempt.
Implements #670
Provider-scoped automation rules using SetConfig now apply their override
to the full MX cluster for the triggering domain, rather than just that
single domain.
Avoid client connection stalls and timeouts during connection bursts by
allowing configuration of the listen backlog, replacing the
rust-internal hardcoded 128 entry limit.
Prevent graceful shutdown from hanging indefinitely waiting for
the queue meta maintainer when shutting down while messages are
in flight.
Closes: #654
Add a centralized LRU cache for compiled templates, avoiding repeated
compilation per HTTP injection request. This yields large performance
gains for Handlebars workloads.
Closes: https://github.com/KumoCorp/kumomta/pull/638
Recent github ubuntu image changes broke the rocksdb build
in our docs CI flow, so switch it to our builder image where
rocksdb is pre-built.
Revise the build to use a pre-installed mkdocs so that we don't
have to docker it.
When an egress path has enable_dane set true but the global DANE
isolation switch is off, defer messages with a 451 transient failure
instead of risking silent DANE downgrades on shared ready queues.
Reset a message's delivery attempt count when it moves to an xfer queue,
preventing unsafe backoff calculation overflow that caused panics, premature
expiry, or endless immediate retries.
Reported-by: @edgarsendernet
Closes: #644
handlebars::Context::wraps serializes its argument with
serde_json::to_value, so passing it an existing serde_json::Value
deep-copies the whole substitution context a second time.
Both Handlebars paths already hold an owned serde_json::Value at that
point (render_to_write, and prepare_context which batch injection uses
once per recipient), so take ownership of it with Context::from
instead. The rendered output is unchanged, and the error case that
wraps reported (a serialization failure) cannot occur for a value that
is already JSON.
---------
Co-authored-by: Akshit Sethi <akshits@adobe.com>
Co-authored-by: Wez Furlong <wez@wezfurlong.org>
Fixes the timeout deadlock where calling a memoized function while loading
the policy would hang waiting for cache population. This also fixes the
same deadlock when cache population triggers a policy reload, and retries
population correctly when an in-progress attempt exits early.
When enabled (default is disabled), domains with insecure MX selection
but valid DANE TLSA records use a distinct site name. This prevents an
insecure domain from denying DANE authentication to DNSSEC secure
sibling domains on the same MX hosts.
When multiple recipient domains share a ready queue, a message for a domain
that publishes MTA-STS enforce will be deferred rather than delivered over
a cleartext connection established by another domain.
MTA-STS and DANE policies can only raise, never lower, the configured
enable_tls level for a destination.
Integration tests and documentation adapted from @terrorobe's PR #616.
closes: https://github.com/KumoCorp/kumomta/pull/616
https://github.com/hickory-dns/hickory-dns/issues/3974 poses problems
for users that need dnssec validation
This commit points to the current ref from the `release/0.26` branch
for the moment; when hickory makes a new release we'll re-point.
This commit also enables live-dns-tests in the main `make test` flow
to make it more likely to detect issues of this sort in future,
at the cost of potentially more flakes in CI.
Per-thread-pool tokio runtime metrics are now exported for every named
runtime, not just the main runtime. An lruttl_populate_duration histogram
records cache populate time. An integration test verifies both metrics are
exported.
Add kumo.serde.load_files which loads and parses a list of JSON and
TOML files with only one await point, which is more efficient than
the individual calls for folks using the sources helper with large
numbers of source toml files.
RFC 3464 section 2.1.2 makes address types case-insensitive, but the
recipient parser only normalizes addresses for an exact lowercase rfc822
match. This leaves angle brackets intact in OOB recipients for RFC822
reports.
Trim the type before comparing it case-insensitively, reuse the existing
SMTP address parser, and cover original and final recipients with tests.
Preserve the reported type spelling and other address types.
closes: https://github.com/KumoCorp/kumomta/pull/640
Add kumo.string.psl_suffix_is_known and kumo.string.psl_domain_is_known,
which report whether the resolved public suffix is explicitly listed in
Mozilla's Public Suffix List.
The psl crate applies a "fallback rule": a domain whose public suffix is
not in the list (e.g. `domain.invalid`) still resolves to a suffix, but
that suffix is not explicitly known. Existing kumo.string.psl_suffix /
psl_domain discard this information, so fallback-resolved suffixes are
indistinguishable from explicitly listed ones.
The new functions return `true` when the suffix is explicitly listed,
or `false` otherwise (fallback-resolved, or no public suffix at all).
closes: https://github.com/KumoCorp/kumomta/pull/592
Redact credentials from both inbound rejection logs and outbound
delivery failure records. Credentials are replaced with [REDACTED] in all
log and diagnostic output.
closes: https://github.com/KumoCorp/kumomta/pull/636
Matches the existing max_connections behavior from SMTP and proxy listeners.
Operators can monitor and alert on endpoint saturation.
refs: https://github.com/KumoCorp/kumomta/issues/631
Precompile static headers once at batch compile time and clone them per
recipient. Only headers that actually use template variables are evaluated
and constructed separately for each recipient.
closes: https://github.com/KumoCorp/kumomta/pull/620
Fixes two concrete bugs: empty message bodies leaked the resident count
upward, and node-to-node transferred messages were never counted, driving
the gauge downward. All modifications are in crates/message, with an
accompanying changelog entry.
Fixes: #588
Reported-by: @kayozaki
Skip failing entries and proceed with healthy alternatives. When all entries
fail, defer delivery with a transient failure listing the lookup errors.
Previously failures were only logged at debug level with no delivery failure
recorded.
Closes#628
When one address family lookup fails in a dual-stack query, limit the
combined cache entry expiry to 60 seconds instead of using the full TTL
of the surviving family. This avoids pinning the failed family out of
use for up to a day. #630
Treat SERVFAIL, REFUSED, and DNSSEC-bogus MX answers as temporary errors
instead of misinterpreting them as NODATA. This prevents such answers from
incorrectly triggering fallback to the domain's A/AAAA records.
Refs: #629
Nobody runs it and it is not as good as RocksDB. I want to remove it
in the future so that users don't unwittingly deploy a poor configuration.
We now log errors on startup to help surface configuration that will
need attention, but we don't block startup.
Since hickory can now be used to enable DANE, the only remaining
use case for the unbound embedded resolver is disolved.
I want to remove it from the tree because:
* I don't want to own the rust crate wrapping libunbound
* Reduces complexity
* Users have reported issues with unbound that simply disappear
by switching back to hickory
* Essentially nobody is running kumomta with unbound enabled,
so it is has no real production coverage: you're a snowflake
if you're running it
Removing it will reduce the potential for a user unwittingly enabling
something that isn't known to work as well as the golden path.
The deprecation manifests as an error printed on startup when
enabling unbound, and adjustments in the docs.
It will continue to work as it does today.
Cache population now runs detached by default for top-level memoized
functions. Other callers waiting on the same key now receive the result
even if the original triggering caller is cancelled mid-populate;
previously that case could impact the other yet-to-be-timed-out
waiters and cause cache population to restart or otherwise fail.
refs: #602
Lets operators tell local concurrency permit wait timeouts apart from real
remote resolver query timeouts, to avoid misdiagnosing overload as remote
deliverability failure.
Adds metrics dns_mx_resolve_permit_timeout, dns_mx_resolve_query_timeout
and dns_mx_resolve_permit_waiters.
Indeterminate MX lookup failures now use a short 30s transient negative TTL,
while NXDOMAIN and policy rejections retain the 5 minute TTL. Lookups blocked
by the concurrency limit are not cached at all. You can configure the transient
TTL with kumo.dns.set_mx_transient_negative_cache_ttl.
This prevents transient failures from causing domains to drop out of shaping
rollups and incorrectly flipping the shaping hash.