Commit Graph
1654 Commits
Author SHA1 Message Date
Wez Furlong 81f191dff6 config: improve typing.lua performance for queues.lua
Add some internal functions that return the set of valid field
names for the various make_XXX config building lua functions
so that we can change the cost of validating the helper config
types via typing.lua from O(N) relatively expensive individual
calls through lua/mlua/rust into O(1) lua table lookups.

This takes the cost of loading a 2000 entry queues.toml down
from an uncomfortable 300ms to below 50ms and the cost of
the get_queue_config event handler down from 240us to 10us.

The same pattern is also applied to listener_domains.lua
and sources.lua.
2026-11-11 07:44:55 +00:00
Wez Furlong 5ae9ba0777 map: make DomainMap, CidrMap and RegexSetMap clones cheap
These types now use Arc shared storage with copy-on-write for inserts.
Clone is now a cheap pointer increment instead of a full deep copy,
removing the per-message cost that scaled linearly with queues.toml size.
2026-11-10 21:07:39 +00:00
Wez Furlong 8f9f63b078 egress_source: add sticky source assignment for egress pools
Add a `StickyWeightedRoundRobin` selection strategy for egress pools that
retains the same assigned source for a message across all retries, while
still using weighted round robin for the first attempt.

Implements #670
2026-11-10 16:14:08 +00:00
Wez Furlong 1d3f34574f queue: check message expiry before queuing
Check expiry whenever a message is placed on a delivery queue, not only
after failed delivery attempts.

Fixes #671
2026-11-10 09:16:28 +00:00
Wez Furlong 88432e2238 shaping: SetConfig from provider rules now uses mx_rollup=true
Provider-scoped automation rules using SetConfig now apply their override
to the full MX cluster for the triggering domain, rather than just that
single domain.
2026-11-10 08:35:51 +00:00
Wez Furlong 8ee5e392ca shaping: add SetProviderConfig automation action
Add an automation action that applies config overrides scoped to an entire
matched provider, rather than only the triggering domain.
2026-11-10 08:35:51 +00:00
Wez Furlong 0a8515194e listener: make listen backlog configurable
Avoid client connection stalls and timeouts during connection bursts by
allowing configuration of the listen backlog, replacing the
rust-internal hardcoded 128 entry limit.
2026-11-09 16:35:07 +00:00
Wez Furlong 958e4668f1 acct: make max_file_size work for accounting logs
The `max_file_size` option of `kumo.aaa.configure_acct_log` had no
effect because the written byte counter was never incremented.
2026-11-08 07:56:56 +00:00
Wez Furlong adbd37063e acct: fix missing flush on shutdown
Ensure that we explicitly flush and wait for the log to finish
writing on shutdown.
2026-11-08 07:21:00 +00:00
Wez Furlong 5814ed7ec1 queue: fix indefinite graceful shutdown hang
Prevent graceful shutdown from hanging indefinitely waiting for
the queue meta maintainer when shutting down while messages are
in flight.

Closes: #654
2026-11-06 13:33:42 +00:00
Wez Furlong 2d1966d2e2 kumo-template: cache compiled templates across requests
Add a centralized LRU cache for compiled templates, avoiding repeated
compilation per HTTP injection request. This yields large performance
gains for Handlebars workloads.

Closes: https://github.com/KumoCorp/kumomta/pull/638
2026-11-06 07:05:23 +00:00
Wez Furlong e61bc0844a docs: fixup ci build
Recent github ubuntu image changes broke the rocksdb build
in our docs CI flow, so switch it to our builder image where
rocksdb is pre-built.

Revise the build to use a pre-installed mkdocs so that we don't
have to docker it.
2026-11-05 11:30:52 +00:00
Wez Furlong 980521eb67 docs: format this page 2026-11-05 11:29:55 +00:00
Wez Furlong 7a32d83ec6 smtp_dispatcher: add send-time DANE consistency guard
When an egress path has enable_dane set true but the global DANE
isolation switch is off, defer messages with a 451 transient failure
instead of risking silent DANE downgrades on shared ready queues.
2026-11-05 08:10:32 +00:00
Wez Furlong 79c1b80174 xfer: reset attempt count when moving messages to xfer queues
Reset a message's delivery attempt count when it moves to an xfer queue,
preventing unsafe backoff calculation overflow that caused panics, premature
expiry, or endless immediate retries.

Reported-by: @edgarsendernet
Closes: #644
2026-11-05 06:20:20 +00:00
3abb6bb615 kumo-template: avoid copying the Handlebars context (#637)
handlebars::Context::wraps serializes its argument with
serde_json::to_value, so passing it an existing serde_json::Value
deep-copies the whole substitution context a second time.

Both Handlebars paths already hold an owned serde_json::Value at that
point (render_to_write, and prepare_context which batch injection uses
once per recipient), so take ownership of it with Context::from
instead. The rendered output is unchanged, and the error case that
wraps reported (a serialization failure) cannot occur for a value that
is already JSON.

---------

Co-authored-by: Akshit Sethi <akshits@adobe.com>
Co-authored-by: Wez Furlong <wez@wezfurlong.org>
2026-11-04 16:42:56 +00:00
Wez Furlong 507781eaa1 memoize: fix deadlock on file-scope memoized calls during policy load
Fixes the timeout deadlock where calling a memoized function while loading
the policy would hang waiting for cache population. This also fixes the
same deadlock when cache population triggers a policy reload, and retries
population correctly when an in-progress attempt exits early.
2026-11-04 14:37:14 +00:00
Wez Furlong 0b64c9a682 dns: isolate DANE-capable insecure MX domains
When enabled (default is disabled), domains with insecure MX selection
but valid DANE TLSA records use a distinct site name. This prevents an
insecure domain from denying DANE authentication to DNSSEC secure
sibling domains on the same MX hosts.
2026-11-04 07:21:38 +00:00
Wez Furlong b0f0148c26 smtp_dispatcher: defer MTA-STS "enforce" mail on shared cleartext connection
When multiple recipient domains share a ready queue, a message for a domain
that publishes MTA-STS enforce will be deferred rather than delivered over
a cleartext connection established by another domain.
2026-11-04 07:21:37 +00:00
Wez Furlong a6bf765101 smtp: only allow MTA-STS and DANE to raise TLS policy
MTA-STS and DANE policies can only raise, never lower, the configured
enable_tls level for a destination.

Integration tests and documentation adapted from @terrorobe's PR #616.

closes: https://github.com/KumoCorp/kumomta/pull/616
2026-11-04 07:21:35 +00:00
Wez Furlong 9075eb4415 smtp_dispatcher: clear tls_info between connections 2026-11-04 07:20:56 +00:00
Wez Furlong 6b517bccee Revert "Update get_queue_config docs (#651)"
This reverts commit 45b3b455fd,
which was merged in error.
2026-10-23 06:40:03 +01:00
Tom Mairs 45b3b455fd Update get_queue_config docs (#651)
* Modify kumo-proxy.md to reflect latest functions.

* Correct queue handling process with error and nil.

* Merge branch 'main' into tm221020262135
2026-10-22 21:47:52 -06:00
Wez Furlong e99031cbbe dns-resolver: mitigate hickory 0.26 dnssec issue
https://github.com/hickory-dns/hickory-dns/issues/3974 poses problems
for users that need dnssec validation

This commit points to the current ref from the `release/0.26` branch
for the moment; when hickory makes a new release we'll re-point.

This commit also enables live-dns-tests in the main `make test` flow
to make it more likely to detect issues of this sort in future,
at the cost of potentially more flakes in CI.
2026-10-22 16:03:28 +01:00
Wez Furlong 685ccdc3ec policy-extras: add cache tuning options for sources.lua
Add the same set of memoize tuning parameters to the sources helper
that are already available on the shaping helper, fully backwards
compatible.
2026-10-21 18:00:06 +01:00
Wez Furlong 95d80d96d3 metrics: export per-thread-pool runtime metrics and cache populate duration
Per-thread-pool tokio runtime metrics are now exported for every named
runtime, not just the main runtime. An lruttl_populate_duration histogram
records cache populate time. An integration test verifies both metrics are
exported.
2026-10-21 18:00:06 +01:00
Wez Furlong 7f57cf07fa config: move Lua VM teardown off pool mutex and worker threads
Use a dedicated background dropper thread for stale Lua VMs. Avoid running
lua_close while holding the pool lock or on tokio worker threads.
2026-10-21 18:00:06 +01:00
Wez Furlong 83fe178032 kumo.serde: add kumo.serde.load_files
Add kumo.serde.load_files which loads and parses a list of JSON and
TOML files with only one await point, which is more efficient than
the individual calls for folks using the sources helper with large
numbers of source toml files.
2026-10-21 18:00:05 +01:00
Harsh Jha 1540e6a51d rfc3464: recognize recipient address types case-insensitively
RFC 3464 section 2.1.2 makes address types case-insensitive, but the
recipient parser only normalizes addresses for an exact lowercase rfc822
match. This leaves angle brackets intact in OOB recipients for RFC822
reports.

Trim the type before comparing it case-insensitively, reuse the existing
SMTP address parser, and cover original and final recipients with tests.
Preserve the reported type spelling and other address types.

closes: https://github.com/KumoCorp/kumomta/pull/640
2026-10-21 10:39:23 +01:00
Wez Furlong 91ac946abb docs: changelog for #615
closes: https://github.com/KumoCorp/kumomta/pull/615
2026-10-21 09:04:45 +01:00
Wez Furlong 0f9cc1f39d smtp_server: drop connection on failed STARTTLS handshake
When a STARTTLS handshake fails, close the connection instead of continuing
to accept cleartext commands.

refs: https://github.com/KumoCorp/kumomta/pull/615
2026-10-21 09:04:43 +01:00
Wez Furlong 746b5632bd smtp_dispatcher: Remove dead cleartext fallback after failed STARTTLS handshake
Receiving servers always close the connection on handshake failure. This
code path never succeeded in practice.

refs: https://github.com/KumoCorp/kumomta/pull/615
2026-10-21 09:04:42 +01:00
Umut Sahin Onder 41afc7914a string: add psl_suffix_is_known / psl_domain_is_known helpers
Add kumo.string.psl_suffix_is_known and kumo.string.psl_domain_is_known,
which report whether the resolved public suffix is explicitly listed in
Mozilla's Public Suffix List.

The psl crate applies a "fallback rule": a domain whose public suffix is
not in the list (e.g. `domain.invalid`) still resolves to a suffix, but
that suffix is not explicitly known. Existing kumo.string.psl_suffix /
psl_domain discard this information, so fallback-resolved suffixes are
indistinguishable from explicitly listed ones.

The new functions return `true` when the suffix is explicitly listed,
or `false` otherwise (fallback-resolved, or no public suffix at all).

closes: https://github.com/KumoCorp/kumomta/pull/592
2026-10-21 09:02:07 +01:00
Wez Furlong 47c1df231e smtp: redact AUTH PLAIN credentials from logs/diagnostics
Redact credentials from both inbound rejection logs and outbound
delivery failure records. Credentials are replaced with [REDACTED] in all
log and diagnostic output.

closes: https://github.com/KumoCorp/kumomta/pull/636
2026-10-21 08:37:31 +01:00
Wez Furlong 442dc4114a docs: Document tls_mode option for inbound and outbound paths
Add reference documentation for the tls_mode option covering StartTLS and
ImplicitTLS modes, plus a changelog entry.

closes: #199
closes: #200
2026-10-16 13:57:55 +01:00
Wez Furlong 7c99973dfc http-inject: Add resizable concurrency limit with saturation metrics
Matches the existing max_connections behavior from SMTP and proxy listeners.
Operators can monitor and alert on endpoint saturation.

refs: https://github.com/KumoCorp/kumomta/issues/631
2026-10-14 14:01:32 +01:00
Wez Furlong ea9086037a kumo-template: avoid repeated serialization for batch template renders
Add a prepared context that is serialized once per recipient rather
than once per template, removing redundant work during batch injection.
2026-10-13 15:00:09 +01:00
Wez Furlong f76e5672c1 inject: avoid per-recipient evaluation for static headers
Precompile static headers once at batch compile time and clone them per
recipient. Only headers that actually use template variables are evaluated
and constructed separately for each recipient.

closes: https://github.com/KumoCorp/kumomta/pull/620
2026-10-12 15:48:00 +01:00
Wez Furlong 170dad668b message: enforce correct residency gauges via RAII
Fixes two concrete bugs: empty message bodies leaked the resident count
upward, and node-to-node transferred messages were never counted, driving
the gauge downward. All modifications are in crates/message, with an
accompanying changelog entry.

Fixes: #588
Reported-by: @kayozaki
2026-10-12 12:09:00 +01:00
Wez Furlong 0c5bf90b66 config: add the remaining missing config.put() calls
Per the previous commit, these are all missed opportunities
to allow reusing the previously built lua context.

refs: #632
2026-10-10 07:28:46 +01:00
Edgar Vaitkevicius 79ef6dc76a Reduced lua context creation overhead in scheduled queue config refreshes by reusing the cached context. 2026-10-10 07:16:47 +01:00
Wez Furlong daf4d942ac smtp: Handle partial mx_list lookup failures gracefully
Skip failing entries and proceed with healthy alternatives. When all entries
fail, defer delivery with a transient failure listing the lookup errors.
Previously failures were only logged at debug level with no delivery failure
recorded.

Closes #628
2026-10-09 17:47:51 +01:00
Wez Furlong 9600aca5f2 dns-resolver: limit cache TTL when one address family fails
When one address family lookup fails in a dual-stack query, limit the
combined cache entry expiry to 60 seconds instead of using the full TTL
of the surviving family. This avoids pinning the failed family out of
use for up to a day. #630
2026-10-09 17:47:51 +01:00
Wez Furlong 74e25a88d1 mailexchanger: fix implicit MX fallback on failed MX lookups
Treat SERVFAIL, REFUSED, and DNSSEC-bogus MX answers as temporary errors
instead of misinterpreting them as NODATA. This prevents such answers from
incorrectly triggering fallback to the domain's A/AAAA records.

Refs: #629
2026-10-09 17:47:51 +01:00
Wez Furlong b1e1262e2b spool: deprecate kind = LocalDisk
Nobody runs it and it is not as good as RocksDB.  I want to remove it
in the future so that users don't unwittingly deploy a poor configuration.

We now log errors on startup to help surface configuration that will
need attention, but we don't block startup.
2026-10-09 17:47:51 +01:00
Wez Furlong 04d7acd892 kumo.dns.configure_unbound_resolver: deprecated!
Since hickory can now be used to enable DANE, the only remaining
use case for the unbound embedded resolver is disolved.

I want to remove it from the tree because:

 * I don't want to own the rust crate wrapping libunbound
 * Reduces complexity
 * Users have reported issues with unbound that simply disappear
   by switching back to hickory
 * Essentially nobody is running kumomta with unbound enabled,
   so it is has no real production coverage: you're a snowflake
   if you're running it

Removing it will reduce the potential for a user unwittingly enabling
something that isn't known to work as well as the golden path.

The deprecation manifests as an error printed on startup when
enabling unbound, and adjustments in the docs.

It will continue to work as it does today.
2026-10-09 17:47:42 +01:00
Wez Furlong c80cb8fabe memoize: prevent cancelled callers from aborting shared cache population
Cache population now runs detached by default for top-level memoized
functions. Other callers waiting on the same key now receive the result
even if the original triggering caller is cancelled mid-populate;
previously that case could impact the other yet-to-be-timed-out
waiters and cause cache population to restart or otherwise fail.

refs: #602
2026-10-09 10:35:44 +01:00
Wez Furlong 4170955400 mailexchanger: Distinguish MX lookup timeout causes in logs and metrics
Lets operators tell local concurrency permit wait timeouts apart from real
remote resolver query timeouts, to avoid misdiagnosing overload as remote
deliverability failure.

Adds metrics dns_mx_resolve_permit_timeout, dns_mx_resolve_query_timeout
and dns_mx_resolve_permit_waiters.
2026-10-08 14:58:52 +01:00
Wez Furlong 3e1932a86a dns: avoid long caching of transient MX lookup failures
Indeterminate MX lookup failures now use a short 30s transient negative TTL,
while NXDOMAIN and policy rejections retain the 5 minute TTL.  Lookups blocked
by the concurrency limit are not cached at all.  You can configure the transient
TTL with kumo.dns.set_mx_transient_negative_cache_ttl.

This prevents transient failures from causing domains to drop out of shaping
rollups and incorrectly flipping the shaping hash.
2026-10-08 14:27:34 +01:00
Wez Furlong 9448f92e11 log_hooks: Add log_parameters pass-through to new_disposition_hook
You can now set per_record and other log options directly via the
new_disposition_hook helper.

Closes: #499
Closes: #500
Closes: #519
Closes: #520
2026-10-08 06:47:54 +01:00