Add some internal functions that return the set of valid field
names for the various make_XXX config building lua functions
so that we can change the cost of validating the helper config
types via typing.lua from O(N) relatively expensive individual
calls through lua/mlua/rust into O(1) lua table lookups.
This takes the cost of loading a 2000 entry queues.toml down
from an uncomfortable 300ms to below 50ms and the cost of
the get_queue_config event handler down from 240us to 10us.
The same pattern is also applied to listener_domains.lua
and sources.lua.
These types now use Arc shared storage with copy-on-write for inserts.
Clone is now a cheap pointer increment instead of a full deep copy,
removing the per-message cost that scaled linearly with queues.toml size.
Add a `StickyWeightedRoundRobin` selection strategy for egress pools that
retains the same assigned source for a message across all retries, while
still using weighted round robin for the first attempt.
Implements #670
Provider-scoped automation rules using SetConfig now apply their override
to the full MX cluster for the triggering domain, rather than just that
single domain.
Avoid client connection stalls and timeouts during connection bursts by
allowing configuration of the listen backlog, replacing the
rust-internal hardcoded 128 entry limit.
When realloc fails, retain the original allocation's existing callstack
record instead of capturing a new one. This keeps memory diagnostics
accurate during allocation failures.
When callstack tracking is enabled, the prior deallocation cleanup was
running after the memory had been returned to the allocator, allowing
a new allocation to reuse the address before its record was cleared.
Use sharded counters keyed by allocation address to reduce global atomic
contention, while preserving separate accounting for Rust allocations.
With callstack tracking disabled, the 256-shard prototype of this change
improved 16-thread allocation/free throughput by roughly 10x for
same-thread frees and 3x for cross-thread frees, using 32 KiB of counter
storage. Single-thread throughput was comparable. These are
microbenchmarks: the gain fell below 1% with roughly 3.5 us of other
work per pair.
Prevent graceful shutdown from hanging indefinitely waiting for
the queue meta maintainer when shutting down while messages are
in flight.
Closes: #654
Add a centralized LRU cache for compiled templates, avoiding repeated
compilation per HTTP injection request. This yields large performance
gains for Handlebars workloads.
Closes: https://github.com/KumoCorp/kumomta/pull/638
Recent github ubuntu image changes broke the rocksdb build
in our docs CI flow, so switch it to our builder image where
rocksdb is pre-built.
Revise the build to use a pre-installed mkdocs so that we don't
have to docker it.
When an egress path has enable_dane set true but the global DANE
isolation switch is off, defer messages with a 451 transient failure
instead of risking silent DANE downgrades on shared ready queues.
Reset a message's delivery attempt count when it moves to an xfer queue,
preventing unsafe backoff calculation overflow that caused panics, premature
expiry, or endless immediate retries.
Reported-by: @edgarsendernet
Closes: #644
handlebars::Context::wraps serializes its argument with
serde_json::to_value, so passing it an existing serde_json::Value
deep-copies the whole substitution context a second time.
Both Handlebars paths already hold an owned serde_json::Value at that
point (render_to_write, and prepare_context which batch injection uses
once per recipient), so take ownership of it with Context::from
instead. The rendered output is unchanged, and the error case that
wraps reported (a serialization failure) cannot occur for a value that
is already JSON.
---------
Co-authored-by: Akshit Sethi <akshits@adobe.com>
Co-authored-by: Wez Furlong <wez@wezfurlong.org>
Fixes the timeout deadlock where calling a memoized function while loading
the policy would hang waiting for cache population. This also fixes the
same deadlock when cache population triggers a policy reload, and retries
population correctly when an in-progress attempt exits early.
All HTTP clients use only the runtime OS trust store, with no compiled-in
root CA bundle. Existing TLS and HTTP behaviour is otherwise unchanged.
This commit was brought to you by the number 4. May your private
keys be sufficiently long, now and into the future.
When enabled (default is disabled), domains with insecure MX selection
but valid DANE TLSA records use a distinct site name. This prevents an
insecure domain from denying DANE authentication to DNSSEC secure
sibling domains on the same MX hosts.
Replace substring checks with exact full-string assertions for MTA-STS deferral
responses. Only the peer's ephemeral port is masked out. No other normalization
is performed. This does not change tested behavior.
When multiple recipient domains share a ready queue, a message for a domain
that publishes MTA-STS enforce will be deferred rather than delivered over
a cleartext connection established by another domain.
MTA-STS and DANE policies can only raise, never lower, the configured
enable_tls level for a destination.
Integration tests and documentation adapted from @terrorobe's PR #616.
closes: https://github.com/KumoCorp/kumomta/pull/616
These were failing in my sandbox environment because the live
DNS zone is too big for UDP responses, and my sandbox's DNS
doesn't handle TCP fallback.
Let's adopt the same fixed upstream pattern that we use for the
DANE/DNSSEC tests so that the live tests stand a better chance
at passing despite the local DNS infra configuration. They are still
prone to flakes if the zone content changes.
https://github.com/hickory-dns/hickory-dns/issues/3974 poses problems
for users that need dnssec validation
This commit points to the current ref from the `release/0.26` branch
for the moment; when hickory makes a new release we'll re-point.
This commit also enables live-dns-tests in the main `make test` flow
to make it more likely to detect issues of this sort in future,
at the cost of potentially more flakes in CI.
Per-thread-pool tokio runtime metrics are now exported for every named
runtime, not just the main runtime. An lruttl_populate_duration histogram
records cache populate time. An integration test verifies both metrics are
exported.
Add kumo.serde.load_files which loads and parses a list of JSON and
TOML files with only one await point, which is more efficient than
the individual calls for folks using the sources helper with large
numbers of source toml files.
Spawn cache populate tasks on the main runtime instead of the caller's
runtime to avoid potential contention with the calling runtime.
For example, if DKIM signing is running in the calling context and
we're trying to run an async heavy workload (serially reading
data from numerous sources) in the populate function, the DKIM
compute heavy work could starve the populate function and prevent
it from making progress.
RFC 3464 section 2.1.2 makes address types case-insensitive, but the
recipient parser only normalizes addresses for an exact lowercase rfc822
match. This leaves angle brackets intact in OOB recipients for RFC822
reports.
Trim the type before comparing it case-insensitively, reuse the existing
SMTP address parser, and cover original and final recipients with tests.
Preserve the reported type spelling and other address types.
closes: https://github.com/KumoCorp/kumomta/pull/640
Since the handshake failure path now drops the connection, recovering the
plaintext stream doesn't serve a purpose. Use plain accept() instead of
into_fallible(). The socket is simply consumed on failure.