# Release 2024.09.02-c5476b89

## Breaking Changes
* None

## Other Changes and Enhancements

* [Pattern matching rollups](../userguide/trafficshaping/rollups.md#pattern-matching-rollups-using-providers) have been introduced in the shaping helper to allow for the definition of Providers, collecting together multiple site names when a given MBP has inconsistent MX patterns.
* The [kumomta-dev container
  image](https://github.com/KumoCorp/kumomta/pkgs/container/kumomta-dev) is now
  a multiarch image, supporting both `linux/amd64` and `linux/arm64`
  architectures.  Simply use `docker pull ghcr.io/kumocorp/kumomta-dev:latest`
  to get the appropriate architecture.
* Split out the banner waiting portion of
  [connect_timeout](../reference/kumo/make_egress_path/connect_timeout.md) into
  a new [banner_timeout](../reference/kumo/make_egress_path/banner_timeout.md)
  option to make it easier to manage the system behavior if, for example, the
  connection is blocked by a firewall. You can now set the connection timeout
  to a smaller value while keeping the banner timeout at a more RFC-compliant,
  longer, value. #196
* New [kcli trace-smtp-client](../reference/kcli/trace-smtp-client.md)
  diagnostic command for observing outbound SMTP sessions. #87
* New [Extended configuration validation mode](../userguide/configuration/policy_helpers.md#validating-your-configuration). #211
* New [kumo.regex](../reference/regex/index.md) and
  [kumo.string](../reference/string/index.md) lua modules. #220
* New `kcli rebind` and
  [/api/admin/rebind/v1](../reference/http/kumod/api_admin_rebind_v1_post.md) HTTP
  endpoint to allow moving/rebinding messages from one scheduled queue to
  another. There is an optional corresponding
  [rebind_message](../reference/events/rebind_message.md) event for more
  advanced rebinding logic. #209
* Moved JSON and TOML functions into a new
  [kumo.serde](../reference/kumo.serde/index.md) module. Those functions are
  also still available under the `kumo` module for backwards compatibility
  sake, but will be removed in a future release. You should standardize on the
  new `kumo.serde` module name moving forwards.
* Added YAML serialization/deserialization functions to
  [kumo.serde](../reference/kumo.serde/index.md).
* You may now run `kumod --validate` to perform extended validation checks
  of the helper configuration in your policy. This can be performed offline/concurrently
  with a running kumod. The output is human readable. The exit code will
  be 0 when no validation errors are detected, non-zero otherwise. #211
* `Rejection` log records now capture the triggering incoming SMTP command
  line, or, in the case of the `technical difficulties` error message,
  the stack trace of the triggering issue.  This information is available
  in `log_record.response.command`.
* Add explicit `close` method to AMQP, HTTP, Kafka and SQLite client objects.
* Added UUID functions in a new [kumo.uuid](../reference/kumo.uuid/index.md)
  module.
* SMTP client log records now include the `source_address` in addition to
  the `egress_pool` and `egress_source` that were already present in the
  record. `source_address` includes proxy server information. #40 #154
* Improved diagnostics around connectivity issues in KumoProxy
* New TLS related options for
  [kumo.make_egress_path](../reference/kumo/make_egress_path/index.md):
  `tls_prefer_openssl`, `openssl_cipher_list`, `openssl_cipher_suites`,
  `openssl_options`, `rustls_cipher_suites`.
* Improved granularity of scheduled queue maintainer wakeups when small
  a `retry_interval` is configured. Previously this was fixed at 1 minute,
  but now it will scale to smaller values.
* proxy-server: added `--no-splice` option to opt out of using `splice(2)`
  on Linux. Switch to using `tokio::io::copy_bidirectional` for non-splice mode.
  Switch to using `tokio_splice::zero_copy_bidirectional` for splice mode.
* Add `/api/check-liveness/v1` API endpoint to determine whether the KumoMTA 
  node is live and ready to receive messages.
* Updated redis crate to 0.26, which improves connection management and pooling.
* Reduced latency during shutdown. Connections that are being attempted to
  unreachable destinations no longer block shutdown until they timeout, and
  are instead terminated immediately.
* Parallelize DNS lookups to improve latency when validating shaping configuration
* TSA log hook now has an option to match rules on the client side before
  deciding to send a record to the daemon, reducing IO pressure on the spool
  and bandwidth between the MTA and the TSA daemon, at the cost of some CPU
  utilization. This defaults to enabled but can be disabled by setting
  `pre_filter = false` in the call to `setup_with_automation`.
* New
  [max_connections](../reference/kumo/start_esmtp_listener/max_connections.md)
  option to set a limit on the number of concurrent incoming connections
  permitted to a given listener. This defaults to `32768`. A new
  `total_connections_denied` counter is increment whenever this limit is hit,
  and whenever a connection is rejected due to being over the memory limit.
* HTTP requests made from a trusted host that set an Authorization header will
  now take the authorization information from the header. Previously the
  authentication handling would stop as soon as we recognized the trusted
  source IP. Either way, the request always did and will still continue
  (assuming that the auth header is valid).
* Improved the performance of the bounce classifier by exposing configurable
  thread pool and caching options.
* [configure_local_logs](../reference/kumo/configure_local_logs/index.md) and
  [define_spool](../reference/kumo/define_spool/index.md) now both allow specifying
  the minimum storage space and inodes required for healthy operation. The
  usage will be periodically monitored and the system will begin rejecting new
  messages when the available storage falls below the minimum, until it
  recovers. Usage is exported via prometheus metrics. The default minimum level
  is `10%`.
* Scheduled queue configuration refreshes are now performed sequentially by
  a dedicated configuration refresh task. In prior releases they could be
  performed concurrently by multiple tasks running in the qmaint pool.
  This change reduces overhead from speculative processing of config changes,
  especially when there are very many scheduled queues.

## Fixes

* Using `expiration` in a DKIM signer would unconditionally raise an error and
  prevent reception of the incoming message.
* Invalid structured headers, such as Message-ID, in combination with other message
  body conformance issues could cause
  [msg:check_fix_conformance](../reference/message/check_fix_conformance.md) to
  raise an error instead of fixing the issue. #216
* Swapped retry-after/reset-after results, and increased timestamp precision
  when using [cluster-backed
  throttles](../reference/kumo/configure_redis_throttles.md). Thanks to @cai-n!
  #217
* Didn't expand the requested SASL mech when responding `504 5.5.4 AUTH {sasl_mech}
  not supported` to unsupported authentication mechanisms.
* Delayed messages were scheduled 1-step further along the exponental backoff schedule
  than intended; rather than being initially delayed for 20 minutes, they would be
  delayed for 40 minutes.
* MTA-STS: failed to load the policy for some sites due to a redirect caused
  by a trailing dot in the policy domain
* redis cluster: `could not acquire connection lease: An error was signalled by
  the server` with some redis cluster deployments when using `kumo.configure_redis_throttles`.
* bounce classifier: when multiple rules can match the same input, the
  resultant classification is now "won" by the rule that was loaded first from
  the earliest containing file. Previously, the order would be perturbed by the
  name of the classification, preferring to match in alphabetical order of the
  classification name. So if you had an "A" classification and a "B"
  classification with the same rule, the "A" classification would be the
  result, even if the "B" rule was the first one listed in your classification
  data file(s).
* Changing the `max_ready` value for a ready queue no longer requires waiting for
  the queue to be reaped before it will take effect.
* When connecting to single-node redis, the username and password were ignored.
* `max_segment_duration` for log files was only processed in idle periods.
* log hooks only partially respected the `back_pressure` parameter, which could
  leave to more system pressure (higher RAM and larger number of concurrent
  tasks) in overload scenarios.
* Improved Content-ID header parsing conformance in
  [msg:check_fix_conformance](../reference/message/check_fix_conformance.md). #259
