This commit doesn't change any behavior, it just adds an explicit
test for exceeding request_body_limit, and expands on the docs
for request_body_limit to clarify the response when the limit
is exceeded.
Replace the older type with the newer one.
Serialize a copy of auth_info when handling deferred generation,
so that we have a lossless representation of that state when
we eventually process the request. That doesn't change really
anything today, but will enable more granular ACL checks in
the future.
Pass the auth_info through to the http_message_generated and
xfer_message_received events to enable more granular access
control policies to be scripted.
This function sets up the accounting log which records authentication
and authorization events to a local log file.
Add some test assertions that we're not seeing any failed authn/authz
in a number of integration tests.
Add explicit check for failed a request to kcli's streaming metrics
parser, which previously would ignore the request status.
This commit allows the policy to return a richer representation of
the authentication information, which can include multiple identities
and group membership information.
This commit is a step towards some general improvements around
our handling of authentication and authorization.
This commit is focused primarily on authorization, but there are
some adjustments to how we track authentication as part of enabling
that.
We now have a separate AuthInfo type that holds the overall
authentication information/context associated with an inbound
SMTP or HTTP session.
It is populated with the peer_address as a fact rather than a statement
of trust.
If authentication via the appropriate lua auth callback is successful,
then the AuthInfo has additional identities added.
There are some types and events for loading access control lists and
matching their rules against an AuthInfo.
There is now a system default ACL that is equivalent to the prior
hard-coded access policy that was encoded into each HTTP endpoint.
This change makes it possible to replace the ACL with a
user-defined ACL.
Later will be some work on authentication to allow more options
for HTTP auth.
* Add AWS SigV4 signature module
- Implements complete AWS Signature Version 4 algorithm
- Supports all AWS services (S3, SNS, SQS, Kinesis Firehose, etc.)
- Includes SHA256 hashing and HMAC-SHA256 signing
- Secure key management via KeySource (files, Vault, inline)
- Session token support for temporary credentials
- Comprehensive documentation and examples
- 5 unit tests and 5 integration tests (all passing)
Exposes kumo.aws.sign_v4() function to Lua for signing AWS API
requests. This enables KumoMTA to integrate with AWS services for
streaming logs to Kinesis Firehose, storing data in S3, sending
notifications via SNS, and more.
Tested with:
- S3 GET/PUT requests
- SNS POST requests
- SQS with query parameters
- Kinesis Firehose PutRecord
updating README.md file
Co-authored-by: Aditya Ganti <adityaganti@mac.mynetworksettings.com>
These functions are similar to the existing
invoke_get_egress_path_config and invoke_get_queue_config functions;
they return the configuration information that is defined by your
overall policy in the same way that the system does.
The intent is for more advanced policies that need that info to
decide how to configure other parts of the policy.
Docs are included, and I fleshed out the missing docs for the two
pre-existing functions.
Saw an MX record that expands to 0.0.0.0 in the wild.
As well as being nonsense, that causes a mail loop, so we should add it
to prohibited_hosts.
I noticed that we also did not document its default value, so the
doc update here shows the before and after for this.
This engine does no templating; the source text is output exactly
as it was entered into the template engine.
Expose template_dialect to the http injection API and add a test
to show that selecting the Static dialect doesn't perform any
template expansion.
refs: https://github.com/KumoCorp/kumomta/issues/446
Continuing in the same vein as 94628f06ab,
ensure that we have a reasonable outcome when a batch of size > 1
encounters 100% ambiguous 452 responses.
Without special consideration we'd end up in an infinite retry loop for
these, as we'd think that we'd hit the recpient limit on the connection
every time.
This commit introduces a small amount of state into the dispatcher so
that we can remember the set of (spoolid, recipient) from the prior
transaction on the session.
When deciding whether something is eligible for immediate retry
we will gate that choice on the current (spoolid, recipient) being
absent from the prior set of ids. In this way, we bound the immediate
retry for a recipient to one per connection.
To make things a bit more robust when trying to decide whether a 452
means "too many recipients" or not, if we're not sure, we now call out
to the bounce classifier which can be configured to emit a
`TooManyRecipients` classification. If the classifier produces either
that or some non-Uncategorized response, then we have a confident
assessment. Otherwise, we're unsure and will factor that into the
retry strategy.
This aggregates the various authentication-results producing auth checks
into a more convenient function.
Part of this change improves some plumbing in the dkim checking (we no
longer raise an error for a missing From, but instead indicate a failed
dkim result), and we now support passing down a resolver name to the
various checking functions, to facilitate testing and other more
advanced use cases.
refs: https://github.com/KumoCorp/kumomta/issues/16
refs: https://github.com/KumoCorp/kumomta/issues/84
This commit adds some plumbing to allow us to collect metadata
about the caches and spit it out into a json file that we can
then use in the docs to show the predefined caches, capacities
and brief comments about their purpose.
We put that info on the set_lruttl_cache_capacity doc page.
Augment the docs to clarify that msg:append_header and
msg:prepend_header do no modification of the value by default.
Adjust the wrapping used by Header:new_unstructured so that it
hard wraps before the SMTP line length limit.
Add optional encode parameter to msg:append_header and
msg:prepend_header so that those methods can opt-in to the
Header:new_unstructured formatting.
Add HeaderMap:append method to mirror the existing HeaderMap:prepend
method.
Add unit tests that demonstrate the effect of wrapping/encoding
for both the "dumb" msg:append_header and msg:prepend_header functions,
and the "smarter" HeaderMap:prepend and HeaderMap:append methods that
always encode their parameters.
Update the docs for the headermap flavor of these methods to indicate
that they will always encode the value.
This allows a (more or less) arbitrary lua function to be used to load
data.
The primary advantage of this is that the size of the resulting
KeySource struct is smaller than it would be to hold the actual data
inline. It also enables the surrounding code to be slightly better
factored.
Add CBC and ecb Block mode aes encryption/decryption.
To facilitate testing this better, and just to be nicer overall, adjust
data-loader's KeySource::key_data variant to support loading raw binary
bytes.
Co-authored-by: Wez Furlong <wez@wezfurlong.org>
Closes: https://github.com/KumoCorp/kumomta/pull/395
Add a convenience function that is intended to be used around the
time that you might be building up the overall set of
Authentication-Results.
Populate missing smtp.mailfrom in SPF authentication result.
We were including the CRLF in the measurement that we were making,
while the default value was the value from the spec excluding the CRLF.
Let's increase the default value to 1000 to make things match up.
This commit does some plumbing work to allow defining alternative
resolvers with different names that can be used to query different
upstream DNS servers.
This is most useful when implement DNSBLs where you might have a custom
DNS server configured with one or more RBL zones that is reserved purely
for RBL lookups.
The plumbing introduces a more regular syntax for defining one of the
various implementations of the Resolver trait, as well as a new
Aggregate resolver impl that can query across multiple Resolvers until a
query is satisfied.
This allows some interesting and powerful configurations, such as
loading a static zone file into memory to query it directly, and/or
blending that together with querying either the system or some other
upstream DNS server as a fallback.
The various lookup functions (except for MX!) have been updated to
accept an optional alternate resolver name, so that they work together
with the above.
A new rbl_lookup function is also provided as a convenience for querying
the most common form of RBLS.
ptr_host and reverse_ip are two string utility functions that are likely
not going to be widely used, but are very convenient to have when you
do have a usecase that requires it!
closes: https://github.com/KumoCorp/kumomta/issues/269
This never worked as well as the other amqp option that we support,
and upgrading it to the current version is a bit of a chore.
I'd rather just drop this dep and keep things more trim.
Since none of our sponsors are using the lapin implementation, let's
mark it as deprecated and we can take it out after we cut the next
stable release.
A "casualty" of this is that the client-ip crate, which we were using
for its InsecureClientIp type for informational purposes, has retired
the InsecureClientIp type because it wasn't trustworthy overall
(https://github.com/imbolc/axum-client-ip/issues/32).
What we do for the time being is configure that crate to use the direct
peer IP. There isn't a way to automagically select the "best"
information available without building in some kind of middleware, and
we probably should make that configurable in order to be fully
trustworthy.
So for now, we're punting on that until someone shows up with some
requirements and sponsorship and we'll make it happen then.
This commit implements a kumomta-specific message transfer
protocol that is intended to be used to migrate messages
from one kumomta node to another.
The transfer is carried out using an HTTP POST request
to the destination node's http listener.
The request includes the full message metadata and body,
in a compressed form.
An xfer request can be made via `kcli xfer` (and thus also via an HTTP API
endpoint). It works similarly to a rebind operation; you specify the
criteria to be used to match scheduled queues, along with the target
node for the xfer, and kumomta will find matching queues, drain out the
messages, make an adjustment to the metadata to capture current
scheduling information, and then place the messages into an xfer queue.
The xfer queue has hard-coded scheduling queue configuration of its own,
with the base retry interval set to 10 seconds, which should be suitably
aggressive for the intended use case.
You may apply shaping to affect the number of concurrent requests in a
similar way to how TSA shaping is configured.
On the receiving side, the incoming xfer sanity check to prohibit
trying to xfer to itself.
The spool id of the Message is not suitable to be reused verbatim on
another node (spool ids include the local mac address and creation
timestamp information, as well as a random component), so the receiving
side will derive an id that should be suitable for use on that node.
The originating node id and spool id will be preserved in metadata to
aid in tracing.
It is possible for an xfer request to target an existing xfer queue, so
that you can correct/update the target in various circumstances. In that
situation the messages will be "simply" moved from the source queue to
the destination queue.
It is possible to cancel an xfer request via `kcli xfer-cancel` (and
thus also via an HTTP API endpoint). You specify the target queue,
which must be an xfer queue, and it will have its messages drained and
the metadata changes that were applied when the xfer was initiated will
be reversed, allowing the messages to then be reinserted into their
originating queue.
refs: https://github.com/KumoCorp/kumomta/issues/311
Wez Furlong
Mike Hillyer
Aditya Ganti
Dario Maiocchi