Replace the older type with the newer one.
Serialize a copy of auth_info when handling deferred generation,
so that we have a lossless representation of that state when
we eventually process the request. That doesn't change really
anything today, but will enable more granular ACL checks in
the future.
Pass the auth_info through to the http_message_generated and
xfer_message_received events to enable more granular access
control policies to be scripted.
This function sets up the accounting log which records authentication
and authorization events to a local log file.
Add some test assertions that we're not seeing any failed authn/authz
in a number of integration tests.
Add explicit check for failed a request to kcli's streaming metrics
parser, which previously would ignore the request status.
This commit allows the policy to return a richer representation of
the authentication information, which can include multiple identities
and group membership information.
This commit is a step towards some general improvements around
our handling of authentication and authorization.
This commit is focused primarily on authorization, but there are
some adjustments to how we track authentication as part of enabling
that.
We now have a separate AuthInfo type that holds the overall
authentication information/context associated with an inbound
SMTP or HTTP session.
It is populated with the peer_address as a fact rather than a statement
of trust.
If authentication via the appropriate lua auth callback is successful,
then the AuthInfo has additional identities added.
There are some types and events for loading access control lists and
matching their rules against an AuthInfo.
There is now a system default ACL that is equivalent to the prior
hard-coded access policy that was encoded into each HTTP endpoint.
This change makes it possible to replace the ACL with a
user-defined ACL.
Later will be some work on authentication to allow more options
for HTTP auth.
Previously we'd ignore the HTTP status code and try to parse the body
as TOML regardless.
This commit raises an explicit error with the HTTP status code and
response body before trying to parse as TOML.
Centralize the logic of walking and calling the registered handlers
so that we don't have 3 or more slightly different versions of it.
Introduce a CallbackDisposition type that can be used to drive
the default handling with more nuance; this will be useful
in a later commit where we need to distinguish between an
explicit nil being returned by a handler, and no handler
being defined.
We found an interesting issue where a MIME part was labelled as `text/`
with no explicit charset (and was thus implicitly "us-ascii", according
to the specs). The content was actually UTF-8 encoded farsi text.
Since encoding_rs treats us-ascii as being an alias for windows-1252,
that part was re-encoded as bogus data, rather than being recognized as
UTF-8.
This commit improves the outcome in this situation:
1. Remove encoding_rs. Replace its Charset type with the Encoding type
that I added to the charset-normalizer crate in
https://github.com/nickspring/charset-normalizer-rs/pull/45
which allows us to actually differentiate between 7-bit ascii
and latin-1.
2. Improve the handling of body part extraction; there were a couple
of cases where we didn't consider the charset for text parts(!)
and now we do, with some additional smarts for when the charset
fails to decode.
3. Improve the conformance checking. We do the majority of this
during parsing, but we don't check for deeper issues such as
the charset not matching the transfer-decoded payload because
that is too costly to do for the majoriy of parses. This commit
introduces a deep conformance check that does validate that we
can extract the part, flagging any failures as
NEEDS_TRANSFER_ENCODING.
4. The charset detection options are threaded through the conformance
checks and into the logic that extracts the parts during a rebuild,
so that we can fix up those parts during rebuild.
The bulk of the conformance checking/fixing has been moved into the
mailparsing crate from the message crate, which makes the above a bit
easier and centralizes that logic better--fewer places to look to figure
things out.
we were producing output like this when `prefix=kumomta_`:
```
# HELP kumomta_total_readyq_runs total number of times a readyq maintainer was run
# TYPE kumomta_total_readyq_runs counter
kumomta_total_readyq_runs 17
# HELP kumomta_connection_count number of active connections
# TYPE kumomta_connection_count gauge
connection_count{service="smtp_client"} 0
connection_count{service="esmtp_listener"} 0
connection_count{service="lua"} 0
```
Those `connection_count` lines should be `kumomta_connection_count`.
* Add AWS SigV4 signature module
- Implements complete AWS Signature Version 4 algorithm
- Supports all AWS services (S3, SNS, SQS, Kinesis Firehose, etc.)
- Includes SHA256 hashing and HMAC-SHA256 signing
- Secure key management via KeySource (files, Vault, inline)
- Session token support for temporary credentials
- Comprehensive documentation and examples
- 5 unit tests and 5 integration tests (all passing)
Exposes kumo.aws.sign_v4() function to Lua for signing AWS API
requests. This enables KumoMTA to integrate with AWS services for
streaming logs to Kinesis Firehose, storing data in S3, sending
notifications via SNS, and more.
Tested with:
- S3 GET/PUT requests
- SNS POST requests
- SQS with query parameters
- Kinesis Firehose PutRecord
updating README.md file
Co-authored-by: Aditya Ganti <adityaganti@mac.mynetworksettings.com>
It doesn't like the full sccache build due to some wonkyness
with openss on this old distro.
Let's just pave over this by not enabling the full build
on that platform.
At some point turning off most features used to make things work in our
environments, but looking at things today, I think it basically makes
it do nothing useful any more.
Turning things off just made the image build a hair faster and the
image a hair lighter.
So let's just turn it back on
In reviewing some live logs, I could see that some timeout errors
were including the debug repr of the SMTP command, which is too verbose
to be useful in the error log line, so let's tidy that up a bit
here to make it easier to understand.
These functions are similar to the existing
invoke_get_egress_path_config and invoke_get_queue_config functions;
they return the configuration information that is defined by your
overall policy in the same way that the system does.
The intent is for more advanced policies that need that info to
decide how to configure other parts of the policy.
Docs are included, and I fleshed out the missing docs for the two
pre-existing functions.
When using an explicit mx_list, rather than consuming the DNS-provided
MX records, we were populating the connection plan LIFO with the mx_list
which is specified as a FIFO.
The result was that we would connect to the addresses in the reverse of
the intended order.
Most users only populate a single entry for mx_list, if they use this at
all, so it has flown under the radar. This issue was introduced
together with the mx_list feature in bcd2946c53.
This commit corrects that issue so that the behavior conforms to the
intended and documented ordering, and tweaks the tests to accommodate
it; the test changes required were:
1. disconnect_in_data was actually sensitive to the order, and encoded
the reverse ordering in its expectations. :-/
It now correctly encodes the correct order, which should allow
detection of regression in the future.
2. disconnect_terminate_ok is sensitive to extraneous TransientFailures.
It was implicitly dependent on the ordering of the mx_list in the
sense that it relied on the FIFO list returning as [127.0.0.1, ::1],
which resulted in trying ::1 first (because that list is consumed as
LIFO), failing due to ::1 not actually being bound in the test
environment, and proceeding to test with 127.0.0.1 being the last
working address.
With the ordering fixed that test started to fail because the second
host in the correct connection plan was ::1 and that was not
reachable, triggering a TransientFailure against the expectations
of the test.
The goal of the test was to verify the behavior in the
exhausted-connection-plan case, rather than the
unreachable-final-mx-case, so the resolution here is to add v6
addresses to the skip_hosts config. Will that come back to haunt us?
Time will tell.
For a sequence like `[BASE64 IP: IPADDR]` we were not previously
able to see the base64 or IP address inside the square brackets.
This commit adds a pre-processing step to strip out parentheticals
surrounded by spaces (or the end of the string) so that our existing
tokenization can take effect.
I spent a few hours today trying a few different matching strategies,
rewriting this crate 3 different ways to try to simplify the logic and
lean more heavily on the regex or fancy_regex crate so that we can avoid
special cases. While they all worked, they all were over an order of
magnitude worse in terms of throughput and performance.
The strategy here is to perform a very cheap SIMD-assisted test to see
if we need to pre-process the string so that we can avoid the overhead
of the additional string allocation and full processing step.
The result is that the performance before and after this change is very
close to neutral when the response contains no brackets, but 33% slower
(and 50% throughput) when brackets are present. That's still 180MiB/s
on my threadripper which is still decent for this application.
While grubbing around in here today, I noticed that a couple of regexeps
had typos: `(:?` when they should have been `(?:`, and I annotated more
sub-expressions to be non-capturing as well.
This commit tidies up a bit of an API wart that stems from
the initial implementation where we couldn't guarantee that
the message crate would know how to get a spool handle to
implicitly load data when needed.
This has resulted in a couple of issues over time where we try to
operate on the message data and it just hasn't been loaded, leading to
transient errors being raised in the best case, but in a possible worst
case, a silent error condition.
During the rebind_message event, if the message is not loaded,
msg:get_data() could return the placeholder empty data value. If the
rebind_message event is trying to mutate the message then the best case
is that an error is raised, a medium-bad case is that operations like
`msg:append_header` will fail to find the header block and thus fail,
and the worst case is that mutation operations that blinding prepend
data (perhaps msg:prepend_header) could end up assigning a bogus
truncated message. While we don't generally recommend this sort of
action, this is potentially a very gnarly consequence of trying things
out.
Let's just head all of these sorts of issues off by making the accessors
async and have them internally load the data as required.
Wez Furlong
Daniel Smith
Aditya GantiandAditya Ganti
Mike Hillyer
crxandWez Furlong
