Note: need to pin hierarchical_hash_wheel_timer back because the latest
version requires a rustc upgrade, and current versions of mlua don't
build against it when the send feature is enabled.
Some upstream peers (e.g. QQ Mail's rate-limiter) silently hold a
proxied TCP connection open indefinitely — sending no data, FIN, or RST
— rather than cleanly refusing. Without some kind of timeout
management, the two file descriptors for such a session remain open for
the lifetime of the process, slowly exhausting the kernel's
file-descriptor table and occupying proxy-server worker slots.
This commit configures kernel level keepalive options with reasonable
defaults to detect and close out this class of connection.
Co-authored-by: Wez Furlong <wez@wezfurlong.org>
Closes: https://github.com/KumoCorp/kumomta/pull/509
The `psl` crate's domain_str / suffix_str do byte-exact lookups against
the public-suffix list, so uppercase or trailing-dot inputs (e.g.
"Example.COM" or "example.com." from a DNS Name) silently return None.
Several call sites were passing through user- or DNS-supplied domains
unmodified.
Add a small psl-utils crate that wraps the psl crate:
* normalize_domain(&str) -> Cow<str>: strips a single trailing dot and
lowercases ASCII; borrows when the input is already normalized so
the hot path (alignment loop) doesn't allocate.
* domain_str / suffix_str: thin re-exports for callers that have
already normalized.
psl-utils becomes the only workspace crate that directly depends on
the psl crate; kumo-dmarc and mod-string switch to depending on
psl-utils instead.
Updated call sites:
* kumo-dmarc record.rs: is_relaxed_aligned / is_strict_aligned now
normalize both inputs first. Replaces the prior eq_ignore_ascii_case
comparisons.
* kumo-dmarc lib.rs: the organizational-domain fallback in
DmarcContext::check now normalizes from_domain before calling
domain_str and compares against the normalized form. Previously a
mixed-case From: header would skip the _dmarc.<org> lookup entirely.
* mod-string lib.rs: the Lua-exposed string.psl_domain and
string.psl_suffix bindings normalize their input. Behavior change:
inputs that previously returned nil due to case or trailing dot now
resolve.
Co-authored-by: Wez Furlong <wez@wezfurlong.org>
Closes: https://github.com/KumoCorp/kumomta/pull/513
The start of reporting support for DMARC. We create the report from the
errors we encounter, though we do not yet have this wired into the email
reporting system.
Our main task is collecting the errors, though where we put them still
needs to be decided. Currently, a temporary file is used.
Once collected, the errors can be aggregated and built into the RFC 7489-compliant report.
Co-authored-by: Wez Furlong <wez@wezfurlong.org>
Closes: https://github.com/KumoCorp/kumomta/pull/463
This is a step towards allowing non-UTF-8 data to be tracked
in the SharedString type.
It is not complete; some tests now panic with UTF-8 input where
they would previously implicitly perform a lossy conversion.
That will be resolved in subsequent commit(s).
This crate queries machine/system information which is intended
to be used in system monitoring.
This information is not centrally collected or retained by kumocorp,
merely reported via an API endpoint which is accessible only to trusted
IPs.
Machine info tries to interrogate information about the running/hosting
cloud platform for the major cloud providers, as well as indicate
whether it is running in a container of some kind.
The full cloud information is not re-exported via the API endpoint at
this time, but a fingerprint that encodes things like the instance-id is
included.
We currently use a rather hacky embedding of rapidoc to provide a
generic browser around the jsonschema export from our API interface.
It's not great for a couple of reasons:
* The font sizes are tiny
* The documentation rapidoc produces is not indexable, being
generated by javascript when the browser loads. This also
prevents making proper links to the various doc pages
This commit introduces a little utility that we can use during
the doc build to translate the schema into documentation files
that can then be processed as normal by the build.
This commit does this just for kumod at this time, but we could
also add tsa daemon in the future if we expand its API surface.
Move TLS configuration and async stream traits from rfc5321
into a new shared kumo-tls-helper crate for reuse in
proxy-server and other crates.
Refs #451
We found an interesting issue where a MIME part was labelled as `text/`
with no explicit charset (and was thus implicitly "us-ascii", according
to the specs). The content was actually UTF-8 encoded farsi text.
Since encoding_rs treats us-ascii as being an alias for windows-1252,
that part was re-encoded as bogus data, rather than being recognized as
UTF-8.
This commit improves the outcome in this situation:
1. Remove encoding_rs. Replace its Charset type with the Encoding type
that I added to the charset-normalizer crate in
https://github.com/nickspring/charset-normalizer-rs/pull/45
which allows us to actually differentiate between 7-bit ascii
and latin-1.
2. Improve the handling of body part extraction; there were a couple
of cases where we didn't consider the charset for text parts(!)
and now we do, with some additional smarts for when the charset
fails to decode.
3. Improve the conformance checking. We do the majority of this
during parsing, but we don't check for deeper issues such as
the charset not matching the transfer-decoded payload because
that is too costly to do for the majoriy of parses. This commit
introduces a deep conformance check that does validate that we
can extract the part, flagging any failures as
NEEDS_TRANSFER_ENCODING.
4. The charset detection options are threaded through the conformance
checks and into the logic that extracts the parts during a rebuild,
so that we can fix up those parts during rebuild.
The bulk of the conformance checking/fixing has been moved into the
mailparsing crate from the message crate, which makes the above a bit
easier and centralizes that logic better--fewer places to look to figure
things out.
* Add AWS SigV4 signature module
- Implements complete AWS Signature Version 4 algorithm
- Supports all AWS services (S3, SNS, SQS, Kinesis Firehose, etc.)
- Includes SHA256 hashing and HMAC-SHA256 signing
- Secure key management via KeySource (files, Vault, inline)
- Session token support for temporary credentials
- Comprehensive documentation and examples
- 5 unit tests and 5 integration tests (all passing)
Exposes kumo.aws.sign_v4() function to Lua for signing AWS API
requests. This enables KumoMTA to integrate with AWS services for
streaming logs to Kinesis Firehose, storing data in S3, sending
notifications via SNS, and more.
Tested with:
- S3 GET/PUT requests
- SNS POST requests
- SQS with query parameters
- Kinesis Firehose PutRecord
updating README.md file
Co-authored-by: Aditya Ganti <adityaganti@mac.mynetworksettings.com>
This is primarily a refactoring of kumo-template to conceptually
support the idea of having multiple dialects, even though it
only defines a single dialect.
It prepares the types/interface for that purpose.
It also pulls in the handlebars crate, but doesn't use anything
from it yet; that'll be in a follow up commit.
refs: https://github.com/KumoCorp/kumomta/issues/446
Add CBC and ecb Block mode aes encryption/decryption.
To facilitate testing this better, and just to be nicer overall, adjust
data-loader's KeySource::key_data variant to support loading raw binary
bytes.
Co-authored-by: Wez Furlong <wez@wezfurlong.org>
Closes: https://github.com/KumoCorp/kumomta/pull/395
There's some kind of bad linker issue in CI that will take a little
bit to run down, and the update wasn't critical, so let's rever it
for the time being.
A "casualty" of this is that the client-ip crate, which we were using
for its InsecureClientIp type for informational purposes, has retired
the InsecureClientIp type because it wasn't trustworthy overall
(https://github.com/imbolc/axum-client-ip/issues/32).
What we do for the time being is configure that crate to use the direct
peer IP. There isn't a way to automagically select the "best"
information available without building in some kind of middleware, and
we probably should make that configurable in order to be fully
trustworthy.
So for now, we're punting on that until someone shows up with some
requirements and sponsorship and we'll make it happen then.
kay ozakiandWez Furlong
JackandWez Furlong
sophiajtandWez Furlong
Tobias HorstandWez Furlong
Kevin Vu
Aditya GantiandAditya Ganti
Dario MaiocchiandWez Furlong