Two main cases:
* If the user accidentally places files in the directory that
are not kumo-jsonl compatible segments, then we won't trip
over them and stop processing.
* If the log segments are incomplete (eg: kumod was SIGKILL'd)
then we log the issue and avance to the next segment
Note: need to pin hierarchical_hash_wheel_timer back because the latest
version requires a rustc upgrade, and current versions of mlua don't
build against it when the send feature is enabled.
Some upstream peers (e.g. QQ Mail's rate-limiter) silently hold a
proxied TCP connection open indefinitely — sending no data, FIN, or RST
— rather than cleanly refusing. Without some kind of timeout
management, the two file descriptors for such a session remain open for
the lifetime of the process, slowly exhausting the kernel's
file-descriptor table and occupying proxy-server worker slots.
This commit configures kernel level keepalive options with reasonable
defaults to detect and close out this class of connection.
Co-authored-by: Wez Furlong <wez@wezfurlong.org>
Closes: https://github.com/KumoCorp/kumomta/pull/509
The `psl` crate's domain_str / suffix_str do byte-exact lookups against
the public-suffix list, so uppercase or trailing-dot inputs (e.g.
"Example.COM" or "example.com." from a DNS Name) silently return None.
Several call sites were passing through user- or DNS-supplied domains
unmodified.
Add a small psl-utils crate that wraps the psl crate:
* normalize_domain(&str) -> Cow<str>: strips a single trailing dot and
lowercases ASCII; borrows when the input is already normalized so
the hot path (alignment loop) doesn't allocate.
* domain_str / suffix_str: thin re-exports for callers that have
already normalized.
psl-utils becomes the only workspace crate that directly depends on
the psl crate; kumo-dmarc and mod-string switch to depending on
psl-utils instead.
Updated call sites:
* kumo-dmarc record.rs: is_relaxed_aligned / is_strict_aligned now
normalize both inputs first. Replaces the prior eq_ignore_ascii_case
comparisons.
* kumo-dmarc lib.rs: the organizational-domain fallback in
DmarcContext::check now normalizes from_domain before calling
domain_str and compares against the normalized form. Previously a
mixed-case From: header would skip the _dmarc.<org> lookup entirely.
* mod-string lib.rs: the Lua-exposed string.psl_domain and
string.psl_suffix bindings normalize their input. Behavior change:
inputs that previously returned nil due to case or trailing dot now
resolve.
Co-authored-by: Wez Furlong <wez@wezfurlong.org>
Closes: https://github.com/KumoCorp/kumomta/pull/513
The start of reporting support for DMARC. We create the report from the
errors we encounter, though we do not yet have this wired into the email
reporting system.
Our main task is collecting the errors, though where we put them still
needs to be decided. Currently, a temporary file is used.
Once collected, the errors can be aggregated and built into the RFC 7489-compliant report.
Co-authored-by: Wez Furlong <wez@wezfurlong.org>
Closes: https://github.com/KumoCorp/kumomta/pull/463
This gives us more flexibility in how we can build our parser,
and is significantly easier to maintain.
Part of this change is allowing the command parser to recognize
the starting command verb in an otherwise failed command line
parse; the intent is to provide slightly better error codes
where SMTP defines them when we encounter such a thing.
This commit doesn't do that; it's already pretty huge.
Previously, we'd use a lossy conversion that would replace
funky sequences with the unicode replacement character.
This commit will instead print the bad bytes as hex-escaped characters.
This can be useful when troubleshooting things in tests; this commit
also turns up info logging for the lua test runner so that we can see
that, although nothing else in this commit actually calls that.
I did use it while troubleshooting things just now though, so it is
generally useful.
These are more immediately useful to me for writing tests, but are
generally useful when you have a policy that might need to to deal
with legacy encoding schemes more directly.
This is ostensibly "just" making the nom parsing functions return
BString instead of String, but the fan-out makes the commit much larger,
as a number of surfaces (eg: as_unstructured) have now changed types.
Removes the deref to str as this is now a fallible operation that
can panic.
Process the fanout of this change, which is dealing with more
possibility that things might be binary in more places.
There are some transitional APIs introduced as deprecated so that I'm
nagged to remove them already: I want to get the bulk of the code aware
of the possibility of binary sorted before I tackle adjusting the
message parser for that, so that that particular commit is more
self-contained.
This is a step towards allowing non-UTF-8 data to be tracked
in the SharedString type.
It is not complete; some tests now panic with UTF-8 input where
they would previously implicitly perform a lossy conversion.
That will be resolved in subsequent commit(s).
This dumps out a trace of all tokio tasks. It is quite expensive,
and currently unsettles the tokio runtime such that you need to
repeatedly call this endpoint in order for a subsequent graceful
shutdown to clock through and complete.
In our thundering herd protection we intentionally limit the number of
pending lookups for a given key to just 1. That is controlled via
a semaphore.
We have pre-existing logic that checks to see if the semaphore has been
closed (eg: by a task being cancelled), but despite this, we do
occasionally have reports from users with `timed out after 120s on
semaphore acquire while waiting for cache to populate` for the shaping
data cache.
This typically correlates with eg: long IO waits due to eg: a spike in
transient failures and usually some sort of logging of headers, but not
always.
The not always situation bothers me as we don't currently have a good
explanation of what might be causing the excessive delay.
This commit adds an additional sanity check: when a new lookup is
initiated, if the semaphore is still open (not cancelled), but has been
open for longer than the populate timeout, then we treat this similarly
to the semaphore being closed: we'll create a new semaphore, reset the
expiration and the caller will typically then promote itself to the task
that will satisfy the lookup.
The side effect of this all waiters should become unblocked and awake
either with an error status or with the result of the new lookup, which
should help to clear any persistent/recurring blocking state associated
with this.
This change doesn't provide more insight (I have something in mind for a
follow-on commit for that), but should help make forward progress.
Continuing thread from prior commit, improve the context we
show during shutdown.
Take the opportunity to switch Mutex to parking_lot which is
slightly faster and doesn't have the warty unwrap() API
surface.
If you have two scheduled queues with the same mx_list and then one of
those domains is updated to use a different mx_list, depending on the
ordering of the updates and subsequent traffic, the changed domain may
continue to have traffic route through the old mx_list.
The reason for this is that protocol configuration for that egress path
comes from the scheduled queue configuration, and there is no explicit
dependency between the two pieces of configuration.
This commit resolves this by adding a sanity check that the protocol
information is consistent with the ready queue instance. If it is no
longer consistent we'll remove the association between the rq name and
the instance, then wind down any associated dispatchers.
It was reported that AWS SES could mark the DKIM signature as failed
when using a specific combination of headers in the header list,
with specific lengths of the other fields.
The reason for this is that we had two passes of wrapping applied
to the header, and they might not agree on the formatting of
the header.
The solution is to remove the second pass and just take a bit more
care to emit the header in a wrapped form in the first instance,
that way there can be no discrepancy or conflict.
closes: https://github.com/KumoCorp/kumomta/pull/483
This crate queries machine/system information which is intended
to be used in system monitoring.
This information is not centrally collected or retained by kumocorp,
merely reported via an API endpoint which is accessible only to trusted
IPs.
Machine info tries to interrogate information about the running/hosting
cloud platform for the major cloud providers, as well as indicate
whether it is running in a container of some kind.
The full cloud information is not re-exported via the API endpoint at
this time, but a fingerprint that encodes things like the instance-id is
included.
Harsh Jha
Aditya Ganti
Wez Furlong
kay ozaki
Jack
sophiajt
dependabot[bot]
Aditya Ganti
Tobias Horst