Commit Graph

3721 Commits

Author SHA1 Message Date
Mike Hillyer 164f109326 Traffic Shaping Refactor First Pass 2026-06-09 13:23:56 -04:00
Wez Furlong 4eedf9361e fix punycode normalization for STARTTLS
I believe this to be a relatively recent regression since we started
to preserve the incoming domain name in RCPT TO, rather than
normalizing it. (2026.04.09-ea3b2a9b)

refs: https://github.com/KumoCorp/kumomta/issues/533
2026-05-30 17:35:28 +01:00
Harsh Jha 00f3131f7d Allow configuring which records TSA skips
Closes: https://github.com/KumoCorp/kumomta/pull/525

Co-authored-by: Wez Furlong <wez@wezfurlong.org>
2026-05-29 15:02:31 +01:00
Wez Furlong c7750deada fix flakey mx_list_refresh test 2026-05-29 11:07:19 +01:00
Aditya Ganti 7bfbbe720a fix(mod-aws-sigv4): apply Trimall, require host header, fix S3 content-sha256
Co-authored-by: Wez Furlong <wez@wezfurlong.org>
Closes: https://github.com/KumoCorp/kumomta/pull/522
2026-05-29 10:16:06 +01:00
Wez Furlong 5fb9dd104d fix regression in serializing original_message in ARF
This got overlooked around the time that 7518ca6de was pushed.
The intent was to serialize as a string when possible, bytes otherwise.
2026-05-29 09:45:22 +01:00
Wez Furlong 06a26f4ba6 kumo-jsonl: make tailer more tolerant of errors
Two main cases:

 * If the user accidentally places files in the directory that
   are not kumo-jsonl compatible segments, then we won't trip
   over them and stop processing.

 * If the log segments are incomplete (eg: kumod was SIGKILL'd)
   then we log the issue and avance to the next segment
2026-05-29 08:57:47 +01:00
Wez Furlong 5a7aa96cf1 docs: update for 2026.05.12-a6845223 release 2026-05-27 06:47:55 +01:00
Wez Furlong a684522394 rustfmt 2026-05-12 16:50:01 +01:00
Wez Furlong 62de291279 docs: suggest folks prefix their metadata names
Should help to avoid possible conflicts with the core metadata names.
2026-05-12 14:35:16 +01:00
Harsh Jha 7cba56bc06 http injection: allow static per-recipient metadata
That metadata is accessible in to via msg:get_meta('extra')

closes: https://github.com/KumoCorp/kumomta/pull/516
Co-authored-by: Wez Furlong <wez@wezfurlong.org>
2026-05-12 14:25:46 +01:00
kay ozaki 1febfcc55e lua counter series using kumo-counter-series
Co-authored-by: Wez Furlong <wez@wezfurlong.org>
closes: https://github.com/KumoCorp/kumomta/pull/507
2026-05-07 15:35:17 +01:00
Wez Furlong 2c0e9982e1 docs: fmt examples in import_headers.md 2026-05-07 14:25:34 +01:00
Wez Furlong 421bff1a41 docs: changelog for #504 2026-05-07 14:21:01 +01:00
Oleg Orekhov c237e16c5f mod-http: allow disabling certificate verification on the client
closes: https://github.com/KumoCorp/kumomta/pull/504
2026-05-07 14:18:09 +01:00
Wez Furlong fcb0981c31 deps: update rust-openssl 2026-05-06 22:50:49 +01:00
Wez Furlong 421b8a289e proxy: keepalive will work on macos and other unix too
So remove the linux only gate
2026-05-06 22:38:50 +01:00
dependabot[bot] d7781ca588 build(deps): bump mozilla-actions/sccache-action from 0.0.9 to 0.0.10 (#514)
Bumps [mozilla-actions/sccache-action](https://github.com/mozilla-actions/sccache-action) from 0.0.9 to 0.0.10.
- [Release notes](https://github.com/mozilla-actions/sccache-action/releases)
- [Commits](https://github.com/mozilla-actions/sccache-action/compare/v0.0.9...v0.0.10)

---
updated-dependencies:
- dependency-name: mozilla-actions/sccache-action
  dependency-version: 0.0.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-06 21:56:42 +01:00
Wez Furlong ab4a47a016 ci: docker: only tag :latest on release tag pushes
The docker/metadata-action config relied on the default flavor
(latest=auto), which adds :latest when the ref is the default branch
or a SemVer git tag. Our release tags are date-based (e.g.
2026.03.04-bb93ecb1) and are not SemVer, so the SemVer rule never
fired and every push to main re-pointed :latest at the dev build.
The inline "produce latest tag when tagging" comment on
type=ref,event=tag was misleading — that entry only emits a tag
matching the git tag name, it does not produce :latest.

Set flavor: latest=false to disable the implicit behavior, and add
an explicit type=raw,value=latest entry gated on github.ref_type ==
'tag' so :latest moves only on release tag pushes.

refs: https://github.com/KumoCorp/kumomta/issues/511
2026-05-06 21:41:33 +01:00
Wez Furlong b0fa4275f3 cargo update
Note: need to pin hierarchical_hash_wheel_timer back because the latest
version requires a rustc upgrade, and current versions of mlua don't
build against it when the send feature is enabled.
2026-05-06 21:32:01 +01:00
dependabot[bot] bd88a3678d build(deps): bump docker/metadata-action from 5 to 6 (#494)
Bumps [docker/metadata-action](https://github.com/docker/metadata-action) from 5 to 6.
- [Release notes](https://github.com/docker/metadata-action/releases)
- [Commits](https://github.com/docker/metadata-action/compare/v5...v6)

---
updated-dependencies:
- dependency-name: docker/metadata-action
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-06 21:15:53 +01:00
Jack c4dbaa743b proxy-server: add tcp_keepalive option
Some upstream peers (e.g. QQ Mail's rate-limiter) silently hold a
proxied TCP connection open indefinitely — sending no data, FIN, or RST
— rather than cleanly refusing.  Without some kind of timeout
management, the two file descriptors for such a session remain open for
the lifetime of the process, slowly exhausting the kernel's
file-descriptor table and occupying proxy-server worker slots.

This commit configures kernel level keepalive options with reasonable
defaults to detect and close out this class of connection.

Co-authored-by: Wez Furlong <wez@wezfurlong.org>

Closes: https://github.com/KumoCorp/kumomta/pull/509
2026-05-06 21:09:10 +01:00
Wez Furlong ee1e5b980e message: add msg:import_headers, a flexible alternative to import_x_headers
import_headers takes an array of per-spec option tables, each describing
how a single header name or pattern should be imported into the message
metadata. Compared to import_x_headers it adds:

 * Trailing-`*` wildcard patterns (e.g. `X-*`) alongside exact names.
   Bare/leading/interior wildcards are rejected at compile time.
 * `match` of `first`, `last` (default), or `all`. `all` captures every
   matching header instance as an array of strings; the others capture
   a string. Specs that produce no matches write nothing.
 * `transform` selects the metadata key style: `snake_case` (default,
   matches the existing import_x_headers behavior), `kebab_case`,
   `camel_case`, or `pascal_case`. Header matching itself is always
   case-insensitive.
 * `target` overrides the metadata key for exact-name specs.
 * `remove` strips the matched headers from the message body in a
   single follow-up pass.

When more than one spec could match a header, the first matching spec
wins, so callers can place specific rules ahead of a wildcard catch-all.

import_x_headers now delegates to import_headers, so its behavior is
unchanged and the two share a single implementation.

retain_headers now passes the header index alongside the &Header to its
closure, which import_headers uses for its post-pass removal step
instead of tracking a parallel counter. Existing callers that don't
need the index ignore it with `_`.

Closes: #515
2026-05-06 16:30:36 +01:00
kay ozaki f2d98eaf62 psl: normalize domains before psl::domain_str / suffix_str lookups
The `psl` crate's domain_str / suffix_str do byte-exact lookups against
the public-suffix list, so uppercase or trailing-dot inputs (e.g.
"Example.COM" or "example.com." from a DNS Name) silently return None.
Several call sites were passing through user- or DNS-supplied domains
unmodified.

Add a small psl-utils crate that wraps the psl crate:

* normalize_domain(&str) -> Cow<str>: strips a single trailing dot and
  lowercases ASCII; borrows when the input is already normalized so
  the hot path (alignment loop) doesn't allocate.
* domain_str / suffix_str: thin re-exports for callers that have
  already normalized.

psl-utils becomes the only workspace crate that directly depends on
the psl crate; kumo-dmarc and mod-string switch to depending on
psl-utils instead.

Updated call sites:

* kumo-dmarc record.rs: is_relaxed_aligned / is_strict_aligned now
  normalize both inputs first. Replaces the prior eq_ignore_ascii_case
  comparisons.
* kumo-dmarc lib.rs: the organizational-domain fallback in
  DmarcContext::check now normalizes from_domain before calling
  domain_str and compares against the normalized form. Previously a
  mixed-case From: header would skip the _dmarc.<org> lookup entirely.
* mod-string lib.rs: the Lua-exposed string.psl_domain and
  string.psl_suffix bindings normalize their input. Behavior change:
  inputs that previously returned nil due to case or trailing dot now
  resolve.

Co-authored-by: Wez Furlong <wez@wezfurlong.org>
Closes: https://github.com/KumoCorp/kumomta/pull/513
2026-05-06 13:34:15 +00:00
kay ozaki 20e26891b0 dmarc: fix alignment evaluation passing unauthenticated mail
Two bugs in the alignment logic added in the prior commit:

* auth_result_is_pass read auth_result.props["result"], but the
  pass/fail status lives in the AuthenticationResult.result field.
  Real callers don't populate a "result" key in props, so every DKIM
  and SPF entry was treated as non-pass and skipped. Take
  &AuthenticationResult and inspect .result directly.

* When no alignment succeeded and no per-method errors were recorded
  (e.g. no DKIM signatures, SPF not pass), evaluate() fell through to
  Disposition::Pass / "Success", silently passing unauthenticated
  mail. Return the published policy disposition with context
  "No aligned DKIM or SPF" instead.

Add regression tests dmarc_both_spf_and_dkim_fail_returns_fail (covers
the fall-through) and dmarc_dkim_relaxed_subdomain_reverse (relaxed
alignment when d= is a subdomain of From).
2026-05-06 13:29:15 +00:00
kay ozaki 1109639c0d enrich props for dkim and dmarc
Surface the underlying DKIM signature tags and the published DMARC
policy tags as auth-result props, so callers (and downstream
Authentication-Results headers) can see what was actually checked.

DKIM (crates/dkim):

* Factor populate_props() over the parsed tagged-header and emit
  header.d, header.i, header.a, header.s, header.c, header.t and
  header.x. The previous code only emitted d/i/a/s on the success
  path.
* On DKIMHeader::parse failure, fall back to a generic TaggedHeader
  parse so the resulting permerror AuthenticationResult still carries
  whatever tags were extractable (e.g. for expired signatures we now
  surface header.d and header.x). Covered by a new roundtrip test.

DMARC (crates/kumo-dmarc, crates/kumod):

* Record now retains the raw key=value tags it parsed, exposed via
  Record::tags().
* DispositionWithContext carries a new props map. Record::evaluate
  initialises it empty; drop the no-longer-relevant ToXml derive.
* DmarcContext::check copies the matched record's tags into the
  result props as policy.<tag> via a new policy_tags() helper.
* kumod's dmarc.rs threads result.props through into the
  AuthenticationResult returned to Lua (rather than starting empty),
  preserving the existing policy.published-domain-policy insertion
  for Quarantine/Reject.
2026-05-06 13:29:15 +00:00
kay ozaki 94fc11dc91 dmarc: pass when either DKIM or SPF aligns, ignore non-pass results
Rework the alignment evaluation in Record::evaluate. Previously every
misaligned DKIM or SPF result produced an alignment_failure, with no
notion of "DKIM didn't pass at all" — and no early-out when one method
did align. DMARC requires only a single passing-and-aligned method, so:

* For each DKIM result, skip entries whose underlying DKIM check did
  not return pass (consulted via the auth-result "result" prop). On
  the first aligned passing signature record cx.dkim_aligned = Pass,
  drop accumulated per-signature errors, and stop iterating.

* For SPF, only consider alignment when the SPF result was pass.
  Source the SPF identity from the auth-result "smtp.mailfrom" /
  "smtp.helo" props (falling back to cx.mail_from_domain), so we
  align against what was actually authenticated upstream.

* If either method aligned, return Pass. Otherwise surface the first
  SPF or DKIM alignment-failure context as the policy result.

Factor the comparisons out into helpers: is_relaxed_aligned (org-domain
match via psl::domain_str) and is_strict_aligned (exact match), both
case-insensitive. Replaces ad-hoc as_bytes() / BStr comparisons.

Add tests dmarc_dkim_ignores_non_pass_results,
dmarc_dkim_continues_until_aligned_result and
dmarc_spf_ignores_non_pass_result, and update the evaluate_ip helper
to populate the "result" / "smtp.mailfrom" props the new logic reads.
2026-05-06 13:24:00 +00:00
kay ozaki 61278dd905 spf: check HELO identity when MAIL FROM is null
RFC 7208 §2.4 says that when the SMTP MAIL FROM is null (e.g. bounces)
the SPF check should fall back to the HELO/EHLO identity with a
local-part of "postmaster". Two gaps prevented that:

* CheckHostParams::check left `domain` empty when MAIL FROM was blank,
  producing a synthesized sender of "postmaster@" with no domain.
  Fall back to ehlo_domain when domain is empty.

* SpfContext::new returned PermError for any sender lacking an '@',
  rejecting bare HELO identities outright. Treat a sender with no '@'
  as ("postmaster", <sender as domain>).

Add a regression test covering SpfContext::new with a bare HELO
identity.
2026-05-06 13:19:53 +00:00
kay ozaki acf27d1b04 make sure trailing semicolon does not cause parsing failure 2026-05-06 13:26:15 +01:00
Wez Furlong 4ab07b7f1c Fix EnvelopeAddress::parse("")
This is similar to https://github.com/KumoCorp/kumomta/pull/512, but
whereas that one was a parser precedence issue, this one is due to
how Null is represented outside of the context of rfc5321.

refs: https://github.com/KumoCorp/kumomta/issues/511
2026-05-05 19:39:27 +01:00
Wez Furlong 4ca193a097 minor fix to doc test in nom-utils 2026-05-05 19:30:46 +01:00
kay ozaki a997a6664b make_message fails for from address which starts with postmaster
closes: https://github.com/KumoCorp/kumomta/pull/512
2026-05-05 19:30:26 +01:00
Wez Furlong 9c5f2e15df rfc5321: fix doc test
This isn't actually a test case, or rust code. Mark it as text.
refs: https://github.com/KumoCorp/kumomta/pull/512
2026-05-05 19:27:03 +01:00
kay ozaki 7921fa5a19 typing.lua boolean-true default prevents mail_auth modules from being disabled
refs: https://github.com/KumoCorp/kumomta/pull/505
2026-04-29 12:06:35 +01:00
kay ozaki de95e63b2a multipart/mixed > multipart/related was not parsing
closes: https://github.com/KumoCorp/kumomta/pull/506
2026-04-29 11:58:03 +01:00
Wez Furlong a4c62d4e2e revert rust compiler bump!
Rust 1.93 causes TimeDelta::__add to fail when adding a userdata to
itself with a bland "error borrowing userdata" error message.

Go back to 1.92 to unblock builds for the moment.
2026-04-29 08:28:06 +01:00
Wez Furlong 07814b08e1 update uuid dep 2026-04-29 08:17:59 +01:00
Wez Furlong 14583a07b8 update rust requirement to 1.94 2026-04-29 08:12:30 +01:00
sophiajt 62b62560de Start of DMARC reporting
The start of reporting support for DMARC. We create the report from the
errors we encounter, though we do not yet have this wired into the email
reporting system.

Our main task is collecting the errors, though where we put them still
needs to be decided. Currently, a temporary file is used.

Once collected, the errors can be aggregated and built into the RFC 7489-compliant report.

Co-authored-by: Wez Furlong <wez@wezfurlong.org>
Closes: https://github.com/KumoCorp/kumomta/pull/463
2026-04-29 08:07:55 +01:00
Wez Furlong 37236c40b1 docs: update for the 2026.04.09-ea3b2a9b release 2026-04-29 08:07:13 +01:00
Wez Furlong ea3b2a9b5f fixup some import warnings 2026-04-09 15:42:41 +01:00
Wez Furlong f068a66f19 rustfmt 2026-04-09 15:42:40 +01:00
Wez Furlong ea87ea6ab3 make HeaderAddress localpart consistent with EnvelopeAddress
This was previously just returning everything before the final at-sign.
Now we use the same parsing approach and return the normalized local
part.

This is technically a breaking change, but it is minor and improves
the overall state of things.
2026-04-09 15:42:40 +01:00
Wez Furlong f8bbf13255 rfc5321: improve consistency of EnvelopeAddress::Display
Drop the source route syntax to make round-tripping and improve
consistency with our other encoding functions for this type.
2026-04-09 15:42:40 +01:00
Wez Furlong 01b595d76f mailparsing: improve conformance with internationalized email and authres
Also fixes up a minor round-trip issue with encoding some of the fields
2026-04-09 15:42:39 +01:00
Wez Furlong 7518ca6de4 serialize bstring as utf8 string when possible
Add some tests to vet this in lua as well.

lua mime parsing functions now also accept bstring input.
2026-04-09 15:42:39 +01:00
Wez Furlong 0ce8718493 lifecycle: fix error propgation from kumo.on('main')
I recently tweaked things to allow ctrl-c to interrupt a long running
--script mode execution, but this had the consequence of breaking
propagation of errors raised in the script to the exit code of
the process, meaning that our lua-based tests would print errors
but not break the runner, making it hard to notice if a regression
was introduced.

This commit resolves that by providing an explicit channel for passing a
return code (rather, an error) through when requesting shutdown.
2026-04-09 15:42:39 +01:00
Wez Furlong 8ee86592b8 mailparsing: partial roll back of BString changes
A number of fields are typed as BString but the parser won't allow
non-UTF8 text through.  Let's type those as String because it
simplifies a lot of of downstream logic.
2026-04-09 15:42:39 +01:00
Wez Furlong facf44d974 message: add test coverage for address header lua interface 2026-04-09 15:42:38 +01:00
Wez Furlong 2814279eb8 remove stale doc comment
This was from an earlier iteration where I was trying too hard
with BString
2026-04-09 15:42:38 +01:00