For eg: the Sender and From headers.
This commit introduces a parser for RFC 5322 message header syntax,
as modified by RFC 2047 for encoded header fields.
This allows tracing incoming SMTP sessions.
Sessions are traced through to the cli in realtime.
The implementation is via a websocket, which makes this also potentially
pluggable into a webui in the future, although care must be taken to
ensure that only authorized system operators can enable this, as it can
reveal authentication secrets in the traced dialog, as well as the
content of messages.
This commit doesn't include documentation on the shape of the websocket
JSON at this time.
refs: https://github.com/KumoCorp/kumomta/issues/44
```
; ./target/debug/kcli trace-smtp-server
[127.0.0.1:51778->127.0.0.1:2025] === Connected {"received_from":"127.0.0.1:51778","received_via":"127.0.0.1:2025","reception_protocol":"ESMTP"}
[127.0.0.1:51778->127.0.0.1:2025] <- 220 foo Welcome to KumoMTA!
[127.0.0.1:51778->127.0.0.1:2025] -> EHLO foo.lan
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_ehlo: Ok
[127.0.0.1:51778->127.0.0.1:2025] <- 250-foo Aloha foo.lan
[127.0.0.1:51778->127.0.0.1:2025] <- 250-PIPELINING
[127.0.0.1:51778->127.0.0.1:2025] <- 250-ENHANCEDSTATUSCODES
[127.0.0.1:51778->127.0.0.1:2025] <- 250 STARTTLS
[127.0.0.1:51778->127.0.0.1:2025] -> STARTTLS
[127.0.0.1:51778->127.0.0.1:2025] <- 220 Ready to Start TLS
[127.0.0.1:51778->127.0.0.1:2025] -> EHLO foo.lan
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_ehlo: Ok
[127.0.0.1:51778->127.0.0.1:2025] <- 250-foo Aloha foo.lan
[127.0.0.1:51778->127.0.0.1:2025] <- 250-PIPELINING
[127.0.0.1:51778->127.0.0.1:2025] <- 250-ENHANCEDSTATUSCODES
[127.0.0.1:51778->127.0.0.1:2025] <- 250 AUTH PLAIN
[127.0.0.1:51778->127.0.0.1:2025] -> AUTH PLAIN AHNjb3R0AHRpZ2Vy
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_auth_plain: Ok: Bool(true)
[127.0.0.1:51778->127.0.0.1:2025] <- 235 2.7.0 AUTH OK!
[127.0.0.1:51778->127.0.0.1:2025] === conn_meta updated to {"authn_id":"scott","authz_id":"scott","received_from":"127.0.0.1:51778","received_via":"127.0.0.1:2025","reception_protocol":"ESMTP"}
[127.0.0.1:51778->127.0.0.1:2025] -> MAIL FROM:<wez@example.com>
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_mail_from: Ok
[127.0.0.1:51778->127.0.0.1:2025] <- 250 OK EnvelopeAddress(\"wez@example.com\")
[127.0.0.1:51778->127.0.0.1:2025] -> RCPT TO:<wez@wezfurlong.org>
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_rcpt_to: Ok
[127.0.0.1:51778->127.0.0.1:2025] <- 250 OK EnvelopeAddress(\"wez@wezfurlong.org\")
[127.0.0.1:51778->127.0.0.1:2025] -> DATA
[127.0.0.1:51778->127.0.0.1:2025] <- 354 Send body; end with CRLF.CRLF
[127.0.0.1:51778->127.0.0.1:2025] -> Date: Sun, 13 Aug 2023 10:10:40 -0700
[127.0.0.1:51778->127.0.0.1:2025] -> To: wez@wezfurlong.org
[127.0.0.1:51778->127.0.0.1:2025] -> From: wez@example.com
[127.0.0.1:51778->127.0.0.1:2025] -> Subject: test Sun, 13 Aug 2023 10:10:40 -0700
[127.0.0.1:51778->127.0.0.1:2025] -> Message-Id: <20230813101040.1426417@foo>
[127.0.0.1:51778->127.0.0.1:2025] -> X-Mailer: swaks v20201014.0 jetmore.org/john/code/swaks/
[127.0.0.1:51778->127.0.0.1:2025] ->
[127.0.0.1:51778->127.0.0.1:2025] -> This is a test mailing
[127.0.0.1:51778->127.0.0.1:2025] ->
[127.0.0.1:51778->127.0.0.1:2025] ->
[127.0.0.1:51778->127.0.0.1:2025] -> .
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_message_received: Ok
[127.0.0.1:51778->127.0.0.1:2025] === Message from=wez@example.comto=wez@wezfurlong.org id=5465de7739fc11ee8af250ebf67f93bd
[127.0.0.1:51778->127.0.0.1:2025] === Message queue=wezfurlong.org relay=true log_arf=false log_oob=false
[127.0.0.1:51778->127.0.0.1:2025] === Message meta: {"authn_id":"scott","authz_id":"scott","received_from":"127.0.0.1:51778","received_via":"127.0.0.1:2025","reception_protocol":"ESMTP"}
[127.0.0.1:51778->127.0.0.1:2025] <- 250 OK ids=5465de7739fc11ee8af250ebf67f93bd
[127.0.0.1:51778->127.0.0.1:2025] -> QUIT
[127.0.0.1:51778->127.0.0.1:2025] <- 221 So long, and thanks for all the fish!
[127.0.0.1:51778->127.0.0.1:2025] === Closed
```
How it works:
* When the lowest preference MX host names match a pattern like
`.mail.protection.outlook.com`, the message has its routing_domain
set to a placeholder domain whose name ends with `.ip_rollup`.
* That results in a scheduled queue name like
`foo.com!outlook.ip_rollup`, which makes it possible to know both the
original domain and the fact that rollup is in use.
* `get_queue_config` can check to see if the routing_domain is set to
something that ends with `.ip_rollup` to override the `mx_list`
in the queue configuration with just the lowest preference IP
addresses from the original domain.
* Now, instead of computing a site_name base around
`foo-com.mail.protection.outlook.com`, which includes the individual
original recipient domain, and would cause there to be a separate
ready queue for each domain, the overridden mx_list causes
the site_name to be eg: `mx_list:[104.47.24.36],[104.47.25.36]`.
That same site_name will be used for every domain that shares those
same IP addresses
* When `get_egress_path_config` is called to get shaping parameters,
it is passed the routing domain `outlook.ip_rollup`. You can use that
name with mx_rollup=false as the key in your shaping.toml, of if you
are directly implementing `get_egress_path_config`, you can use that
name to determine the appropriate configuration.
```lua
kumo.on('get_queue_config', function(domain, tenant, campaign, routing_domain)
local params = {}
rollup.apply_ip_rollup_to_queue_config(domain, routing_domain, params
return kumo.make_queue_config(params)
end)
kumo.on('smtp_server_message_received', function(msg)
rollup.reroute_using_ip_rollup(msg, {
['.mail.protection.outlook.com.'] = 'outlook.ip_rollup',
})
end)
```
In your shaping.toml:
```toml
["outlook.ip_rollup"]
mx_rollup = false
# shaping parameters here
```
Caveats:
* With this technique, we'll never try to use any of the lower
priority/higher preference value MX records for any of the matching
domains.
* The IP addresses to which the MX host names resolve can vary over time.
We'll still queue the mail to the same scheduled queue (eg:
`foo.com!outlook.ip_rollup`), but it's possible for there to be
multiple ready queues with different names based on those changed
IPs. This is actually a feature: if the destination domain has
an outage and are now publishing different IPs, we'll pick those up
and use them.
* Since the ready queue names look like `mx_list:[104.47.24.36],[104.47.25.36]`
it can be hard to intuit just from glancing at that name where those queues go.
This commit causes the scheduled queue maintainer to refresh
the queue config by calling the get_queue_config event approximately
every minute while the queue is alive.
In addition, we now thread the routing_domain through to get_queue_config
You may now remove or replace the test domain suffix from the set
of senders/recipients by usingn `--domain-suffix ''` to remove
it, or some other value to set it to something else.
`--domain` can now be used multiple times to build out the list
of test domains.
We had a user get tripped up by the incorrect example on this page.
I think they probably should have been using the helper, but the
way this page was constructed, it was easy to keep reading past
the bit about the helper and get bogged down by the lua examples.
Rearrange this page to try to avoid that.
I've seen this trip up at least two people so far.
While we're in there, update the example to show how to memoize and make
it both easier to write and more efficient at runtime.
This allows listing out authorization identities.
If the incoming client is authorized to act as one of the listed
`relay_from_authz` authorization identities, then the incoming
session will be added to the relay_from CIDR for the scope of
that connected session, and the result is that they will be allowed
to relay from the associated domain.
This is useful if you want to give a tenant SMTP auth credentials
but only want to allow them to send from the domain(s) that
they are permitted to send as/from.
A couple of scenarios we're shooting for here:
* Startup ordering races, or other "burps" due to eg: restarting the
tsa daemon to update its config
* For "replication" scenarios, you may be running multiple instances
of tsa-daemon on multiple nodes, and list them all in both the
publish and subscribe lists on the clients.
In that situation the client will try to read from each node;
the data its reads should be essentially the same from each of them,
and it is fine if only one out of the set responds, or honestly,
even if none of them respond in the moment, as we'll eventually
be able to read that data.
Errors making those http requests are logged to the diagnostic log,
but the shaping data is otherwise allowed to load.
We now generate more context-rich error messages to help pinpoint
the problem:
```
caused by: failed to parse '10.0.0.1/24' as CIDR notation: host part of address was not zero
```
This is a prototype of a means for dealing with rolling up microsoft's
domains so that shaping can be applied holistically over both eg:
hotmail.com and outlook.com.
Those domains don't work with our default automatic MX based rollup
because they don't return the same MX hostnames.
Checking this in to make it easier for others to test/evaluate.
In the future, we may find a way to integrate this with our main
shaping helpers.
It's easy to typo this and get surprised by the results, so explicitly
check for it and provide a more actionable message than the DNS
resolution error for `cox.` would imply:
```
Entry for domain 'cox' consists of a single DNS label. Domain names in TOML sections need to be quoted like '["cox.com"]` otherwise the '.' will create a nested table rather than being added to the domain name.
error resolving MX for cox: no record found for Query { name: Name("cox."), query_type: A, query_class: IN }. Ignoring the shaping config for that domain.
```
Augments our queue name format to be
`campaign:tenant@domain!routing_domain`.
The routing_domain is optional. If the routing_domain is not set, its
effective value is that of the recipient domain.
You can `msg:set_meta('routing_domain', 'bar.com')` to set the
routing_domain for a message, so if the original recipient was
`user@foo.com`, that would cause the computed queue name for it to be
`foo.com!bar.com`.
The routing_domain is used when deciding on the ready_queue name
and destination MXs, so continuing our example, instead of resolving
`foo.com` MX records we'd resolve `bar.com` and deliver to that site.
The `get_egress_path_config` event `domain` parameter is redefined to be
the effective `routing_domain`.
The `get_queue_config` event `domain` parameter is the regular recipient
domain. The `routing_domain` is not currently made available to
`get_queue_config`. If/when we expose it, it will likely be via a
queue name object instead of adding an additional parameter. That would
be a breaking change.
The consequence of not exposing this parameter is that per-message
routing scenarios for the same domain (but different routing domains)
cannot vary the scheduled queue parmeters (eg: retry intervals). Even
though they would have separate scheduled queue instances, those
instances would have the same scheduled queue parameters. If you need
to be able to do that, then explicitly setting the domain portion of the
queue name would be a way to do that: `msg:set_meta('queue',
'foo.com-via-bar.com!bar.com')`. `get_queue_config` would then be
called with `domain='foo.com-via-bar.com'` and your policy could then
respond accordingly.
Previously, you would do either:
`msg:set_meta('queue', 'smart.host.domain')`
or
`msg:set_meta('queue', '[10.0.0.1]')`
to override the effective domain for a message and cause it to be routed
to somewhere other than the recipient domain.
That was OK for basic smart hosting, but limiting when you wanted to use
multiple candidate hosts.
This commit expands the queue config `protocol` field to support
specifying an explicit list of MX hosts that should be used instead.
The integration tests have been migrated away from the old style to this
new style.
While adding plumbing for this, I uncovered an inconsistency between the
queue name generated for the ready queue and the name used by suspension
handling. The inconsistency was introduced in
0842a0fc8b and related work. This commit
resolves it.
Wez Furlong
Mike Hillyer