I'd initially avoided this because I thought that pest would make
things easier overall, but I'm facing an impossible to debug
situation with the pest parser because the error/diagnostics
are not very friendly.
This commit introduces some plumbing that uses nom to build
the parser; the error reporting is much easier to understand
as we can annotate context information and display it in
an error-chain style with a helpful caret to highlight the
problem spots.
I'll convert more over in subsequent commits; this one is getting
large so this is a checkpoint.
For eg: the Sender and From headers.
This commit introduces a parser for RFC 5322 message header syntax,
as modified by RFC 2047 for encoded header fields.
This allows tracing incoming SMTP sessions.
Sessions are traced through to the cli in realtime.
The implementation is via a websocket, which makes this also potentially
pluggable into a webui in the future, although care must be taken to
ensure that only authorized system operators can enable this, as it can
reveal authentication secrets in the traced dialog, as well as the
content of messages.
This commit doesn't include documentation on the shape of the websocket
JSON at this time.
refs: https://github.com/KumoCorp/kumomta/issues/44
```
; ./target/debug/kcli trace-smtp-server
[127.0.0.1:51778->127.0.0.1:2025] === Connected {"received_from":"127.0.0.1:51778","received_via":"127.0.0.1:2025","reception_protocol":"ESMTP"}
[127.0.0.1:51778->127.0.0.1:2025] <- 220 foo Welcome to KumoMTA!
[127.0.0.1:51778->127.0.0.1:2025] -> EHLO foo.lan
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_ehlo: Ok
[127.0.0.1:51778->127.0.0.1:2025] <- 250-foo Aloha foo.lan
[127.0.0.1:51778->127.0.0.1:2025] <- 250-PIPELINING
[127.0.0.1:51778->127.0.0.1:2025] <- 250-ENHANCEDSTATUSCODES
[127.0.0.1:51778->127.0.0.1:2025] <- 250 STARTTLS
[127.0.0.1:51778->127.0.0.1:2025] -> STARTTLS
[127.0.0.1:51778->127.0.0.1:2025] <- 220 Ready to Start TLS
[127.0.0.1:51778->127.0.0.1:2025] -> EHLO foo.lan
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_ehlo: Ok
[127.0.0.1:51778->127.0.0.1:2025] <- 250-foo Aloha foo.lan
[127.0.0.1:51778->127.0.0.1:2025] <- 250-PIPELINING
[127.0.0.1:51778->127.0.0.1:2025] <- 250-ENHANCEDSTATUSCODES
[127.0.0.1:51778->127.0.0.1:2025] <- 250 AUTH PLAIN
[127.0.0.1:51778->127.0.0.1:2025] -> AUTH PLAIN AHNjb3R0AHRpZ2Vy
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_auth_plain: Ok: Bool(true)
[127.0.0.1:51778->127.0.0.1:2025] <- 235 2.7.0 AUTH OK!
[127.0.0.1:51778->127.0.0.1:2025] === conn_meta updated to {"authn_id":"scott","authz_id":"scott","received_from":"127.0.0.1:51778","received_via":"127.0.0.1:2025","reception_protocol":"ESMTP"}
[127.0.0.1:51778->127.0.0.1:2025] -> MAIL FROM:<wez@example.com>
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_mail_from: Ok
[127.0.0.1:51778->127.0.0.1:2025] <- 250 OK EnvelopeAddress(\"wez@example.com\")
[127.0.0.1:51778->127.0.0.1:2025] -> RCPT TO:<wez@wezfurlong.org>
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_rcpt_to: Ok
[127.0.0.1:51778->127.0.0.1:2025] <- 250 OK EnvelopeAddress(\"wez@wezfurlong.org\")
[127.0.0.1:51778->127.0.0.1:2025] -> DATA
[127.0.0.1:51778->127.0.0.1:2025] <- 354 Send body; end with CRLF.CRLF
[127.0.0.1:51778->127.0.0.1:2025] -> Date: Sun, 13 Aug 2023 10:10:40 -0700
[127.0.0.1:51778->127.0.0.1:2025] -> To: wez@wezfurlong.org
[127.0.0.1:51778->127.0.0.1:2025] -> From: wez@example.com
[127.0.0.1:51778->127.0.0.1:2025] -> Subject: test Sun, 13 Aug 2023 10:10:40 -0700
[127.0.0.1:51778->127.0.0.1:2025] -> Message-Id: <20230813101040.1426417@foo>
[127.0.0.1:51778->127.0.0.1:2025] -> X-Mailer: swaks v20201014.0 jetmore.org/john/code/swaks/
[127.0.0.1:51778->127.0.0.1:2025] ->
[127.0.0.1:51778->127.0.0.1:2025] -> This is a test mailing
[127.0.0.1:51778->127.0.0.1:2025] ->
[127.0.0.1:51778->127.0.0.1:2025] ->
[127.0.0.1:51778->127.0.0.1:2025] -> .
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_message_received: Ok
[127.0.0.1:51778->127.0.0.1:2025] === Message from=wez@example.comto=wez@wezfurlong.org id=5465de7739fc11ee8af250ebf67f93bd
[127.0.0.1:51778->127.0.0.1:2025] === Message queue=wezfurlong.org relay=true log_arf=false log_oob=false
[127.0.0.1:51778->127.0.0.1:2025] === Message meta: {"authn_id":"scott","authz_id":"scott","received_from":"127.0.0.1:51778","received_via":"127.0.0.1:2025","reception_protocol":"ESMTP"}
[127.0.0.1:51778->127.0.0.1:2025] <- 250 OK ids=5465de7739fc11ee8af250ebf67f93bd
[127.0.0.1:51778->127.0.0.1:2025] -> QUIT
[127.0.0.1:51778->127.0.0.1:2025] <- 221 So long, and thanks for all the fish!
[127.0.0.1:51778->127.0.0.1:2025] === Closed
```
How it works:
* When the lowest preference MX host names match a pattern like
`.mail.protection.outlook.com`, the message has its routing_domain
set to a placeholder domain whose name ends with `.ip_rollup`.
* That results in a scheduled queue name like
`foo.com!outlook.ip_rollup`, which makes it possible to know both the
original domain and the fact that rollup is in use.
* `get_queue_config` can check to see if the routing_domain is set to
something that ends with `.ip_rollup` to override the `mx_list`
in the queue configuration with just the lowest preference IP
addresses from the original domain.
* Now, instead of computing a site_name base around
`foo-com.mail.protection.outlook.com`, which includes the individual
original recipient domain, and would cause there to be a separate
ready queue for each domain, the overridden mx_list causes
the site_name to be eg: `mx_list:[104.47.24.36],[104.47.25.36]`.
That same site_name will be used for every domain that shares those
same IP addresses
* When `get_egress_path_config` is called to get shaping parameters,
it is passed the routing domain `outlook.ip_rollup`. You can use that
name with mx_rollup=false as the key in your shaping.toml, of if you
are directly implementing `get_egress_path_config`, you can use that
name to determine the appropriate configuration.
```lua
kumo.on('get_queue_config', function(domain, tenant, campaign, routing_domain)
local params = {}
rollup.apply_ip_rollup_to_queue_config(domain, routing_domain, params
return kumo.make_queue_config(params)
end)
kumo.on('smtp_server_message_received', function(msg)
rollup.reroute_using_ip_rollup(msg, {
['.mail.protection.outlook.com.'] = 'outlook.ip_rollup',
})
end)
```
In your shaping.toml:
```toml
["outlook.ip_rollup"]
mx_rollup = false
# shaping parameters here
```
Caveats:
* With this technique, we'll never try to use any of the lower
priority/higher preference value MX records for any of the matching
domains.
* The IP addresses to which the MX host names resolve can vary over time.
We'll still queue the mail to the same scheduled queue (eg:
`foo.com!outlook.ip_rollup`), but it's possible for there to be
multiple ready queues with different names based on those changed
IPs. This is actually a feature: if the destination domain has
an outage and are now publishing different IPs, we'll pick those up
and use them.
* Since the ready queue names look like `mx_list:[104.47.24.36],[104.47.25.36]`
it can be hard to intuit just from glancing at that name where those queues go.
This commit causes the scheduled queue maintainer to refresh
the queue config by calling the get_queue_config event approximately
every minute while the queue is alive.
In addition, we now thread the routing_domain through to get_queue_config
You may now remove or replace the test domain suffix from the set
of senders/recipients by usingn `--domain-suffix ''` to remove
it, or some other value to set it to something else.
`--domain` can now be used multiple times to build out the list
of test domains.
We had a user get tripped up by the incorrect example on this page.
I think they probably should have been using the helper, but the
way this page was constructed, it was easy to keep reading past
the bit about the helper and get bogged down by the lua examples.
Rearrange this page to try to avoid that.
I've seen this trip up at least two people so far.
While we're in there, update the example to show how to memoize and make
it both easier to write and more efficient at runtime.
Wez Furlong