For eg: the Sender and From headers.
This commit introduces a parser for RFC 5322 message header syntax,
as modified by RFC 2047 for encoded header fields.
This allows tracing incoming SMTP sessions.
Sessions are traced through to the cli in realtime.
The implementation is via a websocket, which makes this also potentially
pluggable into a webui in the future, although care must be taken to
ensure that only authorized system operators can enable this, as it can
reveal authentication secrets in the traced dialog, as well as the
content of messages.
This commit doesn't include documentation on the shape of the websocket
JSON at this time.
refs: https://github.com/KumoCorp/kumomta/issues/44
```
; ./target/debug/kcli trace-smtp-server
[127.0.0.1:51778->127.0.0.1:2025] === Connected {"received_from":"127.0.0.1:51778","received_via":"127.0.0.1:2025","reception_protocol":"ESMTP"}
[127.0.0.1:51778->127.0.0.1:2025] <- 220 foo Welcome to KumoMTA!
[127.0.0.1:51778->127.0.0.1:2025] -> EHLO foo.lan
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_ehlo: Ok
[127.0.0.1:51778->127.0.0.1:2025] <- 250-foo Aloha foo.lan
[127.0.0.1:51778->127.0.0.1:2025] <- 250-PIPELINING
[127.0.0.1:51778->127.0.0.1:2025] <- 250-ENHANCEDSTATUSCODES
[127.0.0.1:51778->127.0.0.1:2025] <- 250 STARTTLS
[127.0.0.1:51778->127.0.0.1:2025] -> STARTTLS
[127.0.0.1:51778->127.0.0.1:2025] <- 220 Ready to Start TLS
[127.0.0.1:51778->127.0.0.1:2025] -> EHLO foo.lan
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_ehlo: Ok
[127.0.0.1:51778->127.0.0.1:2025] <- 250-foo Aloha foo.lan
[127.0.0.1:51778->127.0.0.1:2025] <- 250-PIPELINING
[127.0.0.1:51778->127.0.0.1:2025] <- 250-ENHANCEDSTATUSCODES
[127.0.0.1:51778->127.0.0.1:2025] <- 250 AUTH PLAIN
[127.0.0.1:51778->127.0.0.1:2025] -> AUTH PLAIN AHNjb3R0AHRpZ2Vy
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_auth_plain: Ok: Bool(true)
[127.0.0.1:51778->127.0.0.1:2025] <- 235 2.7.0 AUTH OK!
[127.0.0.1:51778->127.0.0.1:2025] === conn_meta updated to {"authn_id":"scott","authz_id":"scott","received_from":"127.0.0.1:51778","received_via":"127.0.0.1:2025","reception_protocol":"ESMTP"}
[127.0.0.1:51778->127.0.0.1:2025] -> MAIL FROM:<wez@example.com>
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_mail_from: Ok
[127.0.0.1:51778->127.0.0.1:2025] <- 250 OK EnvelopeAddress(\"wez@example.com\")
[127.0.0.1:51778->127.0.0.1:2025] -> RCPT TO:<wez@wezfurlong.org>
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_rcpt_to: Ok
[127.0.0.1:51778->127.0.0.1:2025] <- 250 OK EnvelopeAddress(\"wez@wezfurlong.org\")
[127.0.0.1:51778->127.0.0.1:2025] -> DATA
[127.0.0.1:51778->127.0.0.1:2025] <- 354 Send body; end with CRLF.CRLF
[127.0.0.1:51778->127.0.0.1:2025] -> Date: Sun, 13 Aug 2023 10:10:40 -0700
[127.0.0.1:51778->127.0.0.1:2025] -> To: wez@wezfurlong.org
[127.0.0.1:51778->127.0.0.1:2025] -> From: wez@example.com
[127.0.0.1:51778->127.0.0.1:2025] -> Subject: test Sun, 13 Aug 2023 10:10:40 -0700
[127.0.0.1:51778->127.0.0.1:2025] -> Message-Id: <20230813101040.1426417@foo>
[127.0.0.1:51778->127.0.0.1:2025] -> X-Mailer: swaks v20201014.0 jetmore.org/john/code/swaks/
[127.0.0.1:51778->127.0.0.1:2025] ->
[127.0.0.1:51778->127.0.0.1:2025] -> This is a test mailing
[127.0.0.1:51778->127.0.0.1:2025] ->
[127.0.0.1:51778->127.0.0.1:2025] ->
[127.0.0.1:51778->127.0.0.1:2025] -> .
[127.0.0.1:51778->127.0.0.1:2025] === smtp_server_message_received: Ok
[127.0.0.1:51778->127.0.0.1:2025] === Message from=wez@example.comto=wez@wezfurlong.org id=5465de7739fc11ee8af250ebf67f93bd
[127.0.0.1:51778->127.0.0.1:2025] === Message queue=wezfurlong.org relay=true log_arf=false log_oob=false
[127.0.0.1:51778->127.0.0.1:2025] === Message meta: {"authn_id":"scott","authz_id":"scott","received_from":"127.0.0.1:51778","received_via":"127.0.0.1:2025","reception_protocol":"ESMTP"}
[127.0.0.1:51778->127.0.0.1:2025] <- 250 OK ids=5465de7739fc11ee8af250ebf67f93bd
[127.0.0.1:51778->127.0.0.1:2025] -> QUIT
[127.0.0.1:51778->127.0.0.1:2025] <- 221 So long, and thanks for all the fish!
[127.0.0.1:51778->127.0.0.1:2025] === Closed
```
A couple of scenarios we're shooting for here:
* Startup ordering races, or other "burps" due to eg: restarting the
tsa daemon to update its config
* For "replication" scenarios, you may be running multiple instances
of tsa-daemon on multiple nodes, and list them all in both the
publish and subscribe lists on the clients.
In that situation the client will try to read from each node;
the data its reads should be essentially the same from each of them,
and it is fine if only one out of the set responds, or honestly,
even if none of them respond in the moment, as we'll eventually
be able to read that data.
Errors making those http requests are logged to the diagnostic log,
but the shaping data is otherwise allowed to load.
Use the pause emoji for suspensions, and the wastebasket emoji for
bounces. These are shown in the final column of the respective
sections.
Note that for bounces there will only be a short time window where you
will see a bounced domain show up in the list because the bounce will
remove it from the system fairly quickly.
Allows filtering the results to just those queues associated
with the requested domain. Uses site names to map domains
to their associated ready queues.
This produces commented shaping.toml output corresponding to any config
overrides set by the automation.
For example, this non-sensical rule that applies to every domain:
```toml
[["default".automation]]
regex = "250 2\\.0\\.0 Ok"
action = {SetConfig={name="max_connection_rate", value="100/s"}}
trigger = {Threshold="2/hr"}
duration = "30 secs"
```
when triggered for messages sent to my own domain:
```console
$ curl -s 'http://localhost:8008/get_config_v1'
# Generated by tsa-daemon
# Number of entries: 1
["wezfurlong.org"]
["wezfurlong.org".sources]
["wezfurlong.org".sources.unspecified]
mx_rollup = false
# reason: automation rule: 250 2\.0\.0 Ok
# expires: 2023-08-03T01:47:37+00:00
max_connection_rate = "100/s"
```
This is using an in-memory sqlite db to keep track of events
and actions that we trigger.
In the future, the db will be persisted at a configurable location
on local storage.
Next step is to add endpoints for both config overrides and suspensions
that can be consumed by the mta nodes.
I think something got screwed up somewhere, because I'm sure this
used to work, but: the value wasn't being parsed out of the loaded
data.
Add an integration test to assert that we can load things.
This test may need some auto-detection to run successfully on CI.
Let's see what happens.
This allows us to avoid rebuilding the map on each message reception
when using the dkim helpers, which should improve the performance
for sites with large numbers of signing domains.
The intent is to allow this to be used from shaping.lua,
but a little more plumbing is needed before we can call it
from there.
Added some tests to sanity check that the data looks right.
This moves things around a bit so that we can parse the shaping.toml
data from rust, and expands it to add automation entries per-domain.
A new tool `validate-shaping` can now be used to validate the
shaping configuration independently of starting the server and
injecting mail.
Using the init_with_tsa.lua policy for kumod, together with
tsa_init.lua as the policy for tsa-daemon, logs from kumod
are now sent to tsa-daemon.
It doesn't do anything with them yet.
To support distributed connection limits, adjust the way that
connection limits are handled.
Previously, we'd do a simple in-memory comparison with the limit to
guide whether we had reached the limit.
In a distributed/clustered scenario we need a shared understanding
of the state of the connections, so we're moving to a lease-based
model.
Each connection is associated with a UUID.
Each egress path maintains a set of connections and their UUIDs.
When attempting to make a new connection, we can only proceed if the
number of connections associated with an egress path is below the limit.
When a connection is closed, its UUID is removed from the associated
set.
To handle crashes, kills and netsplits, the path -> uuid association is
also accompanied by an expiration time beyond which it can be assumed
that the lease is no longer valid. We compute that time based on the
worst case timeout value for a single message send on the given pathway.
Each time a connection is ready to obtain a new message, it will try
to extend its lease by that same duration.
This commit introduces these concepts and models them using an in-memory
store for the single node case, and using redis for the cluster case.
When your policy enables `kumo.configure_redis_throttles`, that same
redis configuration is used to back to the connection limiting
functionality.
refs: https://github.com/KumoCorp/kumomta/issues/41
These are two different groups of queues, so there are two different
sets of things to manage them.
kcli now has `suspend(-list|cancel)?` and
`suspend-ready-q(-list|cancel)?` subcommands for establishing a
suspension, listing the suspensions and cancelling a suspension
in the scheduled-q and ready-q namespaces respectively.
The names of the ready queues can be derived from the metrics API:
```console
$ curl -s 'http://localhost:8000/metrics.json' | jq .
...
"ready_count": {
"help": "number of messages in the ready queue",
"type": "gauge",
"value": {
"service": {
"smtp_client:source2->(in1-smtp|in2-smtp).messagingengine.com": 0.0
}
}
},
...
```
From the above, `source2->(in1-smtp|in2-smtp).messagingengine.com` is
the name of the underlying ready queue.
We can and probably should add something to `kcli` to make that slightly
easier to review and manage for the operator.
refs: https://github.com/KumoCorp/kumomta/issues/51
Wez Furlong