Entries now have a uuid that identifies them within a given instance
(NOT within a cluster). The id is returned when the entry is
established:
```console
$ kcli --endpoint http://127.0.0.1:8000 bounce --everything --reason boop
{
"id": "9a946198-9351-4e0e-b460-fc84e066878a",
"bounced": {},
"total_bounced": 0
}
```
You can list entries with their remaining duration:
```console
$ kcli --endpoint http://127.0.0.1:8000 bounce-list
[
{
"id": "9a946198-9351-4e0e-b460-fc84e066878a",
"campaign": null,
"tenant": null,
"domain": null,
"reason": "boop",
"duration": "4m 57s 313ms 891us 882ns"
}
]
```
and cancel them:
```console
$ kcli --endpoint http://127.0.0.1:8000 bounce-cancel --id 9a946198-9351-4e0e-b460-fc84e066878a
removed 9a946198-9351-4e0e-b460-fc84e066878a
```
and the error case if you try it again:
```console
$ kcli --endpoint http://127.0.0.1:8000 bounce-cancel --id 9a946198-9351-4e0e-b460-fc84e066878a
Error: bounce entry 9a946198-9351-4e0e-b460-fc84e066878a not found
```
This doesn't do anything in the current milestone, but I wanted
to get it in there ready for work for the next milestone,
where I plan to make it possible to list and remove/cancel
a bounce, as well as persist them across a restart.
The mail-auth crate doesn't support 1024 bit RSA keys, which are
currently the dominant size of key used by senders at scale.
It is not possible to even configure a 2048 bit RSA key in some DNS
providers, which blocks widespread adoption of larger key sizes.
This commit pulls in the cfdkim crate as an alternative implementation.
I'm not happy with this implementation because the API of the crate
defeats some caching optimizations, doesn't support certain signing
attributes that are supported by mail-auth and the generated header isn't
nicely wrapped.
We'll likely "do something" to resolve this in a future commit.
For now, this implementation is enabled by setting `use_cf = true`
in the signing parameters.
```console
$ kcli --help
KumoMTA CLI.
Full docs available at: <https://docs.kumomta.com>
Usage: kcli --endpoint <ENDPOINT> <COMMAND>
Commands:
bounce
Administratively bounce messages in matching queues
set-log-filter
Changes the diagnostic log filter
help
Print this message or the help of the given subcommand(s)
Options:
--endpoint <ENDPOINT>
URL to reach the KumoMTA HTTP API
-h, --help
Print help (see a summary with '-h')
-V, --version
Print version
```
```console
$ kcli bounce --help
Administratively bounce messages in matching queues.
Each individual message that is bounced will generate a log record capturing the event and then be removed from the spool.
Make sure that you mean it, as there is no going back!
The bounce will be applied immediately to queued messages, and the directive will remain in effect for the duration specified, causing newly received messages or messages that were in a transient state at the time the directive was received, to also be bounced as they are placed back into the matching queue(s).
Usage: kcli --endpoint <ENDPOINT> bounce [OPTIONS] --reason <REASON>
Options:
--domain <DOMAIN>
The domain name to match. If omitted, any domains will match!
--campaign <CAMPAIGN>
The campaign name to match. If omitted, any campaigns will match!
--tenant <TENANT>
The tenant name to match. If omitted, any tenant will match!
--reason <REASON>
The reason to log in the delivery logs
--everything
Purge all queues
--duration <DURATION>
The duration over which matching messages will continue to bounce. The default is '5m'
-h, --help
Print help (see a summary with '-h')
```
```console
$ kcli set-log-filter --help
Changes the diagnostic log filter
See <https://docs.kumomta.com/reference/kumo/set_diagnostic_log_filter/> for more information about the log filter syntax.
Usage: kcli --endpoint <ENDPOINT> set-log-filter <FILTER>
Arguments:
<FILTER>
Options:
-h, --help Print help
```
refs: https://github.com/KumoCorp/kumomta/issues/55
This commit also includes a policy helper
`policy_extras.listener_domains` to make it convenient to define
listener domains in toml and/or json files.
To facilitate this, the DomainMap rust structure has been exposed
to lua code via the new `kumo.domain_map.new` function.
This is a pretty tight, lightweight implementation of SOCKS5
intended to be used to facilitate an MTA asking it to bind and
connect.
It uses the splice(2) syscall to avoid kernel/userspace copies
when passing data through, so it should be quite efficient.
refs: https://github.com/KumoCorp/kumomta/issues/45
This is a bit of a PITA.
In the docker build we end up with some bad linkage on libssl.so.1.1
on Ubuntu. That is the legacy ssl library, to which we should not
have been able to link, and for which there are no apt installable
sources.
See
<https://askubuntu.com/questions/1465498/update-libssl-so-1-1-for-the-latest-version>
Looking at the deps, the source of this linkage is the new HTTP
client bindings.
This commit dances around this issue by using the rustls implementation
for the client instead.
It's possible that this won't be sufficient, but it at least removes
the openssl linkage issue as a source of headaches for now.
I used a config like this for testing:
```
global
log stdout format raw local0 debug
defaults
timeout connect 10s
timeout client 30s
timeout server 30s
log global
listen outboundsmtp
log global
bind 0:2526 accept-proxy
mode tcp
use-server v4 if { src 0.0.0.0/0 }
use-server v6 if { src ::/0 }
server v4 0.0.0.0 source 0.0.0.0 usesrc clientip
server v6 ::: source ::: usesrc clientip
```
Launched via:
```console
$ sudo haproxy -f assets/haproxy.conf -V
```
with the source:
```lua
kumo.define_egress_source {
name = 'w00t',
ha_proxy_server = '127.0.0.1:2526',
ha_proxy_source_address = '192.168.1.92',
}
```
refs: #19
This makes it easier to compose secret management with a variety
of lua functions without having to build that directly into each
of them.
In particular: one can now use vaults or other secret stores that
we add in the future to manage credentials for HTTP clients.
```lua
local response = kumo.http
.build_client({})
:get('https://example.com/')
:send()
print(response:status_code(), response:status_reason())
for k, v in pairs(response:headers()) do
print("Header", k, v)
end
print(response:text())
```
```lua
-- Constructs a connection object in lua
kumo.on('make.lua-sender', function(domain, tenant, campaign)
print 'making lua sender'
local sender = {}
function sender:send(message)
print('Sending a message!', message)
-- To log a transient failure:
-- kumo.reject(420, 'boo!')
-- To log a permanent failure:
-- kumo.reject(500, "perm fail")
-- To log success, return a string with additional data
-- to be logged
return 'Super!'
end
return sender
end)
-- Make everything use the lua "connection":
kumo.on('get_queue_config', function(domain, tenant, campaign)
return kumo.make_queue_config {
protocol = {
custom_lua = {
constructor = 'make.lua-sender',
}
},
}
end)
```
it pulls in an old version of the time crate that flags many
rust projects as being vulnerable to an old CVE.
We still have other references to time that I'll remove in a follow up.
A couple of issues in here:
1. a copypasta incident meant that the v6 lookup was really just a v4
lookup. doh!
2. We would lookup v4 addresses and just be satisfied with those,
ignoring any v6 addresses
We now run the v4 and v6 lookups concurrently and merge the results
together.
We do still list the v4 addresses first in the list of addresses.
We don't try to be smart about using the "right" addresses when
work through the list; we try them all until one works out.
Correctly guessing at which one will work means replicating logic
in the kernel, and would potentially be screwed up when we add in
HAProxy support later.
It doesn't do much more than act as a glorified persistent
cursor at the moment, but the plumbing will enable calling
out to eg: lua or an arbitrary subprocess to process batches
in the future.
This commit adjusts the logger:
* When a log segment is finised, the `w` permission bits are removed.
This is used to signal to the tailer that there will be no additional
data written to that file.
* On startup, kumod will remove the `w` bits from all files in the
configured log directories so that any that were left on by a prior
incarnation will be considered closed when the new generation is
started up.
* Add a `tailer` utility that will essentially `cat` all of the log
files in a given directory in the appropriate order, then go into
log tailing mode, where it will wait for new data to show up.
* When reading data, if it hits EOF, the tailer will wait around until
more data shows up, or until the `w` bit is removed from the file
that it is reading.
The idea is that, in a later commit, the tailer will be able to persist
a checkpoint file to record where it got to, and be able to dispatch
chunks of data to eg: a webhook push utility, advancing the checkpoint
only when the utility indicates success.
For now, running `cargo run -p tailer -- --tail /var/tmp/kumo-logs` is
a convenient way to tail the logs as they are written.
This function returns MX information, including the computed site_name.
Update the queue example to show how to use it to populate the SITE_OF
map used in that example.
This commit switches to using the `Name` type under the covers to ensure
that we use canonical names for our cache keys.
Unfortunately, the upstream bitstring-trees crate didn't expose
a way to query the tree (only iterate it), so I pulled in a copy
of the implementation and adjusted it for that purpose, removing
some unsafe code that replaced values during insertion (for performance)
with a simpler implementation that clones values instead.
With the map implementation in the crate, I adjusted the set
implementation to be a `CidrMap<()>` and dropped the bitstring-trees
dependency.
Improve the big-O cost of finding an entry by storing
the ranges in a compressed radix tree under the covers.
Make it a bit easier to work with the type in code while
we're in here.
"any" is now a possible value for cidrsets, representing
any value.
Several crates depend on the config crate, which means that
several things needed to be rebuilt after committing/amending/syncing.
Separating it out reduces that impact.
Wez Furlong