We don't set received_via for http because axum/hyper make it a
bit weird to access that information and it will require some
effort to figure out how to get that without breaking the
existing proxy-aware client IP determination.
Entries now have a uuid that identifies them within a given instance
(NOT within a cluster). The id is returned when the entry is
established:
```console
$ kcli --endpoint http://127.0.0.1:8000 bounce --everything --reason boop
{
"id": "9a946198-9351-4e0e-b460-fc84e066878a",
"bounced": {},
"total_bounced": 0
}
```
You can list entries with their remaining duration:
```console
$ kcli --endpoint http://127.0.0.1:8000 bounce-list
[
{
"id": "9a946198-9351-4e0e-b460-fc84e066878a",
"campaign": null,
"tenant": null,
"domain": null,
"reason": "boop",
"duration": "4m 57s 313ms 891us 882ns"
}
]
```
and cancel them:
```console
$ kcli --endpoint http://127.0.0.1:8000 bounce-cancel --id 9a946198-9351-4e0e-b460-fc84e066878a
removed 9a946198-9351-4e0e-b460-fc84e066878a
```
and the error case if you try it again:
```console
$ kcli --endpoint http://127.0.0.1:8000 bounce-cancel --id 9a946198-9351-4e0e-b460-fc84e066878a
Error: bounce entry 9a946198-9351-4e0e-b460-fc84e066878a not found
```
This doesn't do anything in the current milestone, but I wanted
to get it in there ready for work for the next milestone,
where I plan to make it possible to list and remove/cancel
a bounce, as well as persist them across a restart.
```console
$ kcli --help
KumoMTA CLI.
Full docs available at: <https://docs.kumomta.com>
Usage: kcli --endpoint <ENDPOINT> <COMMAND>
Commands:
bounce
Administratively bounce messages in matching queues
set-log-filter
Changes the diagnostic log filter
help
Print this message or the help of the given subcommand(s)
Options:
--endpoint <ENDPOINT>
URL to reach the KumoMTA HTTP API
-h, --help
Print help (see a summary with '-h')
-V, --version
Print version
```
```console
$ kcli bounce --help
Administratively bounce messages in matching queues.
Each individual message that is bounced will generate a log record capturing the event and then be removed from the spool.
Make sure that you mean it, as there is no going back!
The bounce will be applied immediately to queued messages, and the directive will remain in effect for the duration specified, causing newly received messages or messages that were in a transient state at the time the directive was received, to also be bounced as they are placed back into the matching queue(s).
Usage: kcli --endpoint <ENDPOINT> bounce [OPTIONS] --reason <REASON>
Options:
--domain <DOMAIN>
The domain name to match. If omitted, any domains will match!
--campaign <CAMPAIGN>
The campaign name to match. If omitted, any campaigns will match!
--tenant <TENANT>
The tenant name to match. If omitted, any tenant will match!
--reason <REASON>
The reason to log in the delivery logs
--everything
Purge all queues
--duration <DURATION>
The duration over which matching messages will continue to bounce. The default is '5m'
-h, --help
Print help (see a summary with '-h')
```
```console
$ kcli set-log-filter --help
Changes the diagnostic log filter
See <https://docs.kumomta.com/reference/kumo/set_diagnostic_log_filter/> for more information about the log filter syntax.
Usage: kcli --endpoint <ENDPOINT> set-log-filter <FILTER>
Arguments:
<FILTER>
Options:
-h, --help Print help
```
refs: https://github.com/KumoCorp/kumomta/issues/55
Systemd doesn't terminate processes using SIGINT, so we didn't
get a chance to flush logs or messages to spool.
This should address that.
refs: https://github.com/KumoCorp/kumomta/issues/46
This commit also includes a policy helper
`policy_extras.listener_domains` to make it convenient to define
listener domains in toml and/or json files.
To facilitate this, the DomainMap rust structure has been exposed
to lua code via the new `kumo.domain_map.new` function.
Switch the configuration plumbing for pools and sources to be pull-based
rather than push based.
In other words, rather than defining them in the `init` event,
you now need to supply them to the new `get_egress_pool` and
`get_egress_source` events.
Data is cached by default for 1 minute. This allows for new sources
to come into being on-demand, and for data to age out and change
over time, without requiring that the server be restarted.
This commit updates the reference section, but there is some content
in the user guide that refers to the old style of configuration that
will need to be updated.
refs: https://github.com/KumoCorp/kumomta/issues/13
Produces an error like:
```
; KUMOD_LOG=kumod=info ./target/debug/kumod --policy simple_policy.lua
2023-05-16T18:16:02.112017Z INFO main kumod::memory: using limits: soft=Some("101.22 GB"), hard=Some("134.96 GB")
Error: callback error
stack traceback:
[C]: in function 'kumo.on'
[string "simple_policy.lua"]:232: in main chunk
caused by: smtp_server_auth_plain event already has a handler defined at simple_policy.lua:226
```
I used a config like this for testing:
```
global
log stdout format raw local0 debug
defaults
timeout connect 10s
timeout client 30s
timeout server 30s
log global
listen outboundsmtp
log global
bind 0:2526 accept-proxy
mode tcp
use-server v4 if { src 0.0.0.0/0 }
use-server v6 if { src ::/0 }
server v4 0.0.0.0 source 0.0.0.0 usesrc clientip
server v6 ::: source ::: usesrc clientip
```
Launched via:
```console
$ sudo haproxy -f assets/haproxy.conf -V
```
with the source:
```lua
kumo.define_egress_source {
name = 'w00t',
ha_proxy_server = '127.0.0.1:2526',
ha_proxy_source_address = '192.168.1.92',
}
```
refs: #19
This makes it easier to compose secret management with a variety
of lua functions without having to build that directly into each
of them.
In particular: one can now use vaults or other secret stores that
we add in the future to manage credentials for HTTP clients.
```lua
local response = kumo.http
.build_client({})
:get('https://example.com/')
:send()
print(response:status_code(), response:status_reason())
for k, v in pairs(response:headers()) do
print("Header", k, v)
end
print(response:text())
```
Still need to write up some proper docs, but the gist is that
the example below will enqueue a message holding the content
of the JsonRecord for each log event, and that message will be
dispatched via a custom lua delivery mechanism.
```lua
kumo.on('init', function()
kumo.configure_log_hook {}
end)
kumo.on('make.lua-sender', function(domain, tenant, campaign)
print 'making lua sender'
local sender = {}
function sender:send(message)
print('Sending a message!', message:get_data())
return 'Super!'
end
return sender
end)
kumo.on('should_enqueue_log_record', function(msg)
local log_record = msg:get_meta 'log_record'
print('should_enqueue_log_record', log_record)
-- avoid an infinite loop caused by logging that we logged
if log_record.queue ~= 'lua-stuff' then
msg:set_meta('queue', 'lua-stuff')
return true
end
return false
end)
kumo.on('get_queue_config', function(domain, tenant, campaign)
print('get_queue_config', domain, tenant, campaign)
if domain == 'lua-stuff' then
return kumo.make_queue_config {
protocol = {
custom_lua = {
constructor = 'make.lua-sender',
},
},
}
end
return kumo.make_queue_config {}
end)
```
```lua
-- Constructs a connection object in lua
kumo.on('make.lua-sender', function(domain, tenant, campaign)
print 'making lua sender'
local sender = {}
function sender:send(message)
print('Sending a message!', message)
-- To log a transient failure:
-- kumo.reject(420, 'boo!')
-- To log a permanent failure:
-- kumo.reject(500, "perm fail")
-- To log success, return a string with additional data
-- to be logged
return 'Super!'
end
return sender
end)
-- Make everything use the lua "connection":
kumo.on('get_queue_config', function(domain, tenant, campaign)
return kumo.make_queue_config {
protocol = {
custom_lua = {
constructor = 'make.lua-sender',
}
},
}
end)
```
Ensure that we trigger an async compatible callback so that
we can use eg: DNS resolution and memoization when processing it.
Fix the site name so that it doesn't include "source->site" and is
just the pure site name.
In some cases (eg: initial spool -> expiration) we don't have
any metadata loaded, causing the logs to say that the metadata
wasn't loaded.
This commit loads it as needed.
it pulls in an old version of the time crate that flags many
rust projects as being vulnerable to an old CVE.
We still have other references to time that I'll remove in a follow up.
This commit adjusts the logger:
* When a log segment is finised, the `w` permission bits are removed.
This is used to signal to the tailer that there will be no additional
data written to that file.
* On startup, kumod will remove the `w` bits from all files in the
configured log directories so that any that were left on by a prior
incarnation will be considered closed when the new generation is
started up.
* Add a `tailer` utility that will essentially `cat` all of the log
files in a given directory in the appropriate order, then go into
log tailing mode, where it will wait for new data to show up.
* When reading data, if it hits EOF, the tailer will wait around until
more data shows up, or until the `w` bit is removed from the file
that it is reading.
The idea is that, in a later commit, the tailer will be able to persist
a checkpoint file to record where it got to, and be able to dispatch
chunks of data to eg: a webhook push utility, advancing the checkpoint
only when the utility indicates success.
For now, running `cargo run -p tailer -- --tail /var/tmp/kumo-logs` is
a convenient way to tail the logs as they are written.
Wez Furlong