Commit Graph

245 Commits

Author SHA1 Message Date
Wez Furlong d385a1a684 logs: expand source_address to include proxy information
Change the field from a SocketAddr to a struct with distinct fields:

```json
    // For SMTP delivery, the source address (and port) that was used.
    // (*Since: Dev Builds Only*)
    "source_address": {
        // The source address. The port number may be unknown and reported
        // as zero when using a proxy protocol.
        "address": "10.0.0.1:53210",
        // If a proxy protocol was used, this field will be
        // set to its name. It may be null/not set for no proxy,
        // "haproxy" or "socks5".
        "protocol": "socks5",
        // If a proxy protocol was used, this field will be
        // set to the proxy server address. It will be null/not set
        // when no proxy was used.
        "server": "192.168.1.1:5000"
    },
```

In #154, the request was to log configuration information here, but I
opted against this as there can be a number of different configuration
fields and the combinatorics for future changes make me uncomfortable
from a code maintainance perspective--it will already be heavy to
try to pass thu all of the existing config information, and as we
add more options in the future it will be awful not just to look at,
but also from a memory and storage overhead.

The approach taken here is to make a little struct that is flexible
enough to convey the desired information without it being too much of a
burden.

closes: #40
closes: #154
2024-07-12 09:22:46 -07:00
Wez Furlong 9a9443be71 logs: add source_address field to SMTP client logs
Previously we would log the pool and source name, but it is
desirable to also log the underlying socket address information,
so here we go!

refs: #40
2024-07-12 08:39:15 -07:00
Wez Furlong b003e49c9c rfc5321: split out banner_timeout from connect_timeout
The motivation for this is:

My test environment is not permitted to reach outbound port 25.
If I run an ad-hoc test without setting up an explicit sink,
I end up with messages that try to reach the public internet.
Since they are blocked at a firewall, each of the MX hosts in
the connection plan is subject to a 60s wait before trying the next
thing.

In addition, this can cause the shutdown to take longer while
we wait for the in-flight delivery attempts to complete.

Making a separate configuration option allows the local administrator
to decide how to split the time waiting for a connection from
the time waiting for the banner.

refs: https://github.com/KumoCorp/kumomta/issues/196
2024-07-12 07:54:43 -07:00
Wez Furlong a00e8d0f0a NEW: add tracing to smtp client, kcli trace-smtp-client
It is now possible to trace outbound SMTP sessions, filtering
by a variety of properties.

Details are in `kcli trace-smtp-client --help` and also in
the docs at /reference/kcli/trace-smtp-client.md

refs: #87
2024-07-11 08:56:25 -07:00
Wez Furlong fdf7fcf249 docs: show how you can compose kumo.encode with kumo.uuid 2024-07-10 06:29:02 -07:00
Wez Furlong f99f852747 mod-sqlite: add :close method 2024-07-09 16:50:48 -07:00
Wez Furlong cde3ba66cb mod-kafka: add explicit close method 2024-07-09 14:09:23 -07:00
Wez Furlong ec795b71d3 mod-http: add client:close() method
Allows explicit closure of any cached connections in the client
object.

Make use of this in the log_hooks:new_json method.
2024-07-09 13:57:46 -07:00
Wez Furlong 73b8b5b4e0 docs: add missing page for configure_unbound_resolver 2024-07-09 13:42:01 -07:00
Wez Furlong c659fe6d4f new: add kumo.uuid lua module 2024-07-09 13:25:32 -07:00
Wez Furlong 58f53687e6 mod-amqp: add explicit client close method
This allows explicit and clean shutdown of a connection.
2024-07-09 09:55:10 -07:00
Tom Mairs 6d4ef9a303 fix typo 2024-07-09 15:09:09 +00:00
Tom Mairs 53b16cefb4 fix typo 2024-07-08 20:40:32 +00:00
Wez Furlong 722f19a078 docs: fix name of requeue_message event in the example 2024-07-08 09:11:34 -07:00
Wez Furlong f7604a3f7f docs: cut over toml examples to toml_data data macro
refs: https://github.com/KumoCorp/kumomta/issues/212
2024-07-03 16:09:52 -07:00
Wez Furlong e49985ff54 logging: capture incoming SMTP command line for Rejections 2024-07-03 06:46:01 -07:00
Wez Furlong ef8cbea354 docs: fixup links to the rapidoc http page 2024-06-27 19:56:53 -07:00
Wez Furlong 5b0d936ef7 docs: improve markdown output for the kcli help 2024-06-27 15:40:39 -07:00
Wez Furlong fa2c6225e8 docs: add kcli reference docs
These are automatically extracted from `kcli --help`.
2024-06-27 15:18:07 -07:00
Wez Furlong d66d6bd750 docs: fix link 2024-06-27 08:39:03 -07:00
Wez Furlong e683fed556 docs: move Log Record to its own reference page
This makes it a bit easier to find and link to.
2024-06-27 08:28:45 -07:00
Wez Furlong f8c3429738 serde: move json/toml to new kumo.serde module. Added YAML.
This helps to avoid bloating kumo-server-common, and makes it
easier to group related functions together in the docs.

The original names are still usable; we'll remove them in a
future release.
2024-06-26 11:08:15 -07:00
Wez Furlong a2d76df7a9 NEW: rebind API and kcli subcommand
refs: https://github.com/KumoCorp/kumomta/issues/209
2024-06-24 12:26:06 -07:00
Wez Furlong 10bb7f2717 Add some string and regex functions
closes: https://github.com/KumoCorp/kumomta/issues/220
2024-06-21 17:20:04 -07:00
Wez Furlong 8673045fff docs: update for latest release 2024-06-10 09:30:17 -07:00
Wez Furlong 29e4ec312d provide helpful reminder that bouncing is async
closes: https://github.com/KumoCorp/kumomta/issues/191
2024-05-30 16:46:03 -07:00
Wez Furlong 03dfa7e870 docs: changelog for https://github.com/KumoCorp/kumomta/pull/179 2024-05-17 07:49:48 -07:00
Wez Furlong a55f726471 docs: utopia openapi doc description updates
The auto-generator changed how it splits the brief/full description
up when rendering it into the openapi spec.
2024-05-17 07:44:39 -07:00
Wez Furlong 4dad617d14 remove EgressPathConfig::suspended logic
When this was added, it was to enable TSA daemon to express
and convey that a given path should be suspended.

The implementation was objectively unpleasant, turning the
fast path of egress source assignment from a simple single
iteration of the weighted round robin logic to N (where N
is the number of sources in a pool) iterations to try
and figure out if a suspension is active.

Now that we have TSA subscriptions that update the admin
suspension information in realtime, we can now simply
rely on that source of information.

This commit removes the logic associated with the deprecated
suspended flag and tidies up the code, making that fast
path a little more fast, and making the code a bit easier
to reason about.
2024-05-02 06:18:14 -07:00
Wez Furlong 9e95303e6b add kumo.available_parallelism()
This can be helpful for writing configuration files for systems
that scale to the available compute resources.
2024-04-29 08:23:27 -07:00
Wez Furlong 3261d1517f redis: add connect_timeout option
closes: https://github.com/KumoCorp/kumomta/issues/168
2024-04-19 12:33:23 -07:00
Wez Furlong b4aa4a38ec http: allow controlling some timeout and diagnostics on the client
Adds a couple of options that provide more control over the default
timeouts and logging.

These are exposed to the shaping helper as `publish_timeout`,
`publish_pool_idle_timeout` and `publish_connection_verbose`.
2024-04-19 07:44:19 -07:00
Wez Furlong 898ed52410 add msg:append_text_html and msg:append_text_html
These can be used to insert tracking pixels/links.

closes: https://github.com/KumoCorp/kumomta/issues/120
2024-03-31 14:04:46 -07:00
Wez Furlong 1fb712e2bf Add msg:set_data()
This method allows replacing the message payload arbitrarily.
This is a building block for more advanced and convenient
modification operations.

refs: https://github.com/KumoCorp/kumomta/issues/117
refs: https://github.com/KumoCorp/kumomta/issues/120
2024-03-31 06:57:59 -07:00
Wez Furlong e94506fd96 Add new Rejection log event
We skip logging the 421 we generate while shutting down because
it feels a bit redundant; you'll see the server shutting down
in the journal anyway.

refs: https://github.com/KumoCorp/kumomta/issues/88
2024-03-29 16:37:37 -07:00
Wez Furlong 3207c45f3d expose inject_v1 API via lua
closes: https://github.com/KumoCorp/kumomta/issues/102
2024-03-29 09:06:54 -07:00
Wez Furlong 755eb848e2 add glob, read_dir
closes: https://github.com/KumoCorp/kumomta/issues/161
2024-03-29 08:25:51 -07:00
Wez Furlong 0c5794ba31 TSA: Add SuspendTenant, SuspendCampaign
refs: https://github.com/KumoCorp/kumomta/issues/113
2024-03-28 13:08:45 -07:00
Wez Furlong e41b4297ea tsa: hook up websocket suspension stream
This commit connects the new websocket based suspension feed
up to shaping.lua. This allows ready-q suspensions to be
enacted in realtime, as well as sets things up to support
scheduled queue suspensions in a later commit.

refs: https://github.com/KumoCorp/kumomta/issues/113
2024-03-28 07:14:36 -07:00
Wez Furlong 39de0e33a6 Expose suspension API to lua
This is very similar to the HTTP suspension API, with the
difference that the suspend method returns just the uuid rather
than the entire suspension object.

refs: https://github.com/KumoCorp/kumomta/issues/113
2024-03-28 07:14:35 -07:00
Wez Furlong be80391239 add kumo.http.connect_websocket
refs: https://github.com/KumoCorp/kumomta/issues/113
2024-03-28 07:14:35 -07:00
Wez Furlong 6003976a10 new: kumo.spawn_task function
This function will spawn a new thread that runs a tokio
localset, which in turn will trigger the specified event
and run it.

On it's own it doesn't do a lot, but it provides a way
to perform background tasks in lua.

```lua
kumo.on('init', function()
  kumo.spawn_task {
    event_name = 'my.task',
    args = { 'hello', 'there' },
  }
end)

kumo.on('my.task', function(args)
  -- Prints: `I am the task.  ["hello","there"]`
  print('I am the task.', kumo.json_encode(args))
end)
```
2024-03-28 07:14:35 -07:00
Wez Furlong a209a470ff docs: clarify how to access headers in logging templates 2024-03-22 08:08:14 -07:00
Wez Furlong 5596d7cf99 http_listener: allow configuring max request size 2024-03-17 09:35:02 -07:00
Tom Mairs d285a7f71a Update timeout defaults 2024-03-15 15:18:06 -06:00
Wez Furlong 5a1999a1b1 Add kumo.make_throttle and throttle_insert_ready_queue event
These allow performing arbitrary rate limiting operations
at both reception time and when messages are moved from the
scheduled queue and into the ready queue.

refs:  https://github.com/KumoCorp/kumomta/issues/149
2024-03-15 12:21:57 -07:00
Wez Furlong a16a886f0e add requeue_message event
In the end I decided to implicitly make the message due for immediate
delivery in the case where the queue was changed; I couldn't think
of a good reason not to do that, and it simplifies the implementation
both of the event internals for anyone implementing the event
themselves in their lua policy.

refs: https://github.com/KumoCorp/kumomta/issues/149
2024-03-08 13:35:06 -07:00
Wez Furlong 33c7557afb msg: add num_attempts, set_due and queue_name methods
refs: https://github.com/KumoCorp/kumomta/issues/149
2024-03-08 13:35:03 -07:00
Wez Furlong 8c43dfe209 add scheduled queue max_message_rate throttle
Needs testing, but I think this will do the job at the raw configuration
level. TSA support for adjusting this setting is a bigger endeavor and
is not included in this commit.

refs: https://github.com/KumoCorp/kumomta/issues/143
2024-03-07 11:00:18 -07:00
Wez Furlong 5bf3d4f3e0 docs: apply proper lua formatting to kafka docs 2024-02-28 09:41:20 -07:00