Commit Graph
1537 Commits
Author SHA1 Message Date
Mike Hillyer b4a149a5f3 Add samples for webhooks to the example config, update webhooks docs page with helper info. 2023-11-14 18:18:47 -05:00
Wez Furlong fa51666051 add log_hooks:new_json shortcut
Basic usage is as simple as:

```lua
local log_hooks = require 'policy-extras.log_hooks'

log_hooks:new_json {
  name = "webhook",
  url = "http://10.0.0.1:4242/log",
}
```
2023-11-14 15:06:48 -07:00
Wez Furlong 4ac3b4e21b log_hooks: set retry_interval to 1m, with max 20m 2023-11-14 14:55:22 -07:00
Wez Furlong f4c6b7e81d cargo update 2023-11-14 14:52:59 -07:00
Mike Hillyer 4d86392add Update example config and page on queue management to reflect the queue helper. 2023-11-14 16:47:54 -05:00
Wez Furlong 10537080f3 docs: changelog for #94 2023-11-13 09:36:46 -07:00
Nathalie Caiandncai 12a317dba3 only add meta to log if not null (#94)
* only add meta to log if not null
* follow norm

Co-authored-by: ncai <ncai@chapsvision.com>
2023-11-13 09:35:10 -07:00
Wez Furlong a03fcb2bf1 docs: fix sqlite auth example 2023-11-10 07:01:58 -07:00
ncai 9d0105e51d use logRecord reception_protocol to avoid loop 2023-11-07 13:11:59 -07:00
Wez Furlong 48d5e26a70 pre_init: fix dispatch
I didn't register this correct for multi-event setup,
so no pre_init event was ever triggered, causing the log_hooks
stuff to not register the log hook.
2023-11-06 20:55:37 -07:00
Wez Furlong d969f87571 multi hooks: use bare return rather than return nil
The way we test whether logs should continue is by looking for
an empty return statement, so we shouldn't `return nil` for these
otherwise we'll indicate that we definitively can't resolve the
queue configuration.
2023-11-06 14:39:37 -07:00
Wez Furlong 0ffbcf7ed0 queue.lua: fix typo 2023-11-06 11:03:06 -07:00
Wez Furlong 68b7979783 queue.lua: allow removing tenant and/or campaign header 2023-11-06 10:59:49 -07:00
Wez Furlong ac7ef32cd3 log_hooks: fix returning the module 2023-11-06 09:50:25 -07:00
Wez Furlong 1c25c0a59b shaping.lua: add extra_files to the TSA inline example 2023-11-06 08:53:34 -07:00
Wez Furlong 951cad53e8 log_hooks: simplify logic and comments a little 2023-11-06 08:46:35 -07:00
Wez Furlong dd93465870 log_hooks: a couple of minor fixes/tweaks 2023-11-06 08:39:31 -07:00
Wez Furlong af92443f7e policy-extras: add log_hooks helper
This should simplify the process of adding log hooks.
2023-11-06 08:31:52 -07:00
Wez Furlong 38179d59c4 docs: fix some broken links 2023-11-06 08:14:11 -07:00
Wez Furlong c1c4ce723e add pre_init event 2023-11-06 08:10:53 -07:00
Wez Furlong 4cb3a4c4e5 cargo update
I had to force-push my dane branch of rust-openssl after a rebase,
which invalidated the commit ref that is included here in Cargo.lock.

Update the deps to allow folks without the stale commit ref to be able
to build.
2023-11-06 06:32:22 -07:00
Wez Furlong 546237d11c deps: trust-dns -> hickory
The project was rebranded relatively recently; switch
to the new version
2023-11-04 10:26:49 -07:00
Wez Furlong e9838cd1a9 new: queue helper
This helps to configure tenant and campaign assignment and scheduled
queue configuration.

refs: https://github.com/KumoCorp/kumomta/issues/90
2023-11-03 13:44:13 -07:00
Wez Furlong 1872dc71a0 shaping: calling shaper.should_enqueue_log_record is no longer required
Allow multiple should_enqueue_log_record hooks to be registered,
and take advantage of that inside the shaping helper.

`shaper.should_enqueue_log_record` no longer needs to be explicitly
plumbed from the config, but we allow calling it still for compatibility
reasons.

We will remove that compatibility after the next stable release.
2023-11-03 08:52:40 -07:00
Wez Furlong 7bb498040b tsa: add some more trace logging
This is helpful when debugging/tracing.
2023-11-03 08:46:36 -07:00
Wez Furlong 3d7007825a shaping: calling shaper.get_queue_config is no longer required
Take advantage of the ability to define multiple get_queue_config
callbacks by internalizing the TSA webhook queue configuration.

`shaper.get_queue_config` no longer needs to be called, and
that fragment of code will continue to "work" by simply not
having an effect so that we don't break folks on upgrade.

We'll remove this compatibility in the release following
the next stable release.
2023-11-02 18:11:17 -07:00
Wez Furlong 929d51deba events: plumbing for registering multiple handlers
Previously, we'd restrict `kumo.on` to allowing just a single
instance of an event to be registered. The purpose of this was
to help surface logical errors where copypasta would result in
a bogus configuration.

With multiple helper lua modules now wanting to take responsibility
for some portion of the event handling, it is becoming more complex
to stitch things together.

It is desirable to allow multiple handlers for certain events,
so that a module can handle just its area of responsibility
without worry other modules about it.

This commit introduces a CallbackSignature type that allows
defining the function signature for event callbacks.

The signature can be pre-created and registered ahead of setting
up any lua contexts, which allows declaring whether an event
can have multiple callbacks registered.

The `get_queue_config` event handler has been set to allow multiple
callbacks.
2023-11-02 18:11:17 -07:00
Tom Mairs f630cbaf09 comments to rebuild after an update 2023-11-02 23:08:53 +00:00
Wez Furlong e176920f74 docs: changelog for #91 2023-11-02 08:35:19 -07:00
Nathalie Caiandncai 8a918177d5 add minijinja_contrib dependency (#91)
Co-authored-by: ncai <ncai@chapsvision.com>
2023-11-02 08:28:57 -07:00
Wez Furlong ffc15f2f74 docs: smtp_client_rewrite_delivery_status: see also memoize 2023-11-01 13:45:05 -07:00
Wez Furlong 10bade362b docs: add version info to rewrite and regex set map 2023-11-01 13:41:34 -07:00
Wez Furlong 343411c3d2 Add event for rewriting a failed smtp response
This will only trigger for messages where we've gotten to MAIL FROM or
later, and where we got a non-250 response.

The event is passed the response + the domain, tenant, campaign and
routing domain.  It can return an alternative response code value
or `null` to indicate no change.
2023-11-01 13:25:15 -07:00
Wez Furlong 8942ef7f86 Add regex_set_map type
This is useful for efficiently matching a string simultaneously against
multiple regular expressions.

The motivating use case is to use it together with the
smtp_client_rewrite_delivery_status event.
2023-11-01 12:52:39 -07:00
Wez Furlong 2b0ef4820f move libunbound to its own repo
You can now find it over at https://github.com/KumoCorp/libunbound-rs
2023-10-30 11:19:17 -07:00
Wez Furlong e731d47c3e rocksdb: trim down the number of retained LOG files
The default is 1000 which is really high. This sets it to 10,
and configures rotation of those daily, so that there should be
no more than 10 days worth of LOG files.
2023-10-27 08:38:06 -07:00
Wez Furlong 8cbc60410d docs: update for dane 2023-10-26 17:03:00 -07:00
Wez Furlong 903189d873 add tls info to Delivery log records
refs: https://github.com/KumoCorp/kumomta/issues/39
refs: https://github.com/KumoCorp/kumomta/issues/8
2023-10-26 17:03:00 -07:00
Wez Furlong 9c951a444d kumod: Add enable_dane option
Adds the ability to enable DANE verification for SMTP.

It is disabled by default because correct operation also
requires working DNSSEC which in turns requires the use
of the unbound resolver.

When enabled, the SMTP client will use the OpenSSL DANE
implementation to verify the peer certificate.

refs: https://github.com/KumoCorp/kumomta/issues/8

```lua
local kumo = require 'kumo'

-- Called on startup to initialize the system
kumo.on('init', function()
  kumo.configure_accounting_db_path '/tmp/acct.db'

  kumo.dns.configure_unbound_resolver {
    options = {
      validate = true, -- enable DNSSEC validation
    },
    -- name_servers = { '1.1.1.1:53' },
  }

  kumo.set_diagnostic_log_filter 'rfc5321=trace,kumod::ready_queue=trace,kumod::smtp_dispatcher=trace,info'
  kumo.start_esmtp_listener {
    listen = '0.0.0.0:2025',
    relay_hosts = { '127.0.0.1', '192.168.1.0/24' },
  }

  kumo.configure_local_logs {
    log_dir = '/var/tmp/kumo-logs',
    max_segment_duration = '1s',
  }

  kumo.start_http_listener {
    listen = '0.0.0.0:8000',
    trusted_hosts = { '127.0.0.1', '::1' },
  }

  kumo.define_spool {
    name = 'data',
    path = '/var/tmp/kumo-spool/data',
    kind = 'RocksDB',
  }

  kumo.define_spool {
    name = 'meta',
    path = '/var/tmp/kumo-spool/meta',
    kind = 'RocksDB',
  }
end)

kumo.on('get_egress_pool', function(pool_name)
  if pool_name == 'pool0' then
    return kumo.make_egress_pool {
      name = 'pool0',
      entries = {
        { name = 'source0' },
      },
    }
  end

  error("I don't know how to configure pool " .. pool_name)
end)

kumo.on('get_egress_source', function(source_name)
  return kumo.make_egress_source {
    name = source_name,
    socks5_proxy_server = '127.0.0.1:5000',
    socks5_proxy_source_address = '0.0.0.0',
  }
end)

kumo.on(
  'get_egress_path_config',
  function(routing_domain, egress_source, site_name)
    print('get_egress_path_config', routing_domain, egress_source, site_name)
    return kumo.make_egress_path {
      enable_tls = 'Disabled',
      enable_dane = true,
      idle_timeout = '5s',
      data_timeout = '10s',
      data_dot_timeout = '15s',
      prohibited_hosts = {},
    }
  end
)

kumo.on('get_queue_config', function(domain, tenant, campaign)
  return kumo.make_queue_config {
    egress_pool = 'pool0',
  }
end)
```
2023-10-26 17:03:00 -07:00
Wez Furlong f3241dc3c7 docs: fix typo 2023-10-26 17:03:00 -07:00
Wez Furlong 0b3c258c28 rfc5321 client: add potential to use openssl for starttls
This code isn't currently reachable (defaults to false), but allows
for a runtime selection between rustls and openssl-based tls.

refs: https://github.com/KumoCorp/kumomta/issues/8
2023-10-26 17:03:00 -07:00
Wez Furlong 8936f001e9 openssl: point to our fork of the rust bindings
This will enable DANE support via openssl in a future commit
2023-10-26 17:03:00 -07:00
Wez Furlong 30731e480c unbreak integration tests
These were broken by 898e08aece
2023-10-26 17:02:30 -07:00
Mike Hillyer ff0869c162 Update TSA page in User Guide to show the use of extra_files 2023-10-26 11:22:16 -04:00
Mike Hillyer 898e08aece Update shaping.toml with feedback provided by mail.com 2023-10-26 08:35:42 -04:00
Wez Furlong fb2cdff808 docs: refine from-source info
Minimize the rust install section to just what is necessary.
Show where the rpms and debs get built, and how to install them.
2023-10-24 13:19:03 -07:00
Wez Furlong f8aaa6f133 cargo update 2023-10-24 11:23:14 -07:00
Wez Furlong 8efd4d5127 docs: clarify install-from-source docs
Avoid referencing implementation details that we may change
(install.sh).

Document what goes where with a bit of explanation.

Remove examples that show launching `kumod &` as we do not recommend
running the service in the background without some kind of supervision
(such as systemd).  Instead, show how how to run the process and leave
it to the reader to decide how they with to deploy it in their
environment.

closes: https://github.com/KumoCorp/kumomta/issues/72
2023-10-24 11:22:00 -07:00
Wez Furlong bec4bc1445 generate-toc.py: format 2023-10-24 11:19:55 -07:00
Tom Mairs 0b36fe4788 Add background/foreground instruction 2023-10-24 00:08:17 +00:00