Commit Graph
248 Commits
Author SHA1 Message Date
Wez Furlong 75d0c6d75a deps: nth time's the charm with this openssl thing 2023-11-29 15:29:30 -07:00
Wez Furlong 7a13941352 deps: update openssl fork again 2023-11-29 12:38:10 -07:00
Wez Furlong b0a8e86884 deps: update openssl fork 2023-11-29 12:15:33 -07:00
Wez Furlong ec937b52ef Allow using arbitrary bounce classification labels
refs: https://github.com/KumoCorp/kumomta/issues/98
2023-11-29 12:09:05 -07:00
Wez Furlong 884be6e4bb cargo update 2023-11-28 10:46:28 -07:00
Wez Furlong c814b960f1 deps: libunbound to 1.19.0 2023-11-22 17:53:44 -07:00
Wez Furlong 0b87c020b4 add openapi.json metadata for http api
Adds `/api-docs/openapi.json` and `/rapidoc` endpoints to both
kumod and tsa-daemon.

The former exposes the subset of the API that is expressable
in the openapi schema as a json file that can be imported into
other tools.

The latter is a single-page web app that consumes the former
to provide an interactive API explorer.
We're using rapidoc for this, because I happen to think it looks
nicest and easiest to use, and we can integrate it into the docs
fairly nicely.

Which leads in nicely to say: I've integrated a read-only version
of rapidoc into the docs, and it even detects and adjusts to the
selected light/dark mode.

The `docs/update-openapi.sh` extracts the openapi.json data from
kumod and tsa-daemon and outputs to the correct place in the docs
directory structure to enable this.

refs: https://github.com/KumoCorp/kumomta/issues/96
2023-11-21 15:30:13 -07:00
Wez Furlong f4c6b7e81d cargo update 2023-11-14 14:52:59 -07:00
Wez Furlong 4cb3a4c4e5 cargo update
I had to force-push my dane branch of rust-openssl after a rebase,
which invalidated the commit ref that is included here in Cargo.lock.

Update the deps to allow folks without the stale commit ref to be able
to build.
2023-11-06 06:32:22 -07:00
Wez Furlong 546237d11c deps: trust-dns -> hickory
The project was rebranded relatively recently; switch
to the new version
2023-11-04 10:26:49 -07:00
Nathalie Caiandncai 8a918177d5 add minijinja_contrib dependency (#91)
Co-authored-by: ncai <ncai@chapsvision.com>
2023-11-02 08:28:57 -07:00
Wez Furlong 8942ef7f86 Add regex_set_map type
This is useful for efficiently matching a string simultaneously against
multiple regular expressions.

The motivating use case is to use it together with the
smtp_client_rewrite_delivery_status event.
2023-11-01 12:52:39 -07:00
Wez Furlong 2b0ef4820f move libunbound to its own repo
You can now find it over at https://github.com/KumoCorp/libunbound-rs
2023-10-30 11:19:17 -07:00
Wez Furlong e731d47c3e rocksdb: trim down the number of retained LOG files
The default is 1000 which is really high. This sets it to 10,
and configures rotation of those daily, so that there should be
no more than 10 days worth of LOG files.
2023-10-27 08:38:06 -07:00
Wez Furlong 9c951a444d kumod: Add enable_dane option
Adds the ability to enable DANE verification for SMTP.

It is disabled by default because correct operation also
requires working DNSSEC which in turns requires the use
of the unbound resolver.

When enabled, the SMTP client will use the OpenSSL DANE
implementation to verify the peer certificate.

refs: https://github.com/KumoCorp/kumomta/issues/8

```lua
local kumo = require 'kumo'

-- Called on startup to initialize the system
kumo.on('init', function()
  kumo.configure_accounting_db_path '/tmp/acct.db'

  kumo.dns.configure_unbound_resolver {
    options = {
      validate = true, -- enable DNSSEC validation
    },
    -- name_servers = { '1.1.1.1:53' },
  }

  kumo.set_diagnostic_log_filter 'rfc5321=trace,kumod::ready_queue=trace,kumod::smtp_dispatcher=trace,info'
  kumo.start_esmtp_listener {
    listen = '0.0.0.0:2025',
    relay_hosts = { '127.0.0.1', '192.168.1.0/24' },
  }

  kumo.configure_local_logs {
    log_dir = '/var/tmp/kumo-logs',
    max_segment_duration = '1s',
  }

  kumo.start_http_listener {
    listen = '0.0.0.0:8000',
    trusted_hosts = { '127.0.0.1', '::1' },
  }

  kumo.define_spool {
    name = 'data',
    path = '/var/tmp/kumo-spool/data',
    kind = 'RocksDB',
  }

  kumo.define_spool {
    name = 'meta',
    path = '/var/tmp/kumo-spool/meta',
    kind = 'RocksDB',
  }
end)

kumo.on('get_egress_pool', function(pool_name)
  if pool_name == 'pool0' then
    return kumo.make_egress_pool {
      name = 'pool0',
      entries = {
        { name = 'source0' },
      },
    }
  end

  error("I don't know how to configure pool " .. pool_name)
end)

kumo.on('get_egress_source', function(source_name)
  return kumo.make_egress_source {
    name = source_name,
    socks5_proxy_server = '127.0.0.1:5000',
    socks5_proxy_source_address = '0.0.0.0',
  }
end)

kumo.on(
  'get_egress_path_config',
  function(routing_domain, egress_source, site_name)
    print('get_egress_path_config', routing_domain, egress_source, site_name)
    return kumo.make_egress_path {
      enable_tls = 'Disabled',
      enable_dane = true,
      idle_timeout = '5s',
      data_timeout = '10s',
      data_dot_timeout = '15s',
      prohibited_hosts = {},
    }
  end
)

kumo.on('get_queue_config', function(domain, tenant, campaign)
  return kumo.make_queue_config {
    egress_pool = 'pool0',
  }
end)
```
2023-10-26 17:03:00 -07:00
Wez Furlong 0b3c258c28 rfc5321 client: add potential to use openssl for starttls
This code isn't currently reachable (defaults to false), but allows
for a runtime selection between rustls and openssl-based tls.

refs: https://github.com/KumoCorp/kumomta/issues/8
2023-10-26 17:03:00 -07:00
Wez Furlong 8936f001e9 openssl: point to our fork of the rust bindings
This will enable DANE support via openssl in a future commit
2023-10-26 17:03:00 -07:00
Wez Furlong f8aaa6f133 cargo update 2023-10-24 11:23:14 -07:00
Wez Furlong 4c36489681 First pass at accounting db
This keeps track of the volume of receptions and deliveries over time.
2023-10-12 20:43:58 -04:00
Wez Furlong 646edfaa87 fix an issue with dot-stuffing in outgoing smtp
This was a 2-in-1 issue really.

When a message line starts with a `.` we are required to encode it
by prefixing the line with an additional `.`, aka: "dot stuffing".

In our implementation of this we would unintentionally remap CRLF
to CR, and then fail to emit CRLF.CRLF at the end of the data
that we'd planned to send.

The result of this was that we're hang waiting for the remote host
to respond to our DATA, meanwhile, it was waiting for the end of
our data.  We'd sit there until either side reached their respective
timeout for the data phase.

This commit fixes up both parts of this bug.
2023-10-04 19:18:14 -07:00
drone ad3ba58dff cargo update. Includes a fix for a DoS in websocket support
I'm not sure if that actually affected us in practice due to
our authentication requirement on that endpoint.

refs: https://github.com/snapview/tungstenite-rs/issues/376
2023-09-25 09:35:29 -07:00
Wez Furlong 808464e713 mod-redis: skip tests if redis server isn't present 2023-09-17 10:26:29 -07:00
Wez Furlong 5538655fd5 allow building on macos
NOTE: macos is NOT a platform supported for production use!

This commit just enables basic build and test to facilitate
offline work when I'm travelling with a mac laptop.

There are important systems functions that are not implemented
in the macOS build.
2023-09-17 08:48:59 -07:00
Wez Furlong e7f20cf814 logs: allow using log_record in templates 2023-09-14 14:34:02 -07:00
Wez Furlong e4bb5d674d add msg:dkim_verify()
This method returns an array of AuthenticationResult reflecting
the verification status.

refs: https://github.com/KumoCorp/kumomta/issues/82
2023-09-14 13:03:41 -07:00
Wez Furlong 08089d8f18 http: allow tracing requests
Add `tower_http::trace=debug` to the diagnostic log filter to capture
tracing spans around each HTTP request.
2023-09-12 13:24:53 -07:00
Wez Furlong fa6acdf378 First run at a kafka integration
It compiles, but is otherwise completely untested as of yet.

Usage:

```lua
local producer = kumo.kafka.build_producer {
  ["bootstrap.servers"] = "localhost:9092",
}
producer:send {
  topic = "my.topic",
  payload = "my payload",
  -- how long to keep trying to submit to kafka
  -- before a lua error will be raised.
  -- This is the default.
  timeout = "1 minute",
}
```

refs: https://github.com/KumoCorp/kumomta/issues/32
2023-09-11 15:30:35 -07:00
Wez Furlong b62242bff7 unbound: do proper probing for funcs and headers
hopefully will fix centos 7 build
2023-09-08 14:08:04 -07:00
Wez Furlong 8948cb0334 tempfile: use consistent version across multiple crates 2023-09-08 11:34:19 -07:00
Wez Furlong 0a4c8aa39e dns-resolver: add ability to configure unbound
Adds a lua function (needs docs) to do basic configuration of
unbound, and a feature for dns-resolver that is really just to
make it more convenient for me to test and compare the two
resolvers via:

cargo nextest run -p dns-resolver --features live-dns-tests

vs:

cargo nextest run -p dns-resolver --features live-dns-tests,default-unbound
2023-09-02 16:55:15 -07:00
Wez Furlong 90f7ff4735 unbound: expose various config methods 2023-09-02 07:26:28 -07:00
Wez Furlong 04131edf6d unbound: add async query interface 2023-09-02 06:47:11 -07:00
Wez Furlong 3bbe7b59b7 add libunbound
This is sufficient to run a basic resolve and return some answers
2023-09-01 20:25:53 -07:00
Wez Furlong 9423f07b66 add: libunbound-sys
This crate vendors in the unbound sources and builds libunbound
for use as a DNS resolver.
2023-09-01 17:13:11 -07:00
Wez Furlong d3b4a191af MTA-STS: expose to path config, implement in smtp_dispatcher
This should be sufficient to enable this feature!

refs: https://github.com/KumoCorp/kumomta/issues/7
2023-08-31 08:54:12 -07:00
Wez Furlong b3c1117005 mta-sts: implement higher level policy fetch/cache 2023-08-31 08:10:36 -07:00
Wez Furlong 8b7776e34d mta-sts: parser/loader for policy dns and http info
refs: https://github.com/KumoCorp/kumomta/issues/7
2023-08-30 16:13:01 -07:00
Wez Furlong a9c8122675 dns-resolver: allow grabbing the resolver
Switch to using ArcSwap to manage owning the resolver; that makes
it a bit cheaper than the rwlock we were using before, and also
allows sharing out a reference to the resolver for use by other
crates that don't really need to take a direct dep on this one.
2023-08-30 08:53:55 -07:00
Wez Furlong 88eca8b838 Add invalid CRLF conformance option to SMTP listener
There's just one option for the whole of the message payload; we don't
distinguish between headers and body for this.

closes: https://github.com/KumoCorp/kumomta/issues/23
closes: https://github.com/KumoCorp/kumomta/issues/22
2023-08-28 18:38:19 -07:00
Wez Furlong 9340e4f71d mailparsing: remove mail-builder from integration tests 2023-08-28 11:45:48 -07:00
Wez Furlong c7eacb63b5 mailparsing: replace mail-parser dep 2023-08-28 11:36:21 -07:00
Wez Furlong 01996f428b mailparsing: replace mail_builder dep in http inject API 2023-08-28 10:19:33 -07:00
Wez Furlong 0eebbc6a17 mailparsing: Date header support 2023-08-28 09:35:48 -07:00
Wez Furlong 0f6bc3abee mailparsing: helper for building a message
This allows specifying plain text, html and attachments and
will combine them into appropriately structured multipart
elements.
2023-08-28 09:05:21 -07:00
Wez Furlong c04dab8a19 Update webpki deps
refs: https://github.com/briansmith/webpki/issues/69
refs: https://github.com/KumoCorp/kumomta/security/dependabot/13
2023-08-25 09:03:10 -07:00
Wez Furlong 4dadafe31d deps: uuid->1.4, cargo update 2023-08-25 09:00:43 -07:00
Wez Furlong 4f57158b6d mailparsing: add helpers for constructing mimeparts
Make it more convenient to build text and attachment parts
2023-08-25 08:55:30 -07:00
Wez Furlong 2ec6386b97 log-types: migrate mailparse -> mailparsing 2023-08-25 08:55:29 -07:00
Wez Furlong 81d5dd77da integration tests+maildir: replace mailparse with mailparsing
Migrate maildir to mailparsing, and update for the fanout in
the integration tests.
2023-08-25 08:55:29 -07:00
Wez Furlong f171ce7b51 maildir: update deps 2023-08-25 08:55:29 -07:00