Commit Graph

79 Commits

Author SHA1 Message Date
Mike Hillyer 475e28fbb5 Update logging documentation. 2023-04-07 21:30:40 -04:00
Wez Furlong 9225644ce7 export ready queue size
```
  "ready_count": {
    "help": "number of messages in the ready queue",
    "type": "gauge",
    "value": {
      "service": {
        "smtp_client:source1->loopback.dummy-mx.example.com": 46.0,
        "smtp_client:source2->loopback.dummy-mx.example.com": 152.0,
      }
    }
  },
```

closes: #30
2023-04-03 10:14:16 -07:00
Wez Furlong 01f125d8ad docs: show how queue name varies based on meta values 2023-03-27 18:10:01 -07:00
Wez Furlong e87a5b18bb Improve performance of finding CRLF.CRLF for large messages by 20x
Using a better tuned algorithm for finding a substring match removes
a CPU bottleneck and improves message throughput by 20x at 100Kb
message sizes.
2023-03-27 09:39:51 -07:00
Wez Furlong ece1b89b00 add admin HTTP API for changing the log level 2023-03-24 10:04:00 -07:00
Wez Furlong 6740a68fad add kumo.set_diagnostic_log_filter for adjusting log levels via config 2023-03-24 09:02:10 -07:00
Wez Furlong 3a26c60438 also export counters as json
expand the docs to show what the data looks like
2023-03-23 10:47:17 -07:00
Wez Furlong cf1ba5f0b2 add key_data source for key data, refactor docs
Break out the KeySource object into its own reference page to
avoid manually duplicating the same information into multiple pages.

Explain how to use data fetched from eg: sqlite.
2023-03-23 08:27:14 -07:00
Wez Furlong 36de652dd2 docs: fix indent 2023-03-22 17:05:03 -07:00
Wez Furlong 0167041ed0 add kumo.memoize as a way to add caching to any lua function 2023-03-22 17:01:07 -07:00
Wez Furlong f9c6eb8ce3 add ed25519 dkim signer 2023-03-16 10:23:54 -07:00
Wez Furlong 09a251d5a1 docs: adjust queue example to show composing domain and source params 2023-03-15 10:13:56 -07:00
Wez Furlong b43e88985c add kumo.dns.lookup_mx
This function returns MX information, including the computed site_name.
Update the queue example to show how to use it to populate the SITE_OF
map used in that example.

This commit switches to using the `Name` type under the covers to ensure
that we use canonical names for our cache keys.
2023-03-15 09:34:23 -07:00
Wez Furlong be0fd76d0b add ehlo_domain config option 2023-03-14 22:26:07 -07:00
Wez Furlong e1f0939d81 add json functions 2023-03-14 20:07:40 -07:00
Wez Furlong 384da92c0e docs: update dkim signing to use header from 2023-03-14 18:53:47 -07:00
Wez Furlong b3f3f877df add functions for working with address headers 2023-03-14 18:53:47 -07:00
Mike Hillyer af04bd7ce0 Update time notation and example script. 2023-03-14 19:12:46 -04:00
Wez Furlong 4c71090669 docs: standardize the header before the generated list of functions 2023-03-14 10:17:55 -07:00
Wez Furlong b56bed5060 docs: fix admonish syntax, explain the global variable thing better 2023-03-11 23:06:59 -07:00
Wez Furlong 269e0702b4 add SMTP AUTH PLAIN support
Verified via:

```
swaks ... --auth plain --tls --auth-user scott --auth-password tiger
```
2023-03-11 16:23:49 -07:00
Mike Hillyer 2a4c53f953 Removing extra trailing line 2023-03-10 23:27:42 -05:00
Wez Furlong 258d9f8f64 docs: add a bit of an explanation about queueing 2023-03-07 08:27:44 -07:00
Wez Furlong 96350031dc add index pages for redis and sqlite 2023-03-06 20:19:16 -07:00
Wez Furlong 7054fb31c9 switch to using human time for the queue related time intervals 2023-03-06 19:33:05 -07:00
Wez Furlong 00c253a9dd add maildir support
This is to facilitate end-to-end testing to validate that the
right bits come out at the end.
2023-03-06 19:05:49 -07:00
Wez Furlong cd5f7efbb0 docs: link to ARF rfc, add note about extensions: 2023-03-05 20:37:45 -07:00
Wez Furlong c6292440d0 docs: fix a typo and add a missing paragraph 2023-03-05 20:34:29 -07:00
Wez Furlong 1bfa3a42af Add custom logging 2023-03-05 20:29:14 -07:00
Wez Furlong a0e73b72b1 allow logging selected meta and headers to the logs 2023-03-05 15:58:19 -07:00
Wez Furlong 1ac929e6b2 add max_segment_duration option to the logger 2023-03-05 15:25:08 -07:00
Wez Furlong 0ca585bd5a Optionally use redis for throttles 2023-03-05 10:39:56 -07:00
Wez Furlong ca78839a59 docs: add missing require 2023-03-05 09:59:06 -07:00
Wez Furlong eaaddbb6b6 add redis client 2023-03-05 09:57:03 -07:00
Wez Furlong 2cf3d2387c add sqlite data access
A bit of a big commit because I wanted sqlite queries to run
async and there isn't a sane way to tell in mlua whether a
given context is safe to yield for an await, which meant making
every case where we call into lua be fully async.

This adds a simple but powerful binding to sqlite that performs
its IO in a blocking tokio thread pool, and returns the results
back to lua.

There is no implicit caching of queries, and no implicit connection
pooling.
2023-03-04 18:07:31 -07:00
Wez Furlong c184214253 allow loading TLS key+cert from hashicorp vault
refactor to allow reusing the same key loader logic for dkim, smtp
and http.
2023-03-04 11:01:35 -07:00
Wez Furlong 38e60f9274 rename vault object fields to have vault_ prefix
This disambiguates for future alternative credential storage
systems.
2023-03-04 10:29:25 -07:00
Wez Furlong c93c9af2eb refine default list of meta fields for supplemental trace header 2023-03-04 10:21:45 -07:00
Wez Furlong 52f7ab6150 add prohibited_hosts and skip_hosts for outbound resolving
While in that part of the code, handle null MX destination domains
(such as `example.com) more nicely.
2023-03-04 10:09:17 -07:00
Wez Furlong 7eb33d2ca5 make smtp server read timeout configurable
client_timeout is the config, and it defaults to 1m
2023-03-04 07:52:53 -07:00
Wez Furlong 6204642270 add trace headers configuration and link to ARF parsing 2023-03-04 07:22:02 -07:00
Wez Furlong 61f974d066 adjust listener domains config to support more relaying options
a domain can now specify:

* relay_from: mail from the domain is allowed to relay anywhere
  if the peer matches the cidr list
* relay_to: mail from anywhere is allowed to relay to the domain
* log_arf: ARF FBL reports are allowed to be received and their
  contents will be logged
* log_oob: oob bounces are allowed to be received and their
  contents will be logged
2023-03-03 22:53:56 -07:00
Wez Furlong 03c3484cd7 rename fbl -> arf
The spec is talking about ARF, so let's use that terminology
2023-03-03 14:34:06 -07:00
Wez Furlong 523492da26 docs: fixup dkim signing 2023-03-03 10:56:22 -07:00
Wez Furlong aa460f52d0 docs: fix typo 2023-03-03 10:34:23 -07:00
Wez Furlong 7d76b3d9d7 add support for retrieving signing keys from hashicorp vault 2023-03-03 10:32:16 -07:00
Wez Furlong ef7fa59355 synthesize OOB log records from OOB reports
* When logging the reception of a message, if we can parse it
  as an rfc3464 oob report, then synthesize OOB records for
  each recipient
* Add config to allow relaying for domains for which bounces
  are received, so that we can accept their message content.
* If the domain config allows bounces but not relaying, then
  the message will not be spooled or queued after reception;
  it will be dropped with no additional logging.
2023-03-03 06:53:33 -07:00
Wez Furlong da5f2433c0 add rfc3464 delivery status report parser 2023-03-02 14:46:30 -07:00
Wez Furlong d5b13a263b Add kumo.configure_bounce_classifier
and connect it to the logger
2023-03-02 08:59:03 -07:00
Wez Furlong 3d521a6468 fix reporting of bounce totals
A significant portion are bounced as they move through the queues,
not just those swept up in the direct application of the API call.

Pass in a shared hash map to track the counts and report on that
at the end of the call instead.
2023-03-01 19:57:47 -07:00