The assets/policy-extras dir is now deployed to
`/opt/kumomta/share/policy-extras` and added to the require path,
so you can do:
```lua
local shaping = require 'policy-extras.shaping'
```
to pull it into your policy.
Ensure that we trigger an async compatible callback so that
we can use eg: DNS resolution and memoization when processing it.
Fix the site name so that it doesn't include "source->site" and is
just the pure site name.
In some cases (eg: initial spool -> expiration) we don't have
any metadata loaded, causing the logs to say that the metadata
wasn't loaded.
This commit loads it as needed.
When given an extension/tagged domain, we didn't include the closing
square bracket when rendering it as text.
Note that this looks like an IPv6 address, but it is not a conforming
IPv6 address which looks like `[IPv6:aaaa::bbbb]` rather than
`[aaaa:bbbb]`. The latter uses tag "aaaa" to define a different
addressing namespace that has no known implementation.
<https://datatracker.ietf.org/doc/html/rfc5321#section-4.1.3>
it pulls in an old version of the time crate that flags many
rust projects as being vulnerable to an old CVE.
We still have other references to time that I'll remove in a follow up.
A couple of issues in here:
1. a copypasta incident meant that the v6 lookup was really just a v4
lookup. doh!
2. We would lookup v4 addresses and just be satisfied with those,
ignoring any v6 addresses
We now run the v4 and v6 lookups concurrently and merge the results
together.
We do still list the v4 addresses first in the list of addresses.
We don't try to be smart about using the "right" addresses when
work through the list; we try them all until one works out.
Correctly guessing at which one will work means replicating logic
in the kernel, and would potentially be screwed up when we add in
HAProxy support later.
It doesn't do much more than act as a glorified persistent
cursor at the moment, but the plumbing will enable calling
out to eg: lua or an arbitrary subprocess to process batches
in the future.
This commit adjusts the logger:
* When a log segment is finised, the `w` permission bits are removed.
This is used to signal to the tailer that there will be no additional
data written to that file.
* On startup, kumod will remove the `w` bits from all files in the
configured log directories so that any that were left on by a prior
incarnation will be considered closed when the new generation is
started up.
* Add a `tailer` utility that will essentially `cat` all of the log
files in a given directory in the appropriate order, then go into
log tailing mode, where it will wait for new data to show up.
* When reading data, if it hits EOF, the tailer will wait around until
more data shows up, or until the `w` bit is removed from the file
that it is reading.
The idea is that, in a later commit, the tailer will be able to persist
a checkpoint file to record where it got to, and be able to dispatch
chunks of data to eg: a webhook push utility, advancing the checkpoint
only when the utility indicates success.
For now, running `cargo run -p tailer -- --tail /var/tmp/kumo-logs` is
a convenient way to tail the logs as they are written.
Break out the KeySource object into its own reference page to
avoid manually duplicating the same information into multiple pages.
Explain how to use data fetched from eg: sqlite.
We were binding as blobs which we could read back as strings,
but which didn't match string inputs! Very confusing!
Always bind as strings.
For the empty rows case: don't return the affected row count
unless the query has 0 columns, as we otherwise cannot distinguish
between no matching results and a single column with 0.
The issue in my case was:
```
2023-03-20T21:38:56.945962Z ERROR localset-0 kumod::mod_kumo: Error in spool: Opening spool data: Corruption: While creating a new Db, wal_dir contains existing log file: : 000004.log
Shutdown requested, please wait while work is saved
Shutdown completed OK!
2023-03-20T21:38:56.966846Z ERROR localset-0 kumod::mod_kumo: Error in spool: Opening spool meta: Corruption: While creating a new Db, wal_dir contains existing log file: : 000004.log
2023-03-20T21:38:57.025241Z ERROR localset-0 kumod: problem initializing: No spools have been defined
smtp listener on 0.0.0.0:25 -> stopping
Error: Initialization raised an error
smtp listener on [::]:25 -> stopping
```
This function returns MX information, including the computed site_name.
Update the queue example to show how to use it to populate the SITE_OF
map used in that example.
This commit switches to using the `Name` type under the covers to ensure
that we use canonical names for our cache keys.
Revert building static/bundled sqlite, as the logic deep in sqlite3-src
makes no attempt to deal with this mess on centos7:
= note: /opt/rh/devtoolset-9/root/usr/libexec/gcc/x86_64-redhat-linux/9/ld: /__w/kumomta/kumomta/target/release/deps/libsqlite3_src-b3c44e6f34aac065.rlib(sqlite3.o):(.data.rel.aSyscall+0xb0): undefined reference to `fcntl64'
collect2: error: ld returned 1 exit status
(I don't blame it!)
Deal with that in the docker image by simply installing sqlite
in the docker image.
And since I'm touching the docker stuff, adjust its install
path and layout to match that of the rpm and deb packages.
Unfortunately, the upstream bitstring-trees crate didn't expose
a way to query the tree (only iterate it), so I pulled in a copy
of the implementation and adjusted it for that purpose, removing
some unsafe code that replaced values during insertion (for performance)
with a simpler implementation that clones values instead.
With the map implementation in the crate, I adjusted the set
implementation to be a `CidrMap<()>` and dropped the bitstring-trees
dependency.
Improve the big-O cost of finding an entry by storing
the ranges in a compressed radix tree under the covers.
Make it a bit easier to work with the type in code while
we're in here.
"any" is now a possible value for cidrsets, representing
any value.
Wez Furlong