This commit removes the queue_name_for_config_change_purposes_only wart
from inside the readyq machinery. This field had a bunch of caveats
and could in some cases become stale if the originating scheduled
queue had aged out.
It's been replaced by a HashSet of Weak refs back to the associated
scheduled queues so that it is now possible to walk the reverse of
the fan-in graph when we need to.
and also in resolve-shaping-domain, for consistency.
We can use the message rate from the scheduled queue config to refine
the computed ceilings/constraints too, which is nice.
This can be computed for an EgressPathConfig and is useful to indicate
what the effective ceilings are for this egress path.
We return this from inspect-ready-q
We recently investigated an issue where a rocksdb had been damaged by
corrupting/removing SST files (it sounded like this was accidentally
self-inflicted by some backup/orchestration infrastructure) leaving the
system in a silently-broken state: writes just wouldn't make progress
and there were no error messages.
Inspecting the `/var/spool/kumomta/data/LOG` log file (which is a
readable text file) revealed messages like:
```
2026/06/12-14:55:31.884227 2875746 [ERROR] [db/compaction/compaction.cc:262] Unable to load table properties for file 29704 --- IO error: No such file or directory: While open a file for random read: /var/spool/kumomta/data/029704.sst: No such file or directory
2026/06/12-14:55:31.884311 2875746 [ERROR] [db/db_impl/db_impl_compaction_flush.cc:3385] Waiting after background compaction error: IO error: No such file or directory: While open a file for random read: /var/spool/kumomta/data/029704.sst: No such file or directory, Accumulated background error counts: 6363
```
This commit improves the observability in this situation by proactively
checking for error conditions:
1. The store() and remove() operations now use our own polling within
a deadline loop rather than spawning a blocking task and delegating
to rocksdb's blocking interface. This allows us to inspect the
background error count and be cancellable, safely respecting and
caller provided smtp max transaction duration.
2. All read and write operations check for IO and Corruption errors
and immediately latch an error state
3. The metrics monitoring task inspects and track background error
counts and latch us into an unhealthy state when the background
error state appears unhealthy and persistent.
4. Additional metrics are exposed to help monitoring and alerting
While adding integration test coverage for this, I found a typo that
meant that spool errors were ignored in the message crate; they got
silently converted to `true` in all cases rather than just mapping
the success case to a `true`.
Integration tests handle the case where an SST file is corrupted
(truncated) during runtime, as well as starting up when an SST file
is missing. These excercise both the foreground and background
error detection paths.
Pre-populate a `to_header` template substitution with the default
formatted `To` header for each recipient. Users can reference it via
`{{ to_header }}` and override it per-recipient in substitutions.
closes: https://github.com/KumoCorp/kumomta/pull/501
This dumps out a trace of all tokio tasks. It is quite expensive,
and currently unsettles the tokio runtime such that you need to
repeatedly call this endpoint in order for a subsequent graceful
shutdown to clock through and complete.
As part of this, we move template compilation to happen before
we queue up deferred generation; the compilation step should
be plenty fast enough that we can reasonably do that synchronously
and report any compilation errors back to the peer.
This crate queries machine/system information which is intended
to be used in system monitoring.
This information is not centrally collected or retained by kumocorp,
merely reported via an API endpoint which is accessible only to trusted
IPs.
Machine info tries to interrogate information about the running/hosting
cloud platform for the major cloud providers, as well as indicate
whether it is running in a container of some kind.
The full cloud information is not re-exported via the API endpoint at
this time, but a fingerprint that encodes things like the instance-id is
included.
I factored out the sample outputs and used an include to pull
in that content.
In the future, we could automate creating those sample data files
based on the live instance, however, it will need some thought
as many metrics are instantiated dynamically. That means that
simply launching the server to dump the initial counters could
miss a large number of the main production counters that are
the most interesting. Likely we'll need something smarter than
that approach. But that is an issue for the future!
A number of manually authored HTTP api docs can now be replaced
by the versions generated from code, making them more accurate
and less likely to drift away from the implementation as
future changes are made, so that's what this commit does.
Define, by convention, a tag like `kcli:bounce` to indicate that
a jsonschema path definition is linked to the `kcli bounce` subcommand.
This commit adds appropriate tags to each command, and teaches
the jsonschematodocs utility how to generate a boilerplate
paragraph containing the link.
This commit changes how the various http endpoints are registered;
previously we had to define the handler function in one place,
then in another, explicitly register that handler with a path
and operation type with the router. If we wanted to also export
documentation about that method, we would need to annotate the handler
with a duplicate of the path and method type.
There was a decent amount of boiler plate and it was very easy to omit
some of that registration information, or for it to potentially drift
or be mistranscribed.
This commit eliminates most of the boiler plate; now we require that
every endpoint be annotated with the utoipa::path macro, which
centralizes the handler, path and method type definition in one place.
To register the route and docs we now have a much simpler
`router_and_docs!` macro that can register both the with the axum router
and with the openapi docs machinery.
DRY when it comes to listing out components, request and response
bodies. We don't need to centrally list those if we're good
at annotating the path macros.
This commit removes a couple of schemas from the docs; that's
ok because the content of those was already inlined into the
respective endpoint docs anyway, so we're not losing anything,
just eliminating a redundant copy of the same information
on a second location.
We currently use a rather hacky embedding of rapidoc to provide a
generic browser around the jsonschema export from our API interface.
It's not great for a couple of reasons:
* The font sizes are tiny
* The documentation rapidoc produces is not indexable, being
generated by javascript when the browser loads. This also
prevents making proper links to the various doc pages
This commit introduces a little utility that we can use during
the doc build to translate the schema into documentation files
that can then be processed as normal by the build.
This commit does this just for kumod at this time, but we could
also add tsa daemon in the future if we expand its API surface.
Previously, if setting `content.headers["To"]` we'd generate one header
for that value, and one for the per-recipient `To` header value.
We now check to see if the former is present; if so, we'll skip
generating the per-recipient `To` header value.
We've been hoping that mkdocs-material will ship the much anticipated
search enhancements for some time, but it's time to recognize that
we need to do something to improve the search results with how
things work right now.
This is a big commit that changes the titles of the various pages
from the code-annotated synopsis to just the name of the function.
This makes it much easier now to match things like `kumo.reject`
directly, but `reject` remains awkward to find.
I think this is the best that we can do at this time.
A few functions have been annotated with the `status: deprecated` to
show as deprecated in the toc/nav (shows with a little trash can next
to the name).
These are almost identical to those used by smtp, the main difference
being that we don't include the Received header by default for
http injection, whereas we do for smtp in order to be compliant
with the relevant rfcs. No such rfcs exist for the injection
protocol, so we can pick a default that makes more sense for the
majority of our target audience.
In systems with very large numbers of queues, it will take an
effectively unbounded amount of time to produce the initial
summary of results if we were to wait for the bounce to be
applied to every queue.
Let's adjust the output of the kcli to indicate that it is async,
and update the docs which already suggested that the numbers would
be partial to indicate that you can only really consume the id
from the response.
There's big explanation about this in the included docs change.
The TL;DR is: we can now optionally queue the request and generate it
asynchronously wrt. the incoming injection request.
There's also a throttle that can be used to constrain this, because
the asynchronous submission disconnects the injection request from
any kind of back pressure that we'd otherwise have to indicate to
the injector that there might be load related issues.
Wez Furlong
Mike Hillyer
Harsh Jha