Export the memory statistics for each spool database to prometheus for
charting and tracking.
Allow the hosting application to request a cache purge and set up a
monitor task to do that when memory usage is too high. That step will
print the memory that it reclaimed when it kicks in.
I noticed that some end to end tests are load shedding, and I think
it is because we initialize headroom to 0 and are racing with
the memory monitor thread startup.
Let's default to non-zero instead.
I think this is manifesting in CI because it has noisy neighbors
on the VM hosts.
I realized that we hadn't connected this up to the memory monitor,
so this commit does that.
I'd like to hook this up for every LruCacheWithTtl, but that
requires some more extensive changes.
I'll look that in a follow up commit.
This provides a simpler and easier to use option for dealing with broken
TLS.
The back story here is that sites with broken TLS ideally should result
in a TLS failure and return us to a clear text SMTP session that we
could use for the remainder if we are in opportunistic mode.
In practice, it's not that simple:
* The rustls implementation provides into_fallible() to enable this mode
of operation, but certain handshake errors result in the session being
unusable once the failed handshake has been communicated
* Openssl is perfectly capable of operating this way, but the Rust
ecosystem bindings do not expose the handshake failure in a way that
allows the clear text session to continue
If we wanted to allow the session to communicate with the failed host in
clear text we will need to make a new separate connection for this case.
There is a concern that this will bump a connection counter in a
reputation system implemented on the destination host, but it is
reasonable to wonder if a site that has broken TLS would be sufficiently
sophisticated to employ such a system.
If we did want to allow making a new connection to the same host, it is
a bit awkward to fit into our current mx host connection plan system.
So, with all that in mind, this commit tries a slightly different
approach:
* A new `remember_broken_tls = "3 days"` field in the egress path config
will enable an in-process cache to keep track of the site_names that
advertise STARTTLS and for which we failed to handshake, or failed to
re-EHLO after a handshake.
* That will cause the connection plan to move on to subsequent hosts, if
any. It won't cause us to re-attempt the current host.
* Subsequent connection attempts (which may be immediate if there are
additional hosts in the current session plan) will consult the
broken tls cache for the site, and use that to decide on the
availability of TLS
* If we previously encountered broken TLS, we will effectively treat
the site as not advertising TLS in EHLO, but will generate an
appropriate error message if TLS is Required.
Since we track the breakage in a cache with per-item TTLs, you
can set the duration as appropriate to your situation.
The cache will be cleared when the process is restarted.
There is no sharing of the cache between nodes.
This is a much lower complexity solution than the existing workaround
using TSA automation rules to override the site with Disabled TLS, and
will help to minimize the number of rules for that purpose in the TSA
daemon.
Note that this commit doesn't change any behavior; we can do a followup
commit to replace those TSA rules with this mechanism.
This reverts commit 401df30099.
The docker merge step is failing:
```
envconfig.Process: assigning PULLREQUEST_DRONE_PULL_REQUEST to Number: converting '' to type int. details: strconv.ParseInt: parsing "": invalid syntax
```
It seems that the compatibility change in
https://github.com/woodpecker-ci/woodpecker/pull/3939 isn't sufficient.
Let's see if we can still run with the environment set manually,
or if we'll need to blackjack our own woodpecker plugin for this.
If you have a get_queue_config event handler that raises an error, it
will propagate back to the injector as a transient error, which is the
desired behavior.
However, we didn't ensure that the message was removed from the spool in
that situation, which could lead to duplicate sends of the same logical
message.
This commit makes things more robust in the context of this kind of
local configuration issue:
* QueueManager::insert_or_unwind is now the preferred way to handle
insertion of the message into the queue during reception at the point
of ingress. It will take care to remove the message from the spool
and to adjust the accounting by logging a Bounce record to complement
the Reception that it assumes that its caller has just logged for the
message.
* Both SMTP and HTTP listeners have been updated to use
insert_or_unwind.
* Spool enumeration has been updated to count failed inserts and to
summarize that the failure is serious and that the messages are not
going to go anywhere until the configuration issue is addressed and
the server is restarted. Note that we cannot recover from this while
the server is online because we have to perform a full spool
enumeration in order to re-discover those messages. (We could
potentially build a thing to remember the complete list of messages,
but I'd rather not try to address what is essentially a configuration
deployment validation problem with engineering on this end of things.)
Dirkjan Ochtman
Mike Hillyer
Mike Hillyer
Wez Furlong
Nathalie Caiandncai