This is useful if your hooks have interior logic to decide to
filter out a given record. For example, you might have multiple
hook endpoints but only messages with certain headers/metadata
should be routed to any one of them for a specific event.
Without the ability to pre-filter, we need to pay the cost of
spooling the event speculatively, and then skipping it when
processing logging for the (hopefully!) batch.
Using pre-filter you can cut out that overhead.
This is not yet documented; we're getting this in to get
some feedback before we finalize this interface.
Previously, we added `batch_size` which acts as a maximum batch
size; this was the simplest possible way to achieve batching
with a constrained upper bound with no additional latency.
This commit introduces a minimum batch size and a maximum latency.
The behavior for delivery handlers that configure min_batch_size > 1
is that once we've started a batch, we'll allow for up to
max_batch_latency time to pass to accumulate more messages into
the batch before we just go ahead and send it.
If you wanted to batch say 5,000-10,000 messages per batch or whatever
you get in a 10 second time period you would set:
* min_batch_size = 5000
* batch_size = 10000
* max_batch_latency = "10s"
Note that the default idle_timeout in the product is "5s" so if you want
max_batch_latency to be effective you will also need to increase the
idle_timeout in your shaping configuration/egress path config for that
handler.
This really is adding batching support to custom lua delivery
protocol handlers, but the main use case for these today is
to implement log hooks.
The way that it works is that you can specify a `batch_size`
as part of setting up the lua protocol handler.
Then, when it is time to send messages, if the batch_size is
the default of 1, the lua delivery logic will invoke the `send` method
on the connection object returned from the constructor. This
is the same as the behavior from before this commit.
However, if the batch_size is greater than 1, then the lua delivery
logic will instead attempt to collect up to batch_size messages
that are immediately available from the ready queue, and then pass
those to a new `send_batch` method.
The send_batch method accepts an array of messages; that array will
always have at least one message, and up to batch_size messages,
depending on the throughput and queue size.
If the send_batch method's return value applies equally to all
messages in the batch, so if it indicates that something failed,
that disposition will apply to all messages.
One of the reasons that I'd avoided implementing batching thus far
was that it makes it awkward to resolve persistent/recurring issues
that are due to a single message in that batch. If the batch is
always retried together then there is a good chance that it will
always fail together.
There's no explicit mitigation for that issue here, but it may
be probablistically mitigated by the jitter that is applied to
messages that transiently fail. If a batch transiently fails,
each message in that batch will be subject to its own random
jitter which should cause an offending message to be retried
with a different subset of messages next time around.
The integration test included here demonstrates the batching
working with an http log hook implementation.
* Patch bug in log_hooks.lua
connection:close was inserted in the wrong place and should be inside the constructor function with the connection object.
This will perform more explicit checks to make sure that the name
is defined and so on. It also now will check for conflicting
log hook names as well.
refs: #211
The way we test whether logs should continue is by looking for
an empty return statement, so we shouldn't `return nil` for these
otherwise we'll indicate that we definitively can't resolve the
queue configuration.
Wez Furlong
Laurent Marchaud
aryeh
ncai