Wez Furlong
b8310da8be
TL;DR: you can easily halve your system performance by logging headers vs. logging meta. This is one of those things that is easy to overlook or forget, but: whenever you need to operate on the message data, rather than its metadata, the aggregate cost is high. In this case, we were recently troubleshooting a system where the CPU was bogged down and we traced it to the logging configuration: a number of message headers were being logged in a configuration that made heavy use of throttles and limits in its traffic shaping, and thus had a large number of Delayed and TransientFailure events being written to the logs. When logging headers, each one of those events requires loading the message from the spool and parsing out the headers. When the average message size is ~100KB this imposes a notable overhead on the CPU and IO utilization of the system. What we recommend instead of logging headers directly is capturing the information that you want to log into the message metadata at the time that the message is received. The message meta is usually already loaded, but is also typically much smaller and easier to decode than the full message content in the cases where it is not loaded. As a result, it is much cheaper to log meta than to log headers. This commit adds some warnings and cross links to help folks be aware of this, and to generally navigate related meta and logging topics more easily via tags.
1.7 KiB
tags
| tags | |
|---|---|
|
headers
Specify a list of message headers to include in the logs. The default is empty.
!!! warning While logging headers directly is convenient and easy to express in the logging configuration, it comes with additional runtime CPU and IO overhead: every discrete event that is logged will result in the message contents being loaded from spool (if they are not already loaded), the message headers being parsed, and the selected headers decoded to be logged.
If your system has CPU and IO to spare, this is a non-issue, but
if you are pushing your system to its limits, and especially if
you have a large scheduled queue with lots of throttled or otherwise
delayed messages, these overheads can dominate the system and harm
overall throughput.
We recommend instead using
[msg:import_x_headers()](../../message/import_x_headers.md) during message
reception to cache a copy of the headers that you desire to log into your
message metadata, then listing those metadata fields in your logger
[meta](meta.md) list *instead* of using `headers`. This will dramatically
reduce the IO and CPU overheads around logging.
kumo.configure_local_logs {
-- You can log headers like this, but it is not recommended!
-- You generally should prefer to log `meta` instead.
headers = { 'Subject' },
}
Please consider using meta rather than headers!
{{since('2023.12.28-63cde9c7', indent=True)}}
Header names can now use simple wildcard suffixes; if the last character
of the header name is * then it will match any string with that prefix.
For example "X-*" will match any header names that start with "X-".