mirror of
https://github.com/mailscope/kumomta.git
synced 2026-09-12 21:42:13 +00:00
Wez Furlong
9e0b019155
For a sequence like `[BASE64 IP: IPADDR]` we were not previously able to see the base64 or IP address inside the square brackets. This commit adds a pre-processing step to strip out parentheticals surrounded by spaces (or the end of the string) so that our existing tokenization can take effect. I spent a few hours today trying a few different matching strategies, rewriting this crate 3 different ways to try to simplify the logic and lean more heavily on the regex or fancy_regex crate so that we can avoid special cases. While they all worked, they all were over an order of magnitude worse in terms of throughput and performance. The strategy here is to perform a very cheap SIMD-assisted test to see if we need to pre-process the string so that we can avoid the overhead of the additional string allocation and full processing step. The result is that the performance before and after this change is very close to neutral when the response contains no brackets, but 33% slower (and 50% throughput) when brackets are present. That's still 180MiB/s on my threadripper which is still decent for this application. While grubbing around in here today, I noticed that a couple of regexeps had typos: `(:?` when they should have been `(?:`, and I annotated more sub-expressions to be non-capturing as well.