A message received via inter-node transfer, on systems hosted on AWS, could
end up with a wildly incorrect far-future timestamp. The underlying mac_address
crate would pick a NIC with the same MAC address as other AWS instances in the
cluster, and that triggered a code path where the collision resolving logic
misinterpreted the timestamp portion of the incoming spool id.
The thing that made this painful was the logic in spool_id.rs: it
misinterpreted the subsecond portion of the timestamp extracted from
the uuid, and due to the way that that field wraps, could produce wildly
inaccurate deltas with a huge multiplier.
As a belt and suspenders treatment, allow the user to influence which
MAC address is selected via the new KUMO_MAC_INTERFACE and
KUMO_MAC_ADDRESS environment variables.
I factored out the sample outputs and used an include to pull
in that content.
In the future, we could automate creating those sample data files
based on the live instance, however, it will need some thought
as many metrics are instantiated dynamically. That means that
simply launching the server to dump the initial counters could
miss a large number of the main production counters that are
the most interesting. Likely we'll need something smarter than
that approach. But that is an issue for the future!
This reverts commit 82a258a92e0e0eef6c7e483ab25f71146cdfeea3; that new
page is incomplete and has invalid syntax.
Let's take it out for now to unblock the doc build.
Wez Furlong
Mike Hillyer
Blake Patton