This implements the file list dump feature, where we dump the file list
by hour after compaction into parquet files and use those files while
searching through files lists. This also adds following variables -
- ZO_FILE_LIST_DUMP_ENABLED : if this dumping is enabled or not, default
is false
- ZO_FILE_LIST_DUMP_DUAL_WRITE : should the file list entries be kept in
the file list even after dumping. default is true, recommended is false.
Default can be changed after more verification.
- ZO_FILE_LIST_DUMP_MIN_HOUR : number of hours in past, only files older
than this will be considered for dumping, to avoid hot files issues.
The dump files entries themselves are stored in the file_list table as
was decided, so they also become a part of the normal compactor flow.
However, there is check to ensure these files are not dumped.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
