Use dynamic fastfield codes for multivalues fixes (only sorting case covered)
Rename get to get_val due to conflict with Vec
use u64 precision instead of u32 for get_range, to allow use of existing fast field interface interface (actually not sure if it would be better have a different interface)
add CodecReader as common interface in fastfield codec crate
add LinearInterpolation to DynamicFastFieldReader
calc estimation and choose best codec
cleanup
support multiple codes
prepend codec id to all fast fields
add new api to create fastfields with access to all data
use new fastfield creation api in initial creation and merge
remove unused collect of data in doc_id_mapping
* prepare for multiple fastfield codecs
prepare for multiple fastfield codecs by wrapping the codecs in an enum #1042
* add FastFieldSerializer trait, add DynamicFastFieldSerializer
add FastFieldSerializer trait
add DynamicFastFieldSerializer enum to wrap all implementors of the FastFieldSerializer trait
* add estimation for fastfield bitpacker
Change Footer version handling, Make compression dynamic
Change Footer version handling
Simplify version handling by switching to JSON instead of binary serialization.
fixes#1058
Make compression dynamic
Instead of choosing the compression during compile time via a feature flag, you can now have multiple compression algorithms enabled and decide during runtime which one to choose via IndexSettings. Changing the compression algorithm on an index is also supported. The information which algorithm was used in the doc store is stored in the DocStoreFooter. The default is the lz4 block format.
fixes#904
Handle merging of different compressors
Fix feature flag names
Add doc store test for all compressors
* Detect if segments are stackackable with sorting, fixes#1038
Detect if segments are stackable when their data ranges on the sort property are disjunct.
Presort segments by thei min value on merge, to enable easier stacking.
* move code to function
* fix and refactor log merge policy, fixes#1035
fixes a bug in log merge policy where an index was wrongly referenced by its index
* cleanup
* fix sort order, improve method names
* use itertools groupby, fix serialization test
* minor improvments
* update names
* add iterator over documents in docstore
When profiling, I saw that around 8% of the time in a merge was spent in look-ups into the skip index. Since the documents in the merge case are read continuously, we can replace the random access with an iterator over the documents.
Merge Time on Sorted Index Before/After:
24s / 19s
Merge Time on Unsorted Index Before/After:
15s / 13,5s
So we can expect 10-20% faster merges.
This iterator is also important if we add sorting based on a field in the documents.
* Update reader.rs
Co-authored-by: Paul Masurel <paul@quickwit.io>
