* fix windows build (#1)
* Fix windows build
* Add doc traits
* Add field value iter
* Add value and serialization
* Adjust order
* Fix bug
* Correct type
* Fix generic bugs
* Reformat code
* Add generic to index writer which I forgot about
* Fix missing generics on single segment writer
* Add missing type export
* Add default methods for convenience
* Cleanup
* Fix more-like-this query to use standard types
* Update API and fix tests
* Add doc traits
* Add field value iter
* Add value and serialization
* Adjust order
* Fix bug
* Correct type
* Rebase main and fix conflicts
* Reformat code
* Merge upstream
* Fix missing generics on single segment writer
* Add missing type export
* Add default methods for convenience
* Cleanup
* Fix more-like-this query to use standard types
* Update API and fix tests
* Add tokenizer improvements from previous commits
* Add tokenizer improvements from previous commits
* Reformat
* Fix unit tests
* Fix unit tests
* Use enum in changes
* Stage changes
* Add new deserializer logic
* Add serializer integration
* Add document deserializer
* Implement new (de)serialization api for existing types
* Fix bugs and type errors
* Add helper implementations
* Fix errors
* Reformat code
* Add unit tests and some code organisation for serialization
* Add unit tests to deserializer
* Add some small docs
* Add support for deserializing serde values
* Reformat
* Fix typo
* Fix typo
* Change repr of facet
* Remove unused trait methods
* Add child value type
* Resolve comments
* Fix build
* Fix more build errors
* Fix more build errors
* Fix the tests I missed
* Fix examples
* fix numerical order, serialize PreTok Str
* fix coverage
* rename Document to TantivyDocument, rename DocumentAccess to Document
add Binary prefix to binary de/serialization
* fix coverage
---------
Co-authored-by: Pascal Seitz <pascal.seitz@gmail.com>
* add memory limit for aggregations
introduce AggregationLimits to set memory consumption limit and bucket limits
memory limit is checked during aggregation, bucket limit is checked before returning the aggregation request.
* Apply suggestions from code review
Co-authored-by: Paul Masurel <paul@quickwit.io>
* add ByteCount with human readable format
---------
Co-authored-by: Paul Masurel <paul@quickwit.io>
* sort index by field
add sort info to IndexSettings
generate docid mapping for sorted field (only fastfield)
remap singlevalue fastfield
* support docid mapping in multivalue fastfield
move docid mapping to serialization step (less intermediate data for mapping)
add support for docid mapping in multivalue fastfield
* handle docid map in bytes fastfield
* forward docid mapping, remap postings
* fix merge conflicts
* move test to index_sorter
* add docid index mapping old->new
add docid mapping for both directions old->new (used in postings) and new->old (used in fast field)
handle mapping in postings recorder
warn instead of info for MAX_TOKEN_LEN
* remap docid in fielnorm
* resort docids in recorder, more extensive tests
* handle index sorting in docstore
handle index sort in docstore, by saving all the docs in a temp docstore file (SegmentComponent::TempStore). On serialization the docid mapping is used to create a docstore in the correct order by reader the old docstore.
add docstore sort tests
refactor tests
* refactor
rename docid doc_id
rename docid_map doc_id_map
rename DocidMapping DocIdMapping
fix typo
* u32 to DocId
* better doc_id_map creation
remove unstable sort
* add non mut method to FastFieldWriters
add _mut prefix to &mut methods
* remove sort_index
* fix clippy issues
* fix SegmentComponent iterator
use std::mem::replace
* fix test
* fmt
* handle indexsettings deserialize
* add reading, writing bytes to doc store
get bytes of document in doc store
add store_bytes method doc writer to accept serialized document
add serialization index settings test
* rename index_sorter to doc_id_mapping
use bufferlender in recorder
* fix compile issue, make sort_by_field optional
* fix test compile
* validate index settings on merge
validate index settings on merge
forward merge info to SegmentSerializer (for TempStore)
* fix doctest
* add itertools, use kmerge
add itertools, use kmerge
push because rustfmt fails
* implement/test merge for fastfield
implement/test merge for fastfield
rename len to num_deleted in DeleteBitSet
* Use precalculated docid mapping in merger
Use precalculated docid mapping in merger for sorted indices instead of on the fly calculation
Add index creation macro benchmark, but commented out for now, since it is not really usable due to long runtimes, and extreme fluctuations. May be better suited in criterion or an external bench bin
* fix fast field reader docs
fix fast field reader docs, Error instead of None returned
add u64s_lenient to fastreader
add create docid mapping benchmark
* add test for multifast field merge
refactor test
add test for multifast field merge
* add num_bytes to BytesFastFieldReader
equivalent to num_vals in MultiValuedFastFieldReader
* add MultiValueLength trait
add MultiValueLength trait in order to unify index creation for BytesFastFieldReader and MultiValuedFastFieldReader in merger
* Add ReaderWithOrdinal, fix
Add ReaderWithOrdinal to associate data to a reader in merger
Fix bytes offset index creation in merger
* add test for merging bytes with sorted docids
* Merge fieldnorm for sorted index
* handle posting list in merge in sorted index
handle posting list in merge in sorted index by using doc id mapping for sorting
reuse SegmentOrdinal type
* handle doc store order in merge in sorted index
* fix typo, cleanup
* make IndexSetting non-optional
* fix type, rename test file
fix type
rename test file
add type
* remove SegmentReaderWithOrdinal accessors
* cargo fmt
* add index sort & merge test to include deletes
* Fix posting list merge issue
Fix posting list merge issue - ensure serializer always gets monotonically increasing doc ids
handle sorting and merging for facets field
* performance: cache field readers, use bytes for doc store merge
* change facet merge test to cover index sorting
* add RawDocument abstraction to access bytes in doc store
* fix deserialization, update changelog
fix deserialization
update changelog
forward error on merge failed
* cache store readers to utilize lru cache (4x performance)
cache store readers, to utilize lru cache (4x faster performance, due to less decompress calls on the block)
* add include_temp_doc_store flag in InnerSegmentMeta
unset flag on deserialization and after finalize of a segment
set flag when creating new instances
Tantivy used to assume that all files could be somehow memory mapped. After this change, Directory return a `FileSlice` that can be reduced and eventually read into an `OwnedBytes` object. Long and blocking io operation are still required by they do not span over the entire file.
* Split Collector into an overall Collector and a per-segment SegmentCollector. Precursor to cross-segment parallelism, and as a side benefit cleans up any per-segment fields from being Option<T> to just T.
* Attempt to add MultiCollector back
* working. Chained collector is broken though
* Fix chained collector
* Fix test
* Make Weight Send+Sync for parallelization purposes
* Expose parameters of RangeQuery for external usage
* Removed &mut self
* fixing tests
* Restored TestCollectors
* blop
* multicollector working
* chained collector working
* test broken
* fixing unit test
* blop
* blop
* Blop
* simplifying APi
* blop
* better syntax
* Simplifying top_collector
* refactoring
* blop
* Sync with master
* Added multithread search
* Collector refactoring
* Schema::builder
* CR and rustdoc
* CR comments
* blop
* Added an executor
* Sorted the segment readers in the searcher
* Update searcher.rs
* Fixed unit testst
* changed the place where we have the sort-segment-by-count heuristic
* using crossbeam::channel
* inlining
* Comments about panics propagating
* Added unit test for executor panicking
* Readded default
* Removed Default impl
* Added unit test for executor
* Compute space usage of a Searcher / SegmentReader / CompositeFile
* Fix typo
* Add serde Serialize/Deserialize for all the SpaceUsage structs
* Fix indexing
* Public methods for consuming space usage information
* #281: Add a space usage method that takes a SegmentComponent to support code that is unaware of particular segment components, and to make it more likely to update methods when a new component type is added.
* Add support for space usage computation of positions skip index file (#281)
* Add some tests for space usage computation (#281)
