We commit close segments when the indexer heap is close to its capacity.
(currently we use a limit of 10_000_000).
Because we do this check before indexing a document, and before
also because serialization starts by closing the postingswriter, and
therefore all of the recorders open for the last document, we may still
overflow the heap.
We don't want to resize the heap because we may have references to objects
in the current heap.
Because of that, heap are actually chained list.
In an ideal settings, the limit should work fine and this overflow behavior should
never be activated.
The fst crate recently added support for sliced `Arc<Vec<u8>>`.
This called for a rewrite of the RAMDirectory for tantivy's RAMDirectory.
Previously every single read was copying data.
In addition:
- RAMDirectory's Write object panic if someone does not flush
right before the destruction of the object.
- In the same spirit, the postings serializer panics if someone
opens a term without closing the previous one.
Closes#16
