first stab

* TopDocs: ensure stable sorting on equal score

When selecting the top K documents by score, we need to ensure stable
sorting. Until now, for documents with the same score, we were relying
on the (arbitrary) order returned by the BinaryHeap used to implement
the collectors.

This patch fixes the problem by explicitly using the doc address when
harvesting the `TopSegmentCollector` and when merging the results in
`TopCollector::merge_fruits()`.

This is important (for example) to implement pagination correctly using
the TopDocs collector. If sorting isn't stable, documents that have the
same score might be ranked in different positions depending on the
specific K that was used, thus appearing in two different pages, or in
none at all.

Fixes gh-671

* TMP: alternative solution (see previous commit)

If we add the constrait that D is also PartialOrd in ComparableDoc<T,
D>, then we can move the comparison by doc address directly in the cmp
implementation of ComparableDoc.

* TMP rebase as first commit: add benchmarks for TopSegmentCollector

* fixup! TMP: alternative solution (see previous commit)

* TMP add changelog entry

* TMP run cargo fmt

CHANGELOG.md

@@ -7,7 +7,9 @@ Tantivy 0.11.0
     - Better handling of whitespaces.
 - Closes #498 - add support for Elastic-style unbounded range queries for alphanumeric types eg. "title:>hello", "weight:>=70.5", "height:<200" (@petr-tik)
 - API change around `Box<BoxableTokenizer>`. See detail in #629
-- Avoid rebuilding Regex automaton whenever a regex query is reused. #630 (@brainlock)
+- Avoid rebuilding Regex automaton whenever a regex query is reused. #639 (@brainlock)
+- Add footer with some metadata to index files. #605 (@fdb-hiroshima)
+- TopDocs collector: ensure stable sorting on equal score. #671 (@brainlock)
  
 ## How to update?
 
@@ -15,6 +17,12 @@ Tantivy 0.11.0
 - Regex are now compiled when the `RegexQuery` instance is built. As a result, it can now return
 an error and handling the `Result` is required.
 
+
+Tantivy 0.10.2
+=====================
+
+- Closes #656. Solving memory leak.
+
 Tantivy 0.10.1
 =====================
 

Cargo.toml

@@ -13,15 +13,15 @@ keywords = ["search", "information", "retrieval"]
 edition = "2018"
 
 [dependencies]
-base64 = "0.10.0"
+base64 = "0.11.0"
 byteorder = "1.0"
+crc32fast = "1.2.0"
 once_cell = "1.0"
 regex ={version = "1.3.0", default-features = false, features = ["std"]}
 tantivy-fst = "0.1"
 memmap = {version = "0.7", optional=true}
 lz4 = {version="1.20", optional=true}
 snap = {version="0.2"}
-derive_builder = "0.7"
 atomicwrites = {version="0.2.2", optional=true}
 tempfile = "3.0"
 log = "0.4"
@@ -31,10 +31,10 @@ serde_json = "1.0"
 num_cpus = "1.2"
 fs2={version="0.4", optional=true}
 itertools = "0.8"
-levenshtein_automata = "0.1"
+levenshtein_automata = {version="0.1", features=["fst_automaton"]}
 notify = {version="4", optional=true}
 bit-set = "0.5"
-uuid = { version = "0.7.2", features = ["v4", "serde"] }
+uuid = { version = "0.8", features = ["v4", "serde"] }
 crossbeam = "0.7"
 futures = "0.1"
 futures-cpupool = "0.1"
@@ -82,7 +82,7 @@ unstable = [] # useful for benches.
 wasm-bindgen = ["uuid/wasm-bindgen"]
 
 [workspace]
-members = ["query-grammar", "incremental-search"]
+members = ["query-grammar"]
 
 [badges]
 travis-ci = { repository = "tantivy-search/tantivy" }

Makefile

@@ -1,3 +1,3 @@
 test:
 	echo "Run test only... No examples."
-	cargo test --all --tests --lib
+	cargo test --tests --lib

README.md

@@ -21,9 +21,9 @@
 [![Become a patron](https://c5.patreon.com/external/logo/become_a_patron_button.png)](https://www.patreon.com/fulmicoton)
 
 
-**Tantivy** is a **full text search engine library** written in rust.
+**Tantivy** is a **full text search engine library** written in Rust.
 
-It is closer to [Apache Lucene](https://lucene.apache.org/) than to [Elasticsearch](https://www.elastic.co/products/elasticsearch) and [Apache Solr](https://lucene.apache.org/solr/) in the sense it is not
+It is closer to [Apache Lucene](https://lucene.apache.org/) than to [Elasticsearch](https://www.elastic.co/products/elasticsearch) or [Apache Solr](https://lucene.apache.org/solr/) in the sense it is not
 an off-the-shelf search engine server, but rather a crate that can be used
 to build such a search engine.
 
@@ -31,7 +31,7 @@ Tantivy is, in fact, strongly inspired by Lucene's design.
 
 # Benchmark
 
-Tantivy is typically faster than Lucene, but the results will depend on 
+Tantivy is typically faster than Lucene, but the results depend on 
 the nature of the queries in your workload.
 
 The following [benchmark](https://tantivy-search.github.io/bench/) break downs 
@@ -40,19 +40,19 @@ performance for different type of queries / collection.
 # Features
 
 - Full-text search
-- Configurable tokenizer. (stemming available for 17 latin languages. Third party support for Chinese ([tantivy-jieba](https://crates.io/crates/tantivy-jieba) and [cang-jie](https://crates.io/crates/cang-jie)) and [Japanese](https://crates.io/crates/tantivy-tokenizer-tiny-segmenter)
+- Configurable tokenizer (stemming available for 17 Latin languages with third party support for Chinese ([tantivy-jieba](https://crates.io/crates/tantivy-jieba) and [cang-jie](https://crates.io/crates/cang-jie)) and [Japanese](https://crates.io/crates/tantivy-tokenizer-tiny-segmenter))
 - Fast (check out the :racehorse: :sparkles: [benchmark](https://tantivy-search.github.io/bench/) :sparkles: :racehorse:)
 - Tiny startup time (<10ms), perfect for command line tools
-- BM25 scoring (the same as lucene)
-- Natural query language `(michael AND jackson) OR "king of pop"`
-- Phrase queries search (`"michael jackson"`)
+- BM25 scoring (the same as Lucene)
+- Natural query language (e.g. `(michael AND jackson) OR "king of pop"`)
+- Phrase queries search (e.g. `"michael jackson"`)
 - Incremental indexing
 - Multithreaded indexing (indexing English Wikipedia takes < 3 minutes on my desktop)
 - Mmap directory
-- SIMD integer compression when the platform/CPU includes the SSE2 instruction set.
-- Single valued and multivalued u64, i64 and f64 fast fields (equivalent of doc values in Lucene)
+- SIMD integer compression when the platform/CPU includes the SSE2 instruction set
+- Single valued and multivalued u64, i64, and f64 fast fields (equivalent of doc values in Lucene)
 - `&[u8]` fast fields
-- Text, i64, u64, f64, dates and hierarchical facet fields
+- Text, i64, u64, f64, dates, and hierarchical facet fields
 - LZ4 compressed document store
 - Range queries
 - Faceted search
@@ -61,43 +61,42 @@ performance for different type of queries / collection.
 
 # Non-features
 
-- Distributed search is out of the scope of tantivy. That being said, tantivy is meant as a
+- Distributed search is out of the scope of Tantivy. That being said, Tantivy is a
 library upon which one could build a distributed search. Serializable/mergeable collector state for instance, 
-are within the scope of tantivy.
+are within the scope of Tantivy.
 
 # Supported OS and compiler
 
-Tantivy works on stable rust (>= 1.27) and supports Linux, MacOS and Windows.
+Tantivy works on stable Rust (>= 1.27) and supports Linux, MacOS, and Windows.
 
 # Getting started
 
-- [tantivy's simple search example](https://tantivy-search.github.io/examples/basic_search.html)
-- [tantivy-cli and its tutorial](https://github.com/tantivy-search/tantivy-cli).
-`tantivy-cli` is an actual command line interface that makes it easy for you to create a search engine,
-index documents and search via the CLI or a small server with a REST API.
-It will walk you through getting a wikipedia search engine up and running in a few minutes.
-- [reference doc for the last released version](https://docs.rs/tantivy/)
+- [Tantivy's simple search example](https://tantivy-search.github.io/examples/basic_search.html)
+- [tantivy-cli and its tutorial](https://github.com/tantivy-search/tantivy-cli) - `tantivy-cli` is an actual command line interface that makes it easy for you to create a search engine,
+index documents, and search via the CLI or a small server with a REST API.
+It walks you through getting a wikipedia search engine up and running in a few minutes.
+- [Reference doc for the last released version](https://docs.rs/tantivy/)
 
 # How can I support this project?
 
 There are many ways to support this project. 
 
-- Use tantivy and tell us about your experience on [gitter](https://gitter.im/tantivy-search/tantivy) or by email (paul.masurel@gmail.com)
+- Use Tantivy and tell us about your experience on [Gitter](https://gitter.im/tantivy-search/tantivy) or by email (paul.masurel@gmail.com)
 - Report bugs
 - Write a blog post
 - Help with documentation by asking questions or submitting PRs
-- Contribute code (you can join [our gitter](https://gitter.im/tantivy-search/tantivy) )
-- Talk about tantivy around you
+- Contribute code (you can join [our Gitter](https://gitter.im/tantivy-search/tantivy))
+- Talk about Tantivy around you
 - Drop a word on on [![Say Thanks!](https://img.shields.io/badge/Say%20Thanks-!-1EAEDB.svg)](https://saythanks.io/to/fulmicoton) or even [![Become a patron](https://c5.patreon.com/external/logo/become_a_patron_button.png)](https://www.patreon.com/fulmicoton)
 
 # Contributing code
 
-We use the GitHub Pull Request workflow - reference a GitHub ticket and/or include a comprehensive commit message when opening a PR.
+We use the GitHub Pull Request workflow: reference a GitHub ticket and/or include a comprehensive commit message when opening a PR.
 
 ## Clone and build locally
 
-Tantivy compiles on stable rust but requires `Rust >= 1.27`.
-To check out and run tests, you can simply run :
+Tantivy compiles on stable Rust but requires `Rust >= 1.27`.
+To check out and run tests, you can simply run:
 
 ```bash
     git clone https://github.com/tantivy-search/tantivy.git
@@ -108,7 +107,7 @@ To check out and run tests, you can simply run :
 ## Run tests
 
 Some tests will not run with just `cargo test` because of `fail-rs`.
-To run the tests exhaustively, run `./run-tests.sh`
+To run the tests exhaustively, run `./run-tests.sh`.
 
 ## Debug
 
@@ -116,13 +115,13 @@ You might find it useful to step through the programme with a debugger.
 
 ### A failing test
 
-Make sure you haven't run `cargo clean` after the most recent `cargo test` or `cargo build` to guarantee that `target/` dir exists. Use this bash script to find the most name of the most recent debug build of tantivy and run it under rust-gdb.
+Make sure you haven't run `cargo clean` after the most recent `cargo test` or `cargo build` to guarantee that the `target/` directory exists. Use this bash script to find the name of the most recent debug build of Tantivy and run it under `rust-gdb`:
 
 ```bash
 find target/debug/ -maxdepth 1 -executable -type f -name "tantivy*" -printf '%TY-%Tm-%Td %TT %p\n' | sort -r | cut -d " " -f 3 | xargs -I RECENT_DBG_TANTIVY rust-gdb RECENT_DBG_TANTIVY
 ```
 
-Now that you are in rust-gdb, you can set breakpoints on lines and methods that match your source-code and run the debug executable with flags that you normally pass to `cargo test` to like this
+Now that you are in `rust-gdb`, you can set breakpoints on lines and methods that match your source code and run the debug executable with flags that you normally pass to `cargo test` like this:
 
 ```bash
 $gdb run --test-threads 1 --test $NAME_OF_TEST
@@ -130,7 +129,7 @@ $gdb run --test-threads 1 --test $NAME_OF_TEST
 
 ### An example
 
-By default, rustc compiles everything in the `examples/` dir in debug mode. This makes it easy for you to make examples to reproduce bugs.
+By default, `rustc` compiles everything in the `examples/` directory in debug mode. This makes it easy for you to make examples to reproduce bugs:
 
 ```bash
 rust-gdb target/debug/examples/$EXAMPLE_NAME

incremental-search/Cargo.toml

@@ -1,10 +0,0 @@
-[package]
-name = "incremental-search"
-version = "0.11.0"
-authors = ["Paul Masurel <paul.masurel@gmail.com>"]
-edition = "2018"
-
-
-[dependencies]
-derive_builder = "0.7"
-tantivy = {path = ".."}

incremental-search/src/bitset.rs

@@ -1,395 +0,0 @@
-use std::fmt;
-use std::u64;
-
-#[derive(Clone, Copy, Eq, PartialEq)]
-pub(crate) struct TinySet(u64);
-
-impl fmt::Debug for TinySet {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        self.into_iter().collect::<Vec<u32>>().fmt(f)
-    }
-}
-
-pub struct TinySetIterator(TinySet);
-impl Iterator for TinySetIterator {
-    type Item = u32;
-
-    fn next(&mut self) -> Option<Self::Item> {
-        self.0.pop_lowest()
-    }
-}
-
-impl IntoIterator for TinySet {
-    type Item = u32;
-    type IntoIter = TinySetIterator;
-    fn into_iter(self) -> Self::IntoIter {
-        TinySetIterator(self)
-    }
-}
-
-impl TinySet {
-    /// Returns an empty `TinySet`.
-    pub fn empty() -> TinySet {
-        TinySet(0u64)
-    }
-
-    /// Returns the complement of the set in `[0, 64[`.
-    fn complement(self) -> TinySet {
-        TinySet(!self.0)
-    }
-
-    /// Returns true iff the `TinySet` contains the element `el`.
-    pub fn contains(self, el: u32) -> bool {
-        !self.intersect(TinySet::singleton(el)).is_empty()
-    }
-
-    /// Returns the intersection of `self` and `other`
-    pub fn intersect(self, other: TinySet) -> TinySet {
-        TinySet(self.0 & other.0)
-    }
-
-    /// Creates a new `TinySet` containing only one element
-    /// within `[0; 64[`
-    #[inline(always)]
-    pub fn singleton(el: u32) -> TinySet {
-        TinySet(1u64 << u64::from(el))
-    }
-
-    /// Insert a new element within [0..64[
-    #[inline(always)]
-    pub fn insert(self, el: u32) -> TinySet {
-        self.union(TinySet::singleton(el))
-    }
-
-    /// Insert a new element within [0..64[
-    #[inline(always)]
-    pub fn insert_mut(&mut self, el: u32) -> bool {
-        let old = *self;
-        *self = old.insert(el);
-        old != *self
-    }
-
-    /// Returns the union of two tinysets
-    #[inline(always)]
-    pub fn union(self, other: TinySet) -> TinySet {
-        TinySet(self.0 | other.0)
-    }
-
-    /// Returns true iff the `TinySet` is empty.
-    #[inline(always)]
-    pub fn is_empty(self) -> bool {
-        self.0 == 0u64
-    }
-
-    /// Returns the lowest element in the `TinySet`
-    /// and removes it.
-    #[inline(always)]
-    pub fn pop_lowest(&mut self) -> Option<u32> {
-        if self.is_empty() {
-            None
-        } else {
-            let lowest = self.0.trailing_zeros() as u32;
-            self.0 ^= TinySet::singleton(lowest).0;
-            Some(lowest)
-        }
-    }
-
-    /// Returns a `TinySet` than contains all values up
-    /// to limit excluded.
-    ///
-    /// The limit is assumed to be strictly lower than 64.
-    pub fn range_lower(upper_bound: u32) -> TinySet {
-        TinySet((1u64 << u64::from(upper_bound % 64u32)) - 1u64)
-    }
-
-    /// Returns a `TinySet` that contains all values greater
-    /// or equal to the given limit, included. (and up to 63)
-    ///
-    /// The limit is assumed to be strictly lower than 64.
-    pub fn range_greater_or_equal(from_included: u32) -> TinySet {
-        TinySet::range_lower(from_included).complement()
-    }
-
-    pub fn clear(&mut self) {
-        self.0 = 0u64;
-    }
-
-    pub fn len(self) -> u32 {
-        self.0.count_ones()
-    }
-}
-
-#[derive(Clone)]
-pub struct BitSet {
-    tinysets: Box<[TinySet]>,
-    len: usize, //< Technically it should be u32, but we
-    // count multiple inserts.
-    // `usize` guards us from overflow.
-    max_value: u32,
-}
-
-fn num_buckets(max_val: u32) -> u32 {
-    (max_val + 63u32) / 64u32
-}
-
-impl BitSet {
-    /// Create a new `BitSet` that may contain elements
-    /// within `[0, max_val[`.
-    pub fn with_max_value(max_value: u32) -> BitSet {
-        let num_buckets = num_buckets(max_value);
-        let tinybisets = vec![TinySet::empty(); num_buckets as usize].into_boxed_slice();
-        BitSet {
-            tinysets: tinybisets,
-            len: 0,
-            max_value,
-        }
-    }
-
-    /// Removes all elements from the `BitSet`.
-    pub fn clear(&mut self) {
-        for tinyset in self.tinysets.iter_mut() {
-            *tinyset = TinySet::empty();
-        }
-    }
-
-    /// Returns the number of elements in the `BitSet`.
-    pub fn len(&self) -> usize {
-        self.len
-    }
-
-    /// Inserts an element in the `BitSet`
-    pub fn insert(&mut self, el: u32) {
-        // we do not check saturated els.
-        let higher = el / 64u32;
-        let lower = el % 64u32;
-        self.len += if self.tinysets[higher as usize].insert_mut(lower) {
-            1
-        } else {
-            0
-        };
-    }
-
-    /// Returns true iff the elements is in the `BitSet`.
-    pub fn contains(&self, el: u32) -> bool {
-        self.tinyset(el / 64u32).contains(el % 64)
-    }
-
-    /// Returns the first non-empty `TinySet` associated to a bucket lower
-    /// or greater than bucket.
-    ///
-    /// Reminder: the tiny set with the bucket `bucket`, represents the
-    /// elements from `bucket * 64` to `(bucket+1) * 64`.
-    pub(crate) fn first_non_empty_bucket(&self, bucket: u32) -> Option<u32> {
-        self.tinysets[bucket as usize..]
-            .iter()
-            .cloned()
-            .position(|tinyset| !tinyset.is_empty())
-            .map(|delta_bucket| bucket + delta_bucket as u32)
-    }
-
-    pub fn max_value(&self) -> u32 {
-        self.max_value
-    }
-
-    /// Returns the tiny bitset representing the
-    /// the set restricted to the number range from
-    /// `bucket * 64` to `(bucket + 1) * 64`.
-    pub(crate) fn tinyset(&self, bucket: u32) -> TinySet {
-        self.tinysets[bucket as usize]
-    }
-}
-
-#[cfg(test)]
-mod tests {
-
-    use super::BitSet;
-    use super::TinySet;
-    use crate::docset::DocSet;
-    use crate::query::BitSetDocSet;
-    use crate::tests;
-    use crate::tests::generate_nonunique_unsorted;
-    use std::collections::BTreeSet;
-    use std::collections::HashSet;
-
-    #[test]
-    fn test_tiny_set() {
-        assert!(TinySet::empty().is_empty());
-        {
-            let mut u = TinySet::empty().insert(1u32);
-            assert_eq!(u.pop_lowest(), Some(1u32));
-            assert!(u.pop_lowest().is_none())
-        }
-        {
-            let mut u = TinySet::empty().insert(1u32).insert(1u32);
-            assert_eq!(u.pop_lowest(), Some(1u32));
-            assert!(u.pop_lowest().is_none())
-        }
-        {
-            let mut u = TinySet::empty().insert(2u32);
-            assert_eq!(u.pop_lowest(), Some(2u32));
-            u.insert_mut(1u32);
-            assert_eq!(u.pop_lowest(), Some(1u32));
-            assert!(u.pop_lowest().is_none());
-        }
-        {
-            let mut u = TinySet::empty().insert(63u32);
-            assert_eq!(u.pop_lowest(), Some(63u32));
-            assert!(u.pop_lowest().is_none());
-        }
-    }
-
-    #[test]
-    fn test_bitset() {
-        let test_against_hashset = |els: &[u32], max_value: u32| {
-            let mut hashset: HashSet<u32> = HashSet::new();
-            let mut bitset = BitSet::with_max_value(max_value);
-            for &el in els {
-                assert!(el < max_value);
-                hashset.insert(el);
-                bitset.insert(el);
-            }
-            for el in 0..max_value {
-                assert_eq!(hashset.contains(&el), bitset.contains(el));
-            }
-            assert_eq!(bitset.max_value(), max_value);
-        };
-
-        test_against_hashset(&[], 0);
-        test_against_hashset(&[], 1);
-        test_against_hashset(&[0u32], 1);
-        test_against_hashset(&[0u32], 100);
-        test_against_hashset(&[1u32, 2u32], 4);
-        test_against_hashset(&[99u32], 100);
-        test_against_hashset(&[63u32], 64);
-        test_against_hashset(&[62u32, 63u32], 64);
-    }
-
-    #[test]
-    fn test_bitset_large() {
-        let arr = generate_nonunique_unsorted(100_000, 5_000);
-        let mut btreeset: BTreeSet<u32> = BTreeSet::new();
-        let mut bitset = BitSet::with_max_value(100_000);
-        for el in arr {
-            btreeset.insert(el);
-            bitset.insert(el);
-        }
-        for i in 0..100_000 {
-            assert_eq!(btreeset.contains(&i), bitset.contains(i));
-        }
-        assert_eq!(btreeset.len(), bitset.len());
-        let mut bitset_docset = BitSetDocSet::from(bitset);
-        for el in btreeset.into_iter() {
-            bitset_docset.advance();
-            assert_eq!(bitset_docset.doc(), el);
-        }
-        assert!(!bitset_docset.advance());
-    }
-
-    #[test]
-    fn test_bitset_num_buckets() {
-        use super::num_buckets;
-        assert_eq!(num_buckets(0u32), 0);
-        assert_eq!(num_buckets(1u32), 1);
-        assert_eq!(num_buckets(64u32), 1);
-        assert_eq!(num_buckets(65u32), 2);
-        assert_eq!(num_buckets(128u32), 2);
-        assert_eq!(num_buckets(129u32), 3);
-    }
-
-    #[test]
-    fn test_tinyset_range() {
-        assert_eq!(
-            TinySet::range_lower(3).into_iter().collect::<Vec<u32>>(),
-            [0, 1, 2]
-        );
-        assert!(TinySet::range_lower(0).is_empty());
-        assert_eq!(
-            TinySet::range_lower(63).into_iter().collect::<Vec<u32>>(),
-            (0u32..63u32).collect::<Vec<_>>()
-        );
-        assert_eq!(
-            TinySet::range_lower(1).into_iter().collect::<Vec<u32>>(),
-            [0]
-        );
-        assert_eq!(
-            TinySet::range_lower(2).into_iter().collect::<Vec<u32>>(),
-            [0, 1]
-        );
-        assert_eq!(
-            TinySet::range_greater_or_equal(3)
-                .into_iter()
-                .collect::<Vec<u32>>(),
-            (3u32..64u32).collect::<Vec<_>>()
-        );
-    }
-
-    #[test]
-    fn test_bitset_len() {
-        let mut bitset = BitSet::with_max_value(1_000);
-        assert_eq!(bitset.len(), 0);
-        bitset.insert(3u32);
-        assert_eq!(bitset.len(), 1);
-        bitset.insert(103u32);
-        assert_eq!(bitset.len(), 2);
-        bitset.insert(3u32);
-        assert_eq!(bitset.len(), 2);
-        bitset.insert(103u32);
-        assert_eq!(bitset.len(), 2);
-        bitset.insert(104u32);
-        assert_eq!(bitset.len(), 3);
-    }
-
-    #[test]
-    fn test_bitset_clear() {
-        let mut bitset = BitSet::with_max_value(1_000);
-        let els = tests::sample(1_000, 0.01f64);
-        for &el in &els {
-            bitset.insert(el);
-        }
-        assert!(els.iter().all(|el| bitset.contains(*el)));
-        bitset.clear();
-        for el in 0u32..1000u32 {
-            assert!(!bitset.contains(el));
-        }
-    }
-}
-
-#[cfg(all(test, feature = "unstable"))]
-mod bench {
-
-    use super::BitSet;
-    use super::TinySet;
-    use test;
-
-    #[bench]
-    fn bench_tinyset_pop(b: &mut test::Bencher) {
-        b.iter(|| {
-            let mut tinyset = TinySet::singleton(test::black_box(31u32));
-            tinyset.pop_lowest();
-            tinyset.pop_lowest();
-            tinyset.pop_lowest();
-            tinyset.pop_lowest();
-            tinyset.pop_lowest();
-            tinyset.pop_lowest();
-        });
-    }
-
-    #[bench]
-    fn bench_tinyset_sum(b: &mut test::Bencher) {
-        let tiny_set = TinySet::empty().insert(10u32).insert(14u32).insert(21u32);
-        b.iter(|| {
-            assert_eq!(test::black_box(tiny_set).into_iter().sum::<u32>(), 45u32);
-        });
-    }
-
-    #[bench]
-    fn bench_tinyarr_sum(b: &mut test::Bencher) {
-        let v = [10u32, 14u32, 21u32];
-        b.iter(|| test::black_box(v).iter().cloned().sum::<u32>());
-    }
-
-    #[bench]
-    fn bench_bitset_initialize(b: &mut test::Bencher) {
-        b.iter(|| BitSet::with_max_value(1_000_000));
-    }
-}

incremental-search/src/lib.rs

@@ -1,266 +0,0 @@
-use tantivy::query::{BooleanQuery, FuzzyTermQuery, EmptyQuery};
-use derive_builder::Builder;
-use std::str::FromStr;
-use tantivy::query::{FuzzyConfiguration, FuzzyConfigurationBuilder, Query, Occur};
-use tantivy::schema::Field;
-use tantivy::{Searcher, TantivyError, DocAddress, Term, Document};
-use tantivy::collector::TopDocs;
-use std::ops::Deref;
-
-
-#[derive(Debug)]
-pub struct IncrementalSearchQuery {
-    pub terms: Vec<String>,
-    pub last_is_prefix: bool,
-}
-
-impl IncrementalSearchQuery {
-    pub fn fuzzy_configurations(&self) -> Vec<FuzzyConfigurations> {
-        if self.terms.is_empty() {
-            return Vec::default();
-        }
-        let single_term_confs: Vec<FuzzyConfigurationBuilder> = (0u8..3u8)
-            .map(|d: u8| {
-                let mut builder = FuzzyConfigurationBuilder::default();
-                builder.distance(d).transposition_cost_one(true);
-                builder
-            })
-            .collect();
-        let mut configurations: Vec<Vec<FuzzyConfigurationBuilder>> = single_term_confs
-            .iter()
-            .map(|conf| vec![conf.clone()])
-            .collect();
-        let mut new_configurations = Vec::new();
-        for _ in 1..self.terms.len() {
-            new_configurations.clear();
-            for single_term_conf in &single_term_confs {
-                for configuration in &configurations {
-                    let mut new_configuration: Vec<FuzzyConfigurationBuilder> = configuration.clone();
-                    new_configuration.push(single_term_conf.clone());
-                    new_configurations.push(new_configuration);
-                }
-            }
-            std::mem::swap(&mut configurations, &mut new_configurations);
-        }
-        if self.last_is_prefix {
-            for configuration in &mut configurations {
-                if let Some(last_conf) = configuration.last_mut() {
-                    last_conf.prefix(true);
-                }
-            }
-        }
-        let mut fuzzy_configurations: Vec<FuzzyConfigurations> = configurations
-            .into_iter()
-            .map(FuzzyConfigurations::from)
-            .collect();
-        fuzzy_configurations.sort_by(|left, right| left.cost.partial_cmp(&right.cost).unwrap());
-        fuzzy_configurations
-    }
-
-    fn search_query(&self, fields: &[Field], configurations: FuzzyConfigurations) -> Box<dyn Query> {
-        if self.terms.is_empty() {
-            Box::new(EmptyQuery)
-        } else if self.terms.len() == 1 {
-            build_query_for_fields(fields, &self.terms[0], &configurations.configurations[0])
-        } else {
-            Box::new(BooleanQuery::from(self.terms.iter()
-                .zip(configurations.configurations.iter())
-                .map(|(term, configuration)|
-                         (Occur::Must, build_query_for_fields(fields, &term, &configuration))
-                )
-                .collect::<Vec<_>>()))
-        }
-    }
-}
-
-#[derive(Debug)]
-pub struct FuzzyConfigurations {
-    configurations: Vec<FuzzyConfiguration>,
-    cost: f64,
-}
-
-
-fn compute_cost(fuzzy_confs: &[FuzzyConfiguration]) -> f64 {
-    fuzzy_confs
-        .iter()
-        .map(|fuzzy_conf| {
-            let weight = if fuzzy_conf.prefix { 30f64 } else { 5f64 };
-            weight * f64::from(fuzzy_conf.distance)
-        })
-        .sum()
-}
-
-impl From<Vec<FuzzyConfigurationBuilder>> for FuzzyConfigurations {
-    fn from(fuzzy_conf_builder: Vec<FuzzyConfigurationBuilder>) -> FuzzyConfigurations {
-        let configurations = fuzzy_conf_builder
-            .into_iter()
-            .map(|conf| conf.build().unwrap())
-            .collect::<Vec<FuzzyConfiguration>>();
-        let cost = compute_cost(&configurations);
-        FuzzyConfigurations {
-            configurations,
-            cost,
-        }
-    }
-}
-
-#[derive(Debug)]
-pub struct ParseIncrementalQueryError;
-
-impl Into<TantivyError> for ParseIncrementalQueryError {
-    fn into(self) -> TantivyError {
-        TantivyError::InvalidArgument(format!("Invalid query: {:?}", self))
-    }
-}
-
-impl FromStr for IncrementalSearchQuery {
-    type Err = ParseIncrementalQueryError;
-
-    fn from_str(query_str: &str) -> Result<Self, Self::Err> {
-        let terms: Vec<String> = query_str
-            .split_whitespace()
-            .map(ToString::to_string)
-            .collect();
-        Ok(IncrementalSearchQuery {
-            terms,
-            last_is_prefix: query_str
-                .chars()
-                .last()
-                .map(|c| !c.is_whitespace())
-                .unwrap_or(false),
-        })
-    }
-}
-
-fn build_query_for_fields(fields: &[Field], term_text: &str, conf: &FuzzyConfiguration) -> Box<dyn Query> {
-    assert!(fields.len() > 0);
-    if fields.len() > 1 {
-        let term_queries: Vec<(Occur, Box<dyn Query>)> = fields
-            .iter()
-            .map(|&field| {
-                let term = Term::from_field_text(field, term_text);
-                let query = FuzzyTermQuery::new_from_configuration(term, conf.clone());
-                let boxed_query: Box<dyn Query> = Box::new(query);
-                (Occur::Must, boxed_query)
-            })
-            .collect();
-        Box::new(BooleanQuery::from(term_queries))
-    } else {
-        let term = Term::from_field_text(fields[0], term_text);
-        Box::new( FuzzyTermQuery::new_from_configuration(term, conf.clone()))
-    }
-
-}
-
-pub struct IncrementalSearchResult {
-    pub docs: Vec<Document>
-}
-
-#[derive(Builder, Default)]
-pub struct IncrementalSearch {
-    nhits: usize,
-    #[builder(default)]
-    search_fields: Vec<Field>,
-    #[builder(default)]
-    return_fields: Vec<Field>,
-}
-
-impl IncrementalSearch {
-
-    pub fn search<S: Deref<Target=Searcher>>(
-        &self,
-        query: &str,
-        searcher: &S,
-    ) -> tantivy::Result<IncrementalSearchResult> {
-        let searcher = searcher.deref();
-        let inc_search_query: IncrementalSearchQuery =
-            FromStr::from_str(query).map_err(Into::<TantivyError>::into)?;
-
-        let mut results: Vec<DocAddress> = Vec::default();
-        let mut remaining = self.nhits;
-        for fuzzy_conf in inc_search_query.fuzzy_configurations() {
-            if remaining == 0 {
-                break;
-            }
-            let query = inc_search_query.search_query(&self.search_fields[..], fuzzy_conf);
-            let new_docs = searcher.search(query.as_ref(), &TopDocs::with_limit(remaining))?;
-            // TODO(pmasurel) remove already added docs.
-            results.extend(new_docs.into_iter()
-                .map(|(_, doc_address)| doc_address));
-            remaining = self.nhits - results.len();
-            if remaining == 0 {
-                break;
-            }
-        }
-        let docs: Vec<Document> = results.into_iter()
-            .map(|doc_address: DocAddress| searcher.doc(doc_address))
-            .collect::<tantivy::Result<_>>()?;
-        Ok(IncrementalSearchResult {
-            docs
-        })
-    }
-}
-#[cfg(test)]
-mod tests {
-    use tantivy::doc;
-    use crate::{IncrementalSearch, IncrementalSearchBuilder, IncrementalSearchQuery};
-    use std::str::FromStr;
-    use tantivy::schema::{SchemaBuilder, TEXT, STORED};
-    use tantivy::Index;
-
-    #[test]
-    fn test_incremental_search() {
-        let incremental_search = IncrementalSearchBuilder::default()
-            .nhits(10)
-            .build()
-            .unwrap();
-    }
-
-    #[test]
-    fn test_incremental_search_query_parse_empty() {
-        let query = IncrementalSearchQuery::from_str("").unwrap();
-        assert_eq!(query.terms, Vec::<String>::new());
-        assert!(!query.last_is_prefix);
-    }
-
-    #[test]
-    fn test_incremental_search_query_parse_trailing_whitespace() {
-        let query = IncrementalSearchQuery::from_str("hello happy tax pa ").unwrap();
-        assert_eq!(query.terms, vec!["hello", "happy", "tax", "pa"]);
-        assert!(!query.last_is_prefix);
-    }
-
-    #[test]
-    fn test_incremental_search_query_parse_unicode_whitespace() {
-        let query = IncrementalSearchQuery::from_str("hello  happy tax pa ").unwrap();
-        assert_eq!(query.terms, vec!["hello", "happy", "tax", "pa"]);
-        assert!(!query.last_is_prefix);
-    }
-
-    #[test]
-    fn test_incremental_search_query_parse() {
-        let query = IncrementalSearchQuery::from_str("hello happy tax pa").unwrap();
-        assert_eq!(query.terms, vec!["hello", "happy", "tax", "pa"]);
-        assert!(query.last_is_prefix);
-    }
-
-    #[test]
-    fn test_blop() {
-        let mut schema_builder = SchemaBuilder::new();
-        let body = schema_builder.add_text_field("body", TEXT | STORED);
-        let schema = schema_builder.build();
-        let index = Index::create_in_ram(schema);
-        let mut index_writer = index.writer_with_num_threads(1, 30_000_000).unwrap();
-        index_writer.add_document(doc!(body=> "hello happy tax payer"));
-        index_writer.commit().unwrap();
-        let reader = index.reader().unwrap();
-        let searcher = reader.searcher();
-        let incremental_search: IncrementalSearch = IncrementalSearchBuilder::default()
-            .nhits(1)
-            .search_fields(vec![body])
-            .build()
-            .unwrap();
-        let top_docs = incremental_search.search("hello hapy t", &searcher).unwrap();
-        assert_eq!(top_docs.docs.len(), 1);
-    }
-}

query-grammar/src/occur.rs

@@ -1,5 +1,8 @@
+use std::fmt;
+use std::fmt::Write;
+
 /// Defines whether a term in a query must be present,
-/// should be present or must not be present.
+/// should be present or must be not present.
 #[derive(Debug, Clone, Hash, Copy, Eq, PartialEq)]
 pub enum Occur {
     /// For a given document to be considered for scoring,
@@ -18,7 +21,7 @@ impl Occur {
     /// - `Should` => '?',
     /// - `Must` => '+'
     /// - `Not` => '-'
-    pub fn to_char(self) -> char {
+    fn to_char(self) -> char {
         match self {
             Occur::Should => '?',
             Occur::Must => '+',
@@ -47,3 +50,9 @@ impl Occur {
         }
     }
 }
+
+impl fmt::Display for Occur {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        f.write_char(self.to_char())
+    }
+}

query-grammar/src/user_input_ast.rs

@@ -151,7 +151,7 @@ impl fmt::Debug for UserInputAST {
                 Ok(())
             }
             UserInputAST::Unary(ref occur, ref subquery) => {
-                write!(formatter, "{}({:?})", occur.to_char(), subquery)
+                write!(formatter, "{}({:?})", occur, subquery)
             }
             UserInputAST::Leaf(ref subquery) => write!(formatter, "{:?}", subquery),
         }

src/collector/count_collector.rs

@@ -123,5 +123,4 @@ mod tests {
             assert_eq!(count_collector.harvest(), 2);
         }
     }
-
 }

src/collector/facet_collector.rs

@@ -515,7 +515,7 @@ mod tests {
     #[should_panic(expected = "Tried to add a facet which is a descendant of \
                                an already added facet.")]
     fn test_misused_facet_collector() {
-        let mut facet_collector = FacetCollector::for_field(Field(0));
+        let mut facet_collector = FacetCollector::for_field(Field::from_field_id(0));
         facet_collector.add_facet(Facet::from("/country"));
         facet_collector.add_facet(Facet::from("/country/europe"));
     }
@@ -546,7 +546,7 @@ mod tests {
 
     #[test]
     fn test_non_used_facet_collector() {
-        let mut facet_collector = FacetCollector::for_field(Field(0));
+        let mut facet_collector = FacetCollector::for_field(Field::from_field_id(0));
         facet_collector.add_facet(Facet::from("/country"));
         facet_collector.add_facet(Facet::from("/countryeurope"));
     }
@@ -599,19 +599,18 @@ mod tests {
             );
         }
     }
-
 }
 
 #[cfg(all(test, feature = "unstable"))]
 mod bench {
 
-    use collector::FacetCollector;
-    use query::AllQuery;
-    use rand::{thread_rng, Rng};
-    use schema::Facet;
-    use schema::Schema;
+    use crate::collector::FacetCollector;
+    use crate::query::AllQuery;
+    use crate::schema::{Facet, Schema};
+    use crate::Index;
+    use rand::seq::SliceRandom;
+    use rand::thread_rng;
     use test::Bencher;
-    use Index;
 
     #[bench]
     fn bench_facet_collector(b: &mut Bencher) {
@@ -628,7 +627,7 @@ mod bench {
             }
         }
         // 40425 docs
-        thread_rng().shuffle(&mut docs[..]);
+        docs[..].shuffle(&mut thread_rng());
 
         let mut index_writer = index.writer_with_num_threads(1, 3_000_000).unwrap();
         for doc in docs {
@@ -637,7 +636,7 @@ mod bench {
         index_writer.commit().unwrap();
         let reader = index.reader().unwrap();
         b.iter(|| {
-            let searcher = index.searcher();
+            let searcher = reader.searcher();
             let facet_collector = FacetCollector::for_field(facet_field);
             searcher.search(&AllQuery, &facet_collector).unwrap();
         });

src/collector/top_collector.rs

@@ -12,6 +12,9 @@ use std::collections::BinaryHeap;
 /// It has a custom implementation of `PartialOrd` that reverses the order. This is because the
 /// default Rust heap is a max heap, whereas a min heap is needed.
 ///
+/// Additionally, it guarantees stable sorting: in case of a tie on the feature, the document
+/// address is used.
+///
 /// WARNING: equality is not what you would expect here.
 /// Two elements are equal if their feature is equal, and regardless of whether `doc`
 /// is equal. This should be perfectly fine for this usage, but let's make sure this
@@ -21,29 +24,37 @@ struct ComparableDoc<T, D> {
     doc: D,
 }
 
-impl<T: PartialOrd, D> PartialOrd for ComparableDoc<T, D> {
+impl<T: PartialOrd, D: PartialOrd> PartialOrd for ComparableDoc<T, D> {
     fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
         Some(self.cmp(other))
     }
 }
 
-impl<T: PartialOrd, D> Ord for ComparableDoc<T, D> {
+impl<T: PartialOrd, D: PartialOrd> Ord for ComparableDoc<T, D> {
     #[inline]
     fn cmp(&self, other: &Self) -> Ordering {
-        other
+        // Reversed to make BinaryHeap work as a min-heap
+        let by_feature = other
             .feature
             .partial_cmp(&self.feature)
-            .unwrap_or_else(|| Ordering::Equal)
+            .unwrap_or(Ordering::Equal);
+
+        let lazy_by_doc_address = || self.doc.partial_cmp(&other.doc).unwrap_or(Ordering::Equal);
+
+        // In case of a tie on the feature, we sort by ascending
+        // `DocAddress` in order to ensure a stable sorting of the
+        // documents.
+        by_feature.then_with(lazy_by_doc_address)
     }
 }
 
-impl<T: PartialOrd, D> PartialEq for ComparableDoc<T, D> {
+impl<T: PartialOrd, D: PartialOrd> PartialEq for ComparableDoc<T, D> {
     fn eq(&self, other: &Self) -> bool {
         self.cmp(other) == Ordering::Equal
     }
 }
 
-impl<T: PartialOrd, D> Eq for ComparableDoc<T, D> {}
+impl<T: PartialOrd, D: PartialOrd> Eq for ComparableDoc<T, D> {}
 
 pub(crate) struct TopCollector<T> {
     limit: usize,
@@ -214,4 +225,94 @@ mod tests {
             ]
         );
     }
+
+    #[test]
+    fn test_top_segment_collector_stable_ordering_for_equal_feature() {
+        // given that the documents are collected in ascending doc id order,
+        // when harvesting we have to guarantee stable sorting in case of a tie
+        // on the score
+        let doc_ids_collection = [4, 5, 6];
+        let score = 3.14;
+
+        let mut top_collector_limit_2 = TopSegmentCollector::new(0, 2);
+        for id in &doc_ids_collection {
+            top_collector_limit_2.collect(*id, score);
+        }
+
+        let mut top_collector_limit_3 = TopSegmentCollector::new(0, 3);
+        for id in &doc_ids_collection {
+            top_collector_limit_3.collect(*id, score);
+        }
+
+        assert_eq!(
+            top_collector_limit_2.harvest(),
+            top_collector_limit_3.harvest()[..2].to_vec(),
+        );
+    }
+}
+
+#[cfg(all(test, feature = "unstable"))]
+mod bench {
+    use super::TopSegmentCollector;
+    use test::Bencher;
+
+    #[bench]
+    fn bench_top_segment_collector_collect_not_at_capacity(b: &mut Bencher) {
+        let mut top_collector = TopSegmentCollector::new(0, 400);
+
+        b.iter(|| {
+            for i in 0..100 {
+                top_collector.collect(i, 0.8);
+            }
+        });
+    }
+
+    #[bench]
+    fn bench_top_segment_collector_collect_at_capacity(b: &mut Bencher) {
+        let mut top_collector = TopSegmentCollector::new(0, 100);
+
+        for i in 0..100 {
+            top_collector.collect(i, 0.8);
+        }
+
+        b.iter(|| {
+            for i in 0..100 {
+                top_collector.collect(i, 0.8);
+            }
+        });
+    }
+
+    #[bench]
+    fn bench_top_segment_collector_collect_and_harvest_many_ties(b: &mut Bencher) {
+        b.iter(|| {
+            let mut top_collector = TopSegmentCollector::new(0, 100);
+
+            for i in 0..100 {
+                top_collector.collect(i, 0.8);
+            }
+
+            // it would be nice to be able to do the setup N times but still
+            // measure only harvest(). We can't since harvest() consumes
+            // the top_collector.
+            top_collector.harvest()
+        });
+    }
+
+    #[bench]
+    fn bench_top_segment_collector_collect_and_harvest_no_tie(b: &mut Bencher) {
+        b.iter(|| {
+            let mut top_collector = TopSegmentCollector::new(0, 100);
+            let mut score = 1.0;
+
+            for i in 0..100 {
+                score += 1.0;
+                top_collector.collect(i, score);
+            }
+
+            // it would be nice to be able to do the setup N times but still
+            // measure only harvest(). We can't since harvest() consumes
+            // the top_collector.
+            top_collector.harvest()
+        });
+    }
 }

src/collector/top_score_collector.rs

@@ -15,13 +15,16 @@ use crate::SegmentLocalId;
 use crate::SegmentReader;
 use std::fmt;
 
-/// The Top Score Collector keeps track of the K documents
+/// The `TopDocs` collector keeps track of the top `K` documents
 /// sorted by their score.
 ///
 /// The implementation is based on a `BinaryHeap`.
 /// The theorical complexity for collecting the top `K` out of `n` documents
 /// is `O(n log K)`.
 ///
+/// This collector guarantees a stable sorting in case of a tie on the
+/// document score. As such, it is suitable to implement pagination.
+///
 /// ```rust
 /// use tantivy::collector::TopDocs;
 /// use tantivy::query::QueryParser;
@@ -428,12 +431,13 @@ impl SegmentCollector for TopScoreSegmentCollector {
 mod tests {
     use super::TopDocs;
     use crate::collector::Collector;
-    use crate::query::{Query, QueryParser};
+    use crate::query::{AllQuery, Query, QueryParser};
     use crate::schema::{Field, Schema, FAST, STORED, TEXT};
     use crate::DocAddress;
     use crate::Index;
     use crate::IndexWriter;
     use crate::Score;
+    use itertools::Itertools;
 
     fn make_index() -> Index {
         let mut schema_builder = Schema::builder();
@@ -494,6 +498,29 @@ mod tests {
         );
     }
 
+    #[test]
+    fn test_top_collector_stable_sorting() {
+        let index = make_index();
+
+        // using AllQuery to get a constant score
+        let searcher = index.reader().unwrap().searcher();
+
+        let page_1 = searcher.search(&AllQuery, &TopDocs::with_limit(2)).unwrap();
+
+        let page_2 = searcher.search(&AllQuery, &TopDocs::with_limit(3)).unwrap();
+
+        // precondition for the test to be meaningful: we did get documents
+        // with the same score
+        assert!(page_1.iter().map(|result| result.0).all_equal());
+        assert!(page_2.iter().map(|result| result.0).all_equal());
+
+        // sanity check since we're relying on make_index()
+        assert_eq!(page_1.len(), 2);
+        assert_eq!(page_2.len(), 3);
+
+        assert_eq!(page_1, &page_2[..page_1.len()]);
+    }
+
     #[test]
     #[should_panic]
     fn test_top_0() {
@@ -551,7 +578,7 @@ mod tests {
             ));
         });
         let searcher = index.reader().unwrap().searcher();
-        let top_collector = TopDocs::with_limit(4).order_by_u64_field(Field(2));
+        let top_collector = TopDocs::with_limit(4).order_by_u64_field(Field::from_field_id(2));
         let segment_reader = searcher.segment_reader(0u32);
         top_collector
             .for_segment(0, segment_reader)
@@ -592,5 +619,4 @@ mod tests {
         let query = query_parser.parse_query(query).unwrap();
         (index, query)
     }
-
 }

src/common/composite_file.rs

@@ -2,7 +2,7 @@ use crate::common::BinarySerializable;
 use crate::common::CountingWriter;
 use crate::common::VInt;
 use crate::directory::ReadOnlySource;
-use crate::directory::WritePtr;
+use crate::directory::{TerminatingWrite, WritePtr};
 use crate::schema::Field;
 use crate::space_usage::FieldUsage;
 use crate::space_usage::PerFieldSpaceUsage;
@@ -42,7 +42,7 @@ pub struct CompositeWrite<W = WritePtr> {
     offsets: HashMap<FileAddr, u64>,
 }
 
-impl<W: Write> CompositeWrite<W> {
+impl<W: TerminatingWrite + Write> CompositeWrite<W> {
     /// Crate a new API writer that writes a composite file
     /// in a given write.
     pub fn wrap(w: W) -> CompositeWrite<W> {
@@ -91,8 +91,7 @@ impl<W: Write> CompositeWrite<W> {
 
         let footer_len = (self.write.written_bytes() - footer_offset) as u32;
         footer_len.serialize(&mut self.write)?;
-        self.write.flush()?;
-        Ok(())
+        self.write.terminate()
     }
 }
 
@@ -200,13 +199,13 @@ mod test {
             let w = directory.open_write(path).unwrap();
             let mut composite_write = CompositeWrite::wrap(w);
             {
-                let mut write_0 = composite_write.for_field(Field(0u32));
+                let mut write_0 = composite_write.for_field(Field::from_field_id(0u32));
                 VInt(32431123u64).serialize(&mut write_0).unwrap();
                 write_0.flush().unwrap();
             }
 
             {
-                let mut write_4 = composite_write.for_field(Field(4u32));
+                let mut write_4 = composite_write.for_field(Field::from_field_id(4u32));
                 VInt(2).serialize(&mut write_4).unwrap();
                 write_4.flush().unwrap();
             }
@@ -216,14 +215,18 @@ mod test {
             let r = directory.open_read(path).unwrap();
             let composite_file = CompositeFile::open(&r).unwrap();
             {
-                let file0 = composite_file.open_read(Field(0u32)).unwrap();
+                let file0 = composite_file
+                    .open_read(Field::from_field_id(0u32))
+                    .unwrap();
                 let mut file0_buf = file0.as_slice();
                 let payload_0 = VInt::deserialize(&mut file0_buf).unwrap().0;
                 assert_eq!(file0_buf.len(), 0);
                 assert_eq!(payload_0, 32431123u64);
             }
             {
-                let file4 = composite_file.open_read(Field(4u32)).unwrap();
+                let file4 = composite_file
+                    .open_read(Field::from_field_id(4u32))
+                    .unwrap();
                 let mut file4_buf = file4.as_slice();
                 let payload_4 = VInt::deserialize(&mut file4_buf).unwrap().0;
                 assert_eq!(file4_buf.len(), 0);
@@ -231,5 +234,4 @@ mod test {
             }
         }
     }
-
 }

src/common/counting_writer.rs

@@ -1,3 +1,5 @@
+use crate::directory::AntiCallToken;
+use crate::directory::TerminatingWrite;
 use std::io;
 use std::io::Write;
 
@@ -42,6 +44,13 @@ impl<W: Write> Write for CountingWriter<W> {
     }
 }
 
+impl<W: TerminatingWrite> TerminatingWrite for CountingWriter<W> {
+    fn terminate_ref(&mut self, token: AntiCallToken) -> io::Result<()> {
+        self.flush()?;
+        self.underlying.terminate_ref(token)
+    }
+}
+
 #[cfg(test)]
 mod test {
 

src/common/serialize.rs

@@ -199,10 +199,7 @@ pub mod test {
     fn test_serialize_string() {
         assert_eq!(serialize_test(String::from("")), 1);
         assert_eq!(serialize_test(String::from("ぽよぽよ")), 1 + 3 * 4);
-        assert_eq!(
-            serialize_test(String::from("富士さん見える。")),
-            1 + 3 * 8
-        );
+        assert_eq!(serialize_test(String::from("富士さん見える。")), 1 + 3 * 8);
     }
 
     #[test]

src/core/index.rs

@@ -26,9 +26,10 @@ use crate::IndexWriter;
 use crate::Result;
 use num_cpus;
 use std::borrow::BorrowMut;
+use std::collections::HashSet;
 use std::fmt;
 #[cfg(feature = "mmap")]
-use std::path::Path;
+use std::path::{Path, PathBuf};
 use std::sync::Arc;
 
 fn load_metas(directory: &dyn Directory, inventory: &SegmentMetaInventory) -> Result<IndexMeta> {
@@ -368,6 +369,11 @@ impl Index {
             .map(SegmentMeta::id)
             .collect())
     }
+
+    /// Returns the set of corrupted files
+    pub fn validate_checksum(&self) -> Result<HashSet<PathBuf>> {
+        self.directory.list_damaged().map_err(Into::into)
+    }
 }
 
 impl fmt::Debug for Index {
@@ -595,5 +601,4 @@ mod tests {
         assert_eq!(searcher.num_docs(), 8_000);
         assert!(mem_right_after_merge_finished < mem_right_after_commit);
     }
-
 }

src/core/segment_id.rs

@@ -76,7 +76,7 @@ impl SegmentId {
 }
 
 /// Error type used when parsing a `SegmentId` from a string fails.
-pub struct SegmentIdParseError(uuid::parser::ParseError);
+pub struct SegmentIdParseError(uuid::Error);
 
 impl Error for SegmentIdParseError {}
 

src/directory/directory.rs

@@ -118,6 +118,8 @@ pub trait Directory: DirectoryClone + fmt::Debug + Send + Sync + 'static {
     ///
     /// Specifically, subsequent writes or flushes should
     /// have no effect on the returned `ReadOnlySource` object.
+    ///
+    /// You should only use this to read files create with [`open_write`]
     fn open_read(&self, path: &Path) -> result::Result<ReadOnlySource, OpenReadError>;
 
     /// Removes a file
@@ -157,6 +159,8 @@ pub trait Directory: DirectoryClone + fmt::Debug + Send + Sync + 'static {
     /// atomic_write.
     ///
     /// This should only be used for small files.
+    ///
+    /// You should only use this to read files create with [`atomic_write`]
     fn atomic_read(&self, path: &Path) -> Result<Vec<u8>, OpenReadError>;
 
     /// Atomically replace the content of a file with data.

src/directory/footer.rs

@@ -0,0 +1,213 @@
+use crate::directory::read_only_source::ReadOnlySource;
+use crate::directory::{AntiCallToken, TerminatingWrite};
+use byteorder::{ByteOrder, LittleEndian};
+use crc32fast::Hasher;
+use std::io;
+use std::io::Write;
+
+const COMMON_FOOTER_SIZE: usize = 4 * 5;
+
+#[derive(Debug, Clone, PartialEq)]
+pub struct Footer {
+    pub tantivy_version: (u32, u32, u32),
+    pub meta: String,
+    pub versioned_footer: VersionedFooter,
+}
+
+impl Footer {
+    pub fn new(versioned_footer: VersionedFooter) -> Self {
+        let tantivy_version = (
+            env!("CARGO_PKG_VERSION_MAJOR").parse().unwrap(),
+            env!("CARGO_PKG_VERSION_MINOR").parse().unwrap(),
+            env!("CARGO_PKG_VERSION_PATCH").parse().unwrap(),
+        );
+        Footer {
+            tantivy_version,
+            meta: format!(
+                "tantivy {}.{}.{}, index v{}",
+                tantivy_version.0,
+                tantivy_version.1,
+                tantivy_version.2,
+                versioned_footer.version()
+            ),
+            versioned_footer,
+        }
+    }
+
+    pub fn to_bytes(&self) -> Vec<u8> {
+        let mut res = self.versioned_footer.to_bytes();
+        res.extend_from_slice(self.meta.as_bytes());
+        let len = res.len();
+        res.resize(len + COMMON_FOOTER_SIZE, 0);
+        let mut common_footer = &mut res[len..];
+        LittleEndian::write_u32(&mut common_footer, self.meta.len() as u32);
+        LittleEndian::write_u32(&mut common_footer[4..], self.tantivy_version.0);
+        LittleEndian::write_u32(&mut common_footer[8..], self.tantivy_version.1);
+        LittleEndian::write_u32(&mut common_footer[12..], self.tantivy_version.2);
+        LittleEndian::write_u32(&mut common_footer[16..], (len + COMMON_FOOTER_SIZE) as u32);
+        res
+    }
+
+    pub fn from_bytes(data: &[u8]) -> Result<Self, io::Error> {
+        let len = data.len();
+        if len < COMMON_FOOTER_SIZE + 4 {
+            // 4 bytes for index version, stored in versioned footer
+            return Err(io::Error::new(
+                    io::ErrorKind::UnexpectedEof,
+                    format!("File corrupted. The footer len must be over 24, while the entire file len is {}", len)
+                    )
+            );
+        }
+
+        let size = LittleEndian::read_u32(&data[len - 4..]) as usize;
+        if len < size as usize {
+            return Err(io::Error::new(
+                io::ErrorKind::UnexpectedEof,
+                format!(
+                    "File corrupted. The footer len is {}, while the entire file len is {}",
+                    size, len
+                ),
+            ));
+        }
+        let footer = &data[len - size as usize..];
+        let meta_len = LittleEndian::read_u32(&footer[size - 20..]) as usize;
+        let tantivy_major = LittleEndian::read_u32(&footer[size - 16..]);
+        let tantivy_minor = LittleEndian::read_u32(&footer[size - 12..]);
+        let tantivy_patch = LittleEndian::read_u32(&footer[size - 8..]);
+        Ok(Footer {
+            tantivy_version: (tantivy_major, tantivy_minor, tantivy_patch),
+            meta: String::from_utf8_lossy(&footer[size - meta_len - 20..size - 20]).into_owned(),
+            versioned_footer: VersionedFooter::from_bytes(&footer[..size - meta_len - 20])?,
+        })
+    }
+
+    pub fn extract_footer(source: ReadOnlySource) -> Result<(Footer, ReadOnlySource), io::Error> {
+        let footer = Footer::from_bytes(source.as_slice())?;
+        let reader = source.slice_to(source.as_slice().len() - footer.size());
+        Ok((footer, reader))
+    }
+
+    pub fn size(&self) -> usize {
+        self.versioned_footer.size() as usize + self.meta.len() + 20
+    }
+}
+
+#[derive(Debug, Clone, PartialEq)]
+pub enum VersionedFooter {
+    UnknownVersion { version: u32, size: u32 },
+    V0(u32), // crc
+}
+
+impl VersionedFooter {
+    pub fn to_bytes(&self) -> Vec<u8> {
+        match self {
+            VersionedFooter::V0(crc) => {
+                let mut res = vec![0; 8];
+                LittleEndian::write_u32(&mut res, 0);
+                LittleEndian::write_u32(&mut res[4..], *crc);
+                res
+            }
+            VersionedFooter::UnknownVersion { .. } => {
+                panic!("Unsupported index should never get serialized");
+            }
+        }
+    }
+
+    pub fn from_bytes(footer: &[u8]) -> Result<Self, io::Error> {
+        assert!(footer.len() >= 4);
+        let version = LittleEndian::read_u32(footer);
+        match version {
+            0 => {
+                if footer.len() == 8 {
+                    Ok(VersionedFooter::V0(LittleEndian::read_u32(&footer[4..])))
+                } else {
+                    Err(io::Error::new(
+                        io::ErrorKind::UnexpectedEof,
+                        format!(
+                            "File corrupted. The versioned footer len is {}, while it should be 8",
+                            footer.len()
+                        ),
+                    ))
+                }
+            }
+            version => Ok(VersionedFooter::UnknownVersion {
+                version,
+                size: footer.len() as u32,
+            }),
+        }
+    }
+
+    pub fn size(&self) -> u32 {
+        match self {
+            VersionedFooter::V0(_) => 8,
+            VersionedFooter::UnknownVersion { size, .. } => *size,
+        }
+    }
+
+    pub fn version(&self) -> u32 {
+        match self {
+            VersionedFooter::V0(_) => 0,
+            VersionedFooter::UnknownVersion { version, .. } => *version,
+        }
+    }
+
+    pub fn crc(&self) -> Option<u32> {
+        match self {
+            VersionedFooter::V0(crc) => Some(*crc),
+            VersionedFooter::UnknownVersion { .. } => None,
+        }
+    }
+}
+
+pub(crate) struct FooterProxy<W: TerminatingWrite> {
+    /// always Some except after terminate call
+    hasher: Option<Hasher>,
+    /// always Some except after terminate call
+    writer: Option<W>,
+}
+
+impl<W: TerminatingWrite> FooterProxy<W> {
+    pub fn new(writer: W) -> Self {
+        FooterProxy {
+            hasher: Some(Hasher::new()),
+            writer: Some(writer),
+        }
+    }
+}
+
+impl<W: TerminatingWrite> Write for FooterProxy<W> {
+    fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
+        let count = self.writer.as_mut().unwrap().write(buf)?;
+        self.hasher.as_mut().unwrap().update(&buf[..count]);
+        Ok(count)
+    }
+
+    fn flush(&mut self) -> io::Result<()> {
+        self.writer.as_mut().unwrap().flush()
+    }
+}
+
+impl<W: TerminatingWrite> TerminatingWrite for FooterProxy<W> {
+    fn terminate_ref(&mut self, _: AntiCallToken) -> io::Result<()> {
+        let crc = self.hasher.take().unwrap().finalize();
+
+        let footer = Footer::new(VersionedFooter::V0(crc)).to_bytes();
+        let mut writer = self.writer.take().unwrap();
+        writer.write_all(&footer)?;
+        writer.terminate()
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use crate::directory::footer::{Footer, VersionedFooter};
+
+    #[test]
+    fn test_serialize_deserialize_footer() {
+        let crc = 123456;
+        let footer = Footer::new(VersionedFooter::V0(crc));
+        let footer_bytes = footer.to_bytes();
+
+        assert_eq!(Footer::from_bytes(&footer_bytes).unwrap(), footer);
+    }
+}

src/directory/managed_directory.rs

@@ -1,5 +1,6 @@
 use crate::core::MANAGED_FILEPATH;
 use crate::directory::error::{DeleteError, IOError, LockError, OpenReadError, OpenWriteError};
+use crate::directory::footer::{Footer, FooterProxy};
 use crate::directory::DirectoryLock;
 use crate::directory::Lock;
 use crate::directory::META_LOCK;
@@ -8,6 +9,7 @@ use crate::directory::{WatchCallback, WatchHandle};
 use crate::error::DataCorruption;
 use crate::Directory;
 use crate::Result;
+use crc32fast::Hasher;
 use serde_json;
 use std::collections::HashSet;
 use std::io;
@@ -207,17 +209,59 @@ impl ManagedDirectory {
         }
         Ok(())
     }
+
+    /// Verify checksum of a managed file
+    pub fn validate_checksum(&self, path: &Path) -> result::Result<bool, OpenReadError> {
+        let reader = self.directory.open_read(path)?;
+        let (footer, data) = Footer::extract_footer(reader)
+            .map_err(|err| IOError::with_path(path.to_path_buf(), err))?;
+        let mut hasher = Hasher::new();
+        hasher.update(data.as_slice());
+        let crc = hasher.finalize();
+        Ok(footer
+            .versioned_footer
+            .crc()
+            .map(|v| v == crc)
+            .unwrap_or(false))
+    }
+
+    /// List files for which checksum does not match content
+    pub fn list_damaged(&self) -> result::Result<HashSet<PathBuf>, OpenReadError> {
+        let mut hashset = HashSet::new();
+        let managed_paths = self
+            .meta_informations
+            .read()
+            .expect("Managed directory rlock poisoned in list damaged.")
+            .managed_paths
+            .clone();
+
+        for path in managed_paths.into_iter() {
+            if !self.validate_checksum(&path)? {
+                hashset.insert(path);
+            }
+        }
+        Ok(hashset)
+    }
 }
 
 impl Directory for ManagedDirectory {
     fn open_read(&self, path: &Path) -> result::Result<ReadOnlySource, OpenReadError> {
-        self.directory.open_read(path)
+        let read_only_source = self.directory.open_read(path)?;
+        let (_footer, reader) = Footer::extract_footer(read_only_source)
+            .map_err(|err| IOError::with_path(path.to_path_buf(), err))?;
+        Ok(reader)
     }
 
     fn open_write(&mut self, path: &Path) -> result::Result<WritePtr, OpenWriteError> {
         self.register_file_as_managed(path)
             .map_err(|e| IOError::with_path(path.to_owned(), e))?;
-        self.directory.open_write(path)
+        Ok(io::BufWriter::new(Box::new(FooterProxy::new(
+            self.directory
+                .open_write(path)?
+                .into_inner()
+                .map_err(|_| ())
+                .expect("buffer should be empty"),
+        ))))
     }
 
     fn atomic_write(&mut self, path: &Path, data: &[u8]) -> io::Result<()> {
@@ -259,8 +303,9 @@ impl Clone for ManagedDirectory {
 #[cfg(test)]
 mod tests_mmap_specific {
 
-    use crate::directory::{Directory, ManagedDirectory, MmapDirectory};
+    use crate::directory::{Directory, ManagedDirectory, MmapDirectory, TerminatingWrite};
     use std::collections::HashSet;
+    use std::fs::OpenOptions;
     use std::io::Write;
     use std::path::{Path, PathBuf};
     use tempfile::TempDir;
@@ -275,8 +320,8 @@ mod tests_mmap_specific {
         {
             let mmap_directory = MmapDirectory::open(&tempdir_path).unwrap();
             let mut managed_directory = ManagedDirectory::wrap(mmap_directory).unwrap();
-            let mut write_file = managed_directory.open_write(test_path1).unwrap();
-            write_file.flush().unwrap();
+            let write_file = managed_directory.open_write(test_path1).unwrap();
+            write_file.terminate().unwrap();
             managed_directory
                 .atomic_write(test_path2, &[0u8, 1u8])
                 .unwrap();
@@ -310,9 +355,9 @@ mod tests_mmap_specific {
 
         let mmap_directory = MmapDirectory::open(&tempdir_path).unwrap();
         let mut managed_directory = ManagedDirectory::wrap(mmap_directory).unwrap();
-        managed_directory
-            .atomic_write(test_path1, &vec![0u8, 1u8])
-            .unwrap();
+        let mut write = managed_directory.open_write(test_path1).unwrap();
+        write.write_all(&[0u8, 1u8]).unwrap();
+        write.terminate().unwrap();
         assert!(managed_directory.exists(test_path1));
 
         let _mmap_read = managed_directory.open_read(test_path1).unwrap();
@@ -331,4 +376,38 @@ mod tests_mmap_specific {
         }
     }
 
+    #[test]
+    fn test_checksum() {
+        let test_path1: &'static Path = Path::new("some_path_for_test");
+        let test_path2: &'static Path = Path::new("other_test_path");
+
+        let tempdir = TempDir::new().unwrap();
+        let tempdir_path = PathBuf::from(tempdir.path());
+
+        let mmap_directory = MmapDirectory::open(&tempdir_path).unwrap();
+        let mut managed_directory = ManagedDirectory::wrap(mmap_directory).unwrap();
+        let mut write = managed_directory.open_write(test_path1).unwrap();
+        write.write_all(&[0u8, 1u8]).unwrap();
+        write.terminate().unwrap();
+
+        let mut write = managed_directory.open_write(test_path2).unwrap();
+        write.write_all(&[3u8, 4u8, 5u8]).unwrap();
+        write.terminate().unwrap();
+
+        assert!(managed_directory.list_damaged().unwrap().is_empty());
+
+        let mut corrupted_path = tempdir_path.clone();
+        corrupted_path.push(test_path2);
+        let mut file = OpenOptions::new()
+            .write(true)
+            .open(&corrupted_path)
+            .unwrap();
+        file.write_all(&[255u8]).unwrap();
+        file.flush().unwrap();
+        drop(file);
+
+        let damaged = managed_directory.list_damaged().unwrap();
+        assert_eq!(damaged.len(), 1);
+        assert!(damaged.contains(test_path2));
+    }
 }

src/directory/mmap_directory.rs

@@ -11,6 +11,7 @@ use crate::directory::error::{
     DeleteError, IOError, OpenDirectoryError, OpenReadError, OpenWriteError,
 };
 use crate::directory::read_only_source::BoxedData;
+use crate::directory::AntiCallToken;
 use crate::directory::Directory;
 use crate::directory::DirectoryLock;
 use crate::directory::Lock;
@@ -18,7 +19,7 @@ use crate::directory::ReadOnlySource;
 use crate::directory::WatchCallback;
 use crate::directory::WatchCallbackList;
 use crate::directory::WatchHandle;
-use crate::directory::WritePtr;
+use crate::directory::{TerminatingWrite, WritePtr};
 use atomicwrites;
 use memmap::Mmap;
 use std::collections::HashMap;
@@ -141,42 +142,28 @@ impl MmapCache {
     }
 }
 
-struct InnerWatcherWrapper {
-    _watcher: Mutex<notify::RecommendedWatcher>,
-    watcher_router: WatchCallbackList,
-}
-
-impl InnerWatcherWrapper {
-    pub fn new(path: &Path) -> Result<(Self, Receiver<notify::RawEvent>), notify::Error> {
-        let (tx, watcher_recv): (Sender<RawEvent>, Receiver<RawEvent>) = channel();
-        // We need to initialize the
-        let mut watcher = notify::raw_watcher(tx)?;
-        watcher.watch(path, RecursiveMode::Recursive)?;
-        let inner = InnerWatcherWrapper {
-            _watcher: Mutex::new(watcher),
-            watcher_router: Default::default(),
-        };
-        Ok((inner, watcher_recv))
-    }
-}
-
-#[derive(Clone)]
 struct WatcherWrapper {
-    inner: Arc<InnerWatcherWrapper>,
+    _watcher: Mutex<notify::RecommendedWatcher>,
+    watcher_router: Arc<WatchCallbackList>,
 }
 
 impl WatcherWrapper {
     pub fn new(path: &Path) -> Result<Self, OpenDirectoryError> {
-        let (inner, watcher_recv) = InnerWatcherWrapper::new(path).map_err(|err| match err {
-            notify::Error::PathNotFound => OpenDirectoryError::DoesNotExist(path.to_owned()),
-            _ => {
-                panic!("Unknown error while starting watching directory {:?}", path);
-            }
-        })?;
-        let watcher_wrapper = WatcherWrapper {
-            inner: Arc::new(inner),
-        };
-        let watcher_wrapper_clone = watcher_wrapper.clone();
+        let (tx, watcher_recv): (Sender<RawEvent>, Receiver<RawEvent>) = channel();
+        // We need to initialize the
+        let watcher = notify::raw_watcher(tx)
+            .and_then(|mut watcher| {
+                watcher.watch(path, RecursiveMode::Recursive)?;
+                Ok(watcher)
+            })
+            .map_err(|err| match err {
+                notify::Error::PathNotFound => OpenDirectoryError::DoesNotExist(path.to_owned()),
+                _ => {
+                    panic!("Unknown error while starting watching directory {:?}", path);
+                }
+            })?;
+        let watcher_router: Arc<WatchCallbackList> = Default::default();
+        let watcher_router_clone = watcher_router.clone();
         thread::Builder::new()
             .name("meta-file-watch-thread".to_string())
             .spawn(move || {
@@ -187,7 +174,7 @@ impl WatcherWrapper {
                             // We might want to be more accurate than this at one point.
                             if let Some(filename) = changed_path.file_name() {
                                 if filename == *META_FILEPATH {
-                                    watcher_wrapper_clone.inner.watcher_router.broadcast();
+                                    watcher_router_clone.broadcast();
                                 }
                             }
                         }
@@ -200,13 +187,15 @@ impl WatcherWrapper {
                         }
                     }
                 }
-            })
-            .expect("Failed to spawn thread to watch meta.json");
-        Ok(watcher_wrapper)
+            })?;
+        Ok(WatcherWrapper {
+            _watcher: Mutex::new(watcher),
+            watcher_router,
+        })
     }
 
     pub fn watch(&mut self, watch_callback: WatchCallback) -> WatchHandle {
-        self.inner.watcher_router.subscribe(watch_callback)
+        self.watcher_router.subscribe(watch_callback)
     }
 }
 
@@ -265,7 +254,7 @@ impl MmapDirectoryInner {
             }
         }
         if let Some(watch_wrapper) = self.watcher.write().unwrap().as_mut() {
-            return Ok(watch_wrapper.watch(watch_callback));
+            Ok(watch_wrapper.watch(watch_callback))
         } else {
             unreachable!("At this point, watch wrapper is supposed to be initialized");
         }
@@ -412,6 +401,12 @@ impl Seek for SafeFileWriter {
     }
 }
 
+impl TerminatingWrite for SafeFileWriter {
+    fn terminate_ref(&mut self, _: AntiCallToken) -> io::Result<()> {
+        self.flush()
+    }
+}
+
 impl Directory for MmapDirectory {
     fn open_read(&self, path: &Path) -> result::Result<ReadOnlySource, OpenReadError> {
         debug!("Open Read {:?}", path);

src/directory/mod.rs

@@ -9,6 +9,7 @@ mod mmap_directory;
 
 mod directory;
 mod directory_lock;
+mod footer;
 mod managed_directory;
 mod ram_directory;
 mod read_only_source;
@@ -24,18 +25,49 @@ pub use self::ram_directory::RAMDirectory;
 pub use self::read_only_source::ReadOnlySource;
 pub(crate) use self::watch_event_router::WatchCallbackList;
 pub use self::watch_event_router::{WatchCallback, WatchHandle};
-use std::io::{BufWriter, Write};
+use std::io::{self, BufWriter, Write};
 
 #[cfg(feature = "mmap")]
 pub use self::mmap_directory::MmapDirectory;
 
 pub use self::managed_directory::ManagedDirectory;
 
+/// Struct used to prevent from calling [`terminate_ref`](trait.TerminatingWrite#method.terminate_ref) directly
+pub struct AntiCallToken(());
+
+/// Trait used to indicate when no more write need to be done on a writer
+pub trait TerminatingWrite: Write {
+    /// Indicate that the writer will no longer be used. Internally call terminate_ref.
+    fn terminate(mut self) -> io::Result<()>
+    where
+        Self: Sized,
+    {
+        self.terminate_ref(AntiCallToken(()))
+    }
+
+    /// You should implement this function to define custom behavior.
+    /// This function should flush any buffer it may hold.
+    fn terminate_ref(&mut self, _: AntiCallToken) -> io::Result<()>;
+}
+
+impl<W: TerminatingWrite + ?Sized> TerminatingWrite for Box<W> {
+    fn terminate_ref(&mut self, token: AntiCallToken) -> io::Result<()> {
+        self.as_mut().terminate_ref(token)
+    }
+}
+
+impl<W: TerminatingWrite> TerminatingWrite for BufWriter<W> {
+    fn terminate_ref(&mut self, a: AntiCallToken) -> io::Result<()> {
+        self.flush()?;
+        self.get_mut().terminate_ref(a)
+    }
+}
+
 /// Write object for Directory.
 ///
 /// `WritePtr` are required to implement both Write
 /// and Seek.
-pub type WritePtr = BufWriter<Box<dyn Write>>;
+pub type WritePtr = BufWriter<Box<dyn TerminatingWrite>>;
 
 #[cfg(test)]
 mod tests;

src/directory/ram_directory.rs

@@ -1,8 +1,9 @@
 use crate::core::META_FILEPATH;
 use crate::directory::error::{DeleteError, OpenReadError, OpenWriteError};
+use crate::directory::AntiCallToken;
 use crate::directory::WatchCallbackList;
-use crate::directory::WritePtr;
 use crate::directory::{Directory, ReadOnlySource, WatchCallback, WatchHandle};
+use crate::directory::{TerminatingWrite, WritePtr};
 use fail::fail_point;
 use std::collections::HashMap;
 use std::fmt;
@@ -71,6 +72,12 @@ impl Write for VecWriter {
     }
 }
 
+impl TerminatingWrite for VecWriter {
+    fn terminate_ref(&mut self, _: AntiCallToken) -> io::Result<()> {
+        self.flush()
+    }
+}
+
 #[derive(Default)]
 struct InnerDirectory {
     fs: HashMap<PathBuf, ReadOnlySource>,

src/directory/tests.rs

@@ -127,7 +127,7 @@ fn test_watch(directory: &mut dyn Directory) {
         assert!(directory
             .atomic_write(Path::new("meta.json"), b"random_test_data_2")
             .is_ok());
-        for _ in 0..100 {
+        for _ in 0..1_000 {
             if counter.load(Ordering::SeqCst) > i {
                 break;
             }

src/directory/watch_event_router.rs

@@ -152,5 +152,4 @@ mod tests {
         thread::sleep(Duration::from_millis(WAIT_TIME));
         assert_eq!(2, counter.load(Ordering::SeqCst));
     }
-
 }

src/fastfield/mod.rs

@@ -429,7 +429,6 @@ mod tests {
             }
         }
     }
-
 }
 
 #[cfg(all(test, feature = "unstable"))]
@@ -437,9 +436,9 @@ mod bench {
     use super::tests::FIELD;
     use super::tests::{generate_permutation, SCHEMA};
     use super::*;
-    use common::CompositeFile;
-    use directory::{Directory, RAMDirectory, WritePtr};
-    use fastfield::FastFieldReader;
+    use crate::common::CompositeFile;
+    use crate::directory::{Directory, RAMDirectory, WritePtr};
+    use crate::fastfield::FastFieldReader;
     use std::collections::HashMap;
     use std::path::Path;
     use test::{self, Bencher};
@@ -537,5 +536,4 @@ mod bench {
             });
         }
     }
-
 }

src/fastfield/multivalued/writer.rs

@@ -5,8 +5,8 @@ use crate::postings::UnorderedTermId;
 use crate::schema::{Document, Field};
 use crate::termdict::TermOrdinal;
 use crate::DocId;
+use fnv::FnvHashMap;
 use itertools::Itertools;
-use std::collections::HashMap;
 use std::io;
 
 /// Writer for multi-valued (as in, more than one value per document)
@@ -102,7 +102,7 @@ impl MultiValueIntFastFieldWriter {
     pub fn serialize(
         &self,
         serializer: &mut FastFieldSerializer,
-        mapping_opt: Option<&HashMap<UnorderedTermId, TermOrdinal>>,
+        mapping_opt: Option<&FnvHashMap<UnorderedTermId, TermOrdinal>>,
     ) -> io::Result<()> {
         {
             // writing the offset index

src/fastfield/readers.rs

@@ -59,8 +59,7 @@ impl FastFieldReaders {
             fast_bytes: Default::default(),
             fast_fields_composite: fast_fields_composite.clone(),
         };
-        for (field_id, field_entry) in schema.fields().iter().enumerate() {
-            let field = Field(field_id as u32);
+        for (field, field_entry) in schema.fields() {
             let field_type = field_entry.field_type();
             if field_type == &FieldType::Bytes {
                 let idx_reader = fast_fields_composite

src/fastfield/writer.rs

@@ -6,6 +6,7 @@ use crate::fastfield::{BytesFastFieldWriter, FastFieldSerializer};
 use crate::postings::UnorderedTermId;
 use crate::schema::{Cardinality, Document, Field, FieldType, Schema};
 use crate::termdict::TermOrdinal;
+use fnv::FnvHashMap;
 use std::collections::HashMap;
 use std::io;
 
@@ -23,8 +24,7 @@ impl FastFieldsWriter {
         let mut multi_values_writers = Vec::new();
         let mut bytes_value_writers = Vec::new();
 
-        for (field_id, field_entry) in schema.fields().iter().enumerate() {
-            let field = Field(field_id as u32);
+        for (field, field_entry) in schema.fields() {
             let default_value = match *field_entry.field_type() {
                 FieldType::I64(_) => common::i64_to_u64(0i64),
                 FieldType::F64(_) => common::f64_to_u64(0.0f64),
@@ -116,7 +116,7 @@ impl FastFieldsWriter {
     pub fn serialize(
         &self,
         serializer: &mut FastFieldSerializer,
-        mapping: &HashMap<Field, HashMap<UnorderedTermId, TermOrdinal>>,
+        mapping: &HashMap<Field, FnvHashMap<UnorderedTermId, TermOrdinal>>,
     ) -> io::Result<()> {
         for field_writer in &self.single_value_writers {
             field_writer.serialize(serializer)?;

src/fieldnorm/writer.rs

@@ -22,11 +22,14 @@ impl FieldNormsWriter {
     pub(crate) fn fields_with_fieldnorm(schema: &Schema) -> Vec<Field> {
         schema
             .fields()
-            .iter()
-            .enumerate()
-            .filter(|&(_, field_entry)| field_entry.is_indexed())
-            .map(|(field, _)| Field(field as u32))
-            .collect::<Vec<Field>>()
+            .filter_map(|(field, field_entry)| {
+                if field_entry.is_indexed() {
+                    Some(field)
+                } else {
+                    None
+                }
+            })
+            .collect::<Vec<_>>()
     }
 
     /// Initialize with state for tracking the field norm fields
@@ -35,7 +38,7 @@ impl FieldNormsWriter {
         let fields = FieldNormsWriter::fields_with_fieldnorm(schema);
         let max_field = fields
             .iter()
-            .map(|field| field.0)
+            .map(Field::field_id)
             .max()
             .map(|max_field_id| max_field_id as usize + 1)
             .unwrap_or(0);
@@ -50,8 +53,8 @@ impl FieldNormsWriter {
     ///
     /// Will extend with 0-bytes for documents that have not been seen.
     pub fn fill_up_to_max_doc(&mut self, max_doc: DocId) {
-        for &field in self.fields.iter() {
-            self.fieldnorms_buffer[field.0 as usize].resize(max_doc as usize, 0u8);
+        for field in self.fields.iter() {
+            self.fieldnorms_buffer[field.field_id() as usize].resize(max_doc as usize, 0u8);
         }
     }
 
@@ -64,7 +67,7 @@ impl FieldNormsWriter {
     /// * field     - the field being set
     /// * fieldnorm - the number of terms present in document `doc` in field `field`
     pub fn record(&mut self, doc: DocId, field: Field, fieldnorm: u32) {
-        let fieldnorm_buffer: &mut Vec<u8> = &mut self.fieldnorms_buffer[field.0 as usize];
+        let fieldnorm_buffer: &mut Vec<u8> = &mut self.fieldnorms_buffer[field.field_id() as usize];
         assert!(
             fieldnorm_buffer.len() <= doc as usize,
             "Cannot register a given fieldnorm twice"
@@ -77,7 +80,7 @@ impl FieldNormsWriter {
     /// Serialize the seen fieldnorm values to the serializer for all fields.
     pub fn serialize(&self, fieldnorms_serializer: &mut FieldNormsSerializer) -> io::Result<()> {
         for &field in self.fields.iter() {
-            let fieldnorm_values: &[u8] = &self.fieldnorms_buffer[field.0 as usize][..];
+            let fieldnorm_values: &[u8] = &self.fieldnorms_buffer[field.field_id() as usize][..];
             fieldnorms_serializer.serialize_field(field, fieldnorm_values)?;
         }
         Ok(())

src/indexer/delete_queue.rs

@@ -258,7 +258,7 @@ mod tests {
         let delete_queue = DeleteQueue::new();
 
         let make_op = |i: usize| {
-            let field = Field(1u32);
+            let field = Field::from_field_id(1u32);
             DeleteOperation {
                 opstamp: i as u64,
                 term: Term::from_field_u64(field, i as u64),

src/indexer/index_writer.rs

@@ -8,6 +8,7 @@ use crate::core::SegmentId;
 use crate::core::SegmentMeta;
 use crate::core::SegmentReader;
 use crate::directory::DirectoryLock;
+use crate::directory::TerminatingWrite;
 use crate::docset::DocSet;
 use crate::error::TantivyError;
 use crate::fastfield::write_delete_bitset;
@@ -168,6 +169,7 @@ pub(crate) fn advance_deletes(
             segment = segment.with_delete_meta(num_deleted_docs as u32, target_opstamp);
             let mut delete_file = segment.open_write(SegmentComponent::DELETE)?;
             write_delete_bitset(&delete_bitset, &mut delete_file)?;
+            delete_file.terminate()?;
         }
     }
     segment_entry.set_meta(segment.meta().clone());
@@ -730,7 +732,7 @@ impl IndexWriter {
                 }
                 UserOperation::Add(document) => {
                     let add_operation = AddOperation { opstamp, document };
-                    adds.push(add_operation);
+                    adds.push(add_operpation);
                 }
             }
         }
@@ -1177,5 +1179,4 @@ mod tests {
         assert!(clear_again.is_ok());
         assert!(commit_again.is_ok());
     }
-
 }

src/indexer/merge_operation.rs

@@ -7,6 +7,10 @@ use std::collections::HashSet;
 pub struct MergeOperationInventory(Inventory<InnerMergeOperation>);
 
 impl MergeOperationInventory {
+    pub fn num_merge_operations(&self) -> usize {
+        self.0.list().len()
+    }
+
     pub fn segment_in_merge(&self) -> HashSet<SegmentId> {
         let mut segment_in_merge = HashSet::default();
         for merge_op in self.0.list() {

src/indexer/merge_policy.rs

@@ -12,6 +12,10 @@ pub struct MergeCandidate(pub Vec<SegmentId>);
 /// Every time a the list of segments changes, the segment updater
 /// asks the merge policy if some segments should be merged.
 pub trait MergePolicy: marker::Send + marker::Sync + Debug {
+    fn maximum_num_threads(&self) -> Option<usize> {
+        None
+    }
+
     /// Given the list of segment metas, returns the list of merge candidates.
     ///
     /// This call happens on the segment updater thread, and will block

src/indexer/merger.rs

@@ -190,8 +190,7 @@ impl IndexMerger {
         fast_field_serializer: &mut FastFieldSerializer,
         mut term_ord_mappings: HashMap<Field, TermOrdinalMapping>,
     ) -> Result<()> {
-        for (field_id, field_entry) in self.schema.fields().iter().enumerate() {
-            let field = Field(field_id as u32);
+        for (field, field_entry) in self.schema.fields() {
             let field_type = field_entry.field_type();
             match *field_type {
                 FieldType::HierarchicalFacet => {
@@ -649,15 +648,12 @@ impl IndexMerger {
         serializer: &mut InvertedIndexSerializer,
     ) -> Result<HashMap<Field, TermOrdinalMapping>> {
         let mut term_ordinal_mappings = HashMap::new();
-        for (field_ord, field_entry) in self.schema.fields().iter().enumerate() {
+        for (field, field_entry) in self.schema.fields() {
             if field_entry.is_indexed() {
-                let indexed_field = Field(field_ord as u32);
-                if let Some(term_ordinal_mapping) = self.write_postings_for_field(
-                    indexed_field,
-                    field_entry.field_type(),
-                    serializer,
-                )? {
-                    term_ordinal_mappings.insert(indexed_field, term_ordinal_mapping);
+                if let Some(term_ordinal_mapping) =
+                    self.write_postings_for_field(field, field_entry.field_type(), serializer)?
+                {
+                    term_ordinal_mappings.insert(field, term_ordinal_mapping);
                 }
             }
         }

src/indexer/segment_register.rs

@@ -134,5 +134,4 @@ mod tests {
         }
         assert_eq!(segment_ids(&segment_register), vec![segment_id_merged]);
     }
-
 }

src/indexer/segment_updater.rs

@@ -39,6 +39,7 @@ use std::sync::Arc;
 use std::sync::RwLock;
 use std::thread;
 use std::thread::JoinHandle;
+use std::time::Duration;
 
 /// Save the index meta file.
 /// This operation is atomic :
@@ -200,6 +201,12 @@ impl SegmentUpdater {
     }
 
     pub fn add_segment(&self, segment_entry: SegmentEntry) -> bool {
+        let max_num_threads_opt = self.0.merge_policy.read().unwrap().maximum_num_threads();
+        if let Some(max_num_threads) = max_num_threads_opt {
+            while self.0.merge_operations.num_merge_operations() >= max_num_threads_opt {
+                std::thread::sleep(Duration::from_secs(1u64));
+            }
+        }
         self.run_async(|segment_updater| {
             segment_updater.0.segment_manager.add_segment(segment_entry);
             segment_updater.consider_merge_options();

src/indexer/segment_writer.rs

@@ -6,11 +6,11 @@ use crate::fieldnorm::FieldNormsWriter;
 use crate::indexer::segment_serializer::SegmentSerializer;
 use crate::postings::compute_table_size;
 use crate::postings::MultiFieldPostingsWriter;
-use crate::schema::FieldEntry;
 use crate::schema::FieldType;
 use crate::schema::Schema;
 use crate::schema::Term;
 use crate::schema::Value;
+use crate::schema::{Field, FieldEntry};
 use crate::tokenizer::BoxedTokenizer;
 use crate::tokenizer::FacetTokenizer;
 use crate::tokenizer::{TokenStream, Tokenizer};
@@ -70,12 +70,10 @@ impl SegmentWriter {
         let table_num_bits = initial_table_size(memory_budget)?;
         let segment_serializer = SegmentSerializer::for_segment(&mut segment)?;
         let multifield_postings = MultiFieldPostingsWriter::new(schema, table_num_bits);
-        let tokenizers =
-            schema
-                .fields()
-                .iter()
-                .map(FieldEntry::field_type)
-                .map(|field_type| match *field_type {
+        let tokenizers = schema
+            .fields()
+            .map(
+                |(_, field_entry): (Field, &FieldEntry)| match field_entry.field_type() {
                     FieldType::Str(ref text_options) => text_options
                         .get_indexing_options()
                         .and_then(|text_index_option| {
@@ -83,8 +81,9 @@ impl SegmentWriter {
                             segment.index().tokenizers().get(tokenizer_name)
                         }),
                     _ => None,
-                })
-                .collect();
+                },
+            )
+            .collect();
         Ok(SegmentWriter {
             max_doc: 0,
             multifield_postings,
@@ -160,7 +159,7 @@ impl SegmentWriter {
                 }
                 FieldType::Str(_) => {
                     let num_tokens = if let Some(ref mut tokenizer) =
-                        self.tokenizers[field.0 as usize]
+                        self.tokenizers[field.field_id() as usize]
                     {
                         let texts: Vec<&str> = field_values
                             .iter()
@@ -296,5 +295,4 @@ mod tests {
         assert_eq!(initial_table_size(10_000_000).unwrap(), 17);
         assert_eq!(initial_table_size(1_000_000_000).unwrap(), 19);
     }
-
 }

src/lib.rs

@@ -212,15 +212,13 @@ pub type Score = f32;
 pub type SegmentLocalId = u32;
 
 impl DocAddress {
-    /// Return the segment ordinal.
-    /// The segment ordinal is an id identifying the segment
-    /// hosting the document. It is only meaningful, in the context
-    /// of a searcher.
+    /// Return the segment ordinal id that identifies the segment
+    /// hosting the document in the `Searcher` it is called from.
     pub fn segment_ord(self) -> SegmentLocalId {
         self.0
     }
 
-    /// Return the segment local `DocId`
+    /// Return the segment-local `DocId`
     pub fn doc(self) -> DocId {
         self.1
     }
@@ -229,11 +227,11 @@ impl DocAddress {
 /// `DocAddress` contains all the necessary information
 /// to identify a document given a `Searcher` object.
 ///
-/// It consists in an id identifying its segment, and
-/// its segment-local `DocId`.
+/// It consists of an id identifying its segment, and
+/// a segment-local `DocId`.
 ///
 /// The id used for the segment is actually an ordinal
-/// in the list of segment hold by a `Searcher`.
+/// in the list of `Segment`s held by a `Searcher`.
 #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
 pub struct DocAddress(pub SegmentLocalId, pub DocId);
 

src/postings/compression/mod.rs

@@ -274,13 +274,15 @@ pub mod tests {
 mod bench {
 
     use super::*;
+    use rand::rngs::StdRng;
+    use rand::Rng;
     use rand::SeedableRng;
-    use rand::{Rng, XorShiftRng};
     use test::Bencher;
 
     fn generate_array_with_seed(n: usize, ratio: f64, seed_val: u8) -> Vec<u32> {
-        let seed: &[u8; 16] = &[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, seed_val];
-        let mut rng: XorShiftRng = XorShiftRng::from_seed(*seed);
+        let mut seed: [u8; 32] = [0; 32];
+        seed[31] = seed_val;
+        let mut rng = StdRng::from_seed(seed);
         (0u32..).filter(|_| rng.gen_bool(ratio)).take(n).collect()
     }
 

src/postings/mod.rs

@@ -356,9 +356,9 @@ pub mod tests {
 
     #[test]
     fn test_skip_next() {
-        let term_0 = Term::from_field_u64(Field(0), 0);
-        let term_1 = Term::from_field_u64(Field(0), 1);
-        let term_2 = Term::from_field_u64(Field(0), 2);
+        let term_0 = Term::from_field_u64(Field::from_field_id(0), 0);
+        let term_1 = Term::from_field_u64(Field::from_field_id(0), 1);
+        let term_2 = Term::from_field_u64(Field::from_field_id(0), 2);
 
         let num_docs = 300u32;
 
@@ -511,19 +511,19 @@ pub mod tests {
     }
 
     pub static TERM_A: Lazy<Term> = Lazy::new(|| {
-        let field = Field(0);
+        let field = Field::from_field_id(0);
         Term::from_field_text(field, "a")
     });
     pub static TERM_B: Lazy<Term> = Lazy::new(|| {
-        let field = Field(0);
+        let field = Field::from_field_id(0);
         Term::from_field_text(field, "b")
     });
     pub static TERM_C: Lazy<Term> = Lazy::new(|| {
-        let field = Field(0);
+        let field = Field::from_field_id(0);
         Term::from_field_text(field, "c")
     });
     pub static TERM_D: Lazy<Term> = Lazy::new(|| {
-        let field = Field(0);
+        let field = Field::from_field_id(0);
         Term::from_field_text(field, "d")
     });
 
@@ -622,23 +622,23 @@ pub mod tests {
             assert!(!postings_unopt.advance());
         }
     }
-
 }
 
 #[cfg(all(test, feature = "unstable"))]
 mod bench {
 
     use super::tests::*;
-    use docset::SkipResult;
-    use query::Intersection;
-    use schema::IndexRecordOption;
+    use crate::docset::SkipResult;
+    use crate::query::Intersection;
+    use crate::schema::IndexRecordOption;
+    use crate::tests;
+    use crate::DocSet;
     use test::{self, Bencher};
-    use tests;
-    use DocSet;
 
     #[bench]
     fn bench_segment_postings(b: &mut Bencher) {
-        let searcher = INDEX.searcher();
+        let reader = INDEX.reader().unwrap();
+        let searcher = reader.searcher();
         let segment_reader = searcher.segment_reader(0);
 
         b.iter(|| {
@@ -652,7 +652,8 @@ mod bench {
 
     #[bench]
     fn bench_segment_intersection(b: &mut Bencher) {
-        let searcher = INDEX.searcher();
+        let reader = INDEX.reader().unwrap();
+        let searcher = reader.searcher();
         let segment_reader = searcher.segment_reader(0);
         b.iter(|| {
             let segment_postings_a = segment_reader
@@ -682,7 +683,8 @@ mod bench {
     }
 
     fn bench_skip_next(p: f64, b: &mut Bencher) {
-        let searcher = INDEX.searcher();
+        let reader = INDEX.reader().unwrap();
+        let searcher = reader.searcher();
         let segment_reader = searcher.segment_reader(0);
         let docs = tests::sample(segment_reader.num_docs(), p);
 
@@ -737,7 +739,8 @@ mod bench {
 
     #[bench]
     fn bench_iterate_segment_postings(b: &mut Bencher) {
-        let searcher = INDEX.searcher();
+        let reader = INDEX.reader().unwrap();
+        let searcher = reader.searcher();
         let segment_reader = searcher.segment_reader(0);
         b.iter(|| {
             let n: u32 = test::black_box(17);

src/postings/postings_writer.rs

@@ -12,6 +12,7 @@ use crate::tokenizer::TokenStream;
 use crate::tokenizer::{Token, MAX_TOKEN_LEN};
 use crate::DocId;
 use crate::Result;
+use fnv::FnvHashMap;
 use std::collections::HashMap;
 use std::io;
 use std::marker::PhantomData;
@@ -60,12 +61,12 @@ fn make_field_partition(
         .iter()
         .map(|(key, _, _)| Term::wrap(key).field())
         .enumerate();
-    let mut prev_field = Field(u32::max_value());
+    let mut prev_field_opt = None;
     let mut fields = vec![];
     let mut offsets = vec![];
     for (offset, field) in term_offsets_it {
-        if field != prev_field {
-            prev_field = field;
+        if Some(field) != prev_field_opt {
+            prev_field_opt = Some(field);
             fields.push(field);
             offsets.push(offset);
         }
@@ -85,8 +86,7 @@ impl MultiFieldPostingsWriter {
         let term_index = TermHashMap::new(table_bits);
         let per_field_postings_writers: Vec<_> = schema
             .fields()
-            .iter()
-            .map(|field_entry| posting_from_field_entry(field_entry))
+            .map(|(_, field_entry)| posting_from_field_entry(field_entry))
             .collect();
         MultiFieldPostingsWriter {
             heap: MemoryArena::new(),
@@ -106,7 +106,8 @@ impl MultiFieldPostingsWriter {
         field: Field,
         token_stream: &mut dyn TokenStream,
     ) -> u32 {
-        let postings_writer = self.per_field_postings_writers[field.0 as usize].deref_mut();
+        let postings_writer =
+            self.per_field_postings_writers[field.field_id() as usize].deref_mut();
         postings_writer.index_text(
             &mut self.term_index,
             doc,
@@ -117,7 +118,8 @@ impl MultiFieldPostingsWriter {
     }
 
     pub fn subscribe(&mut self, doc: DocId, term: &Term) -> UnorderedTermId {
-        let postings_writer = self.per_field_postings_writers[term.field().0 as usize].deref_mut();
+        let postings_writer =
+            self.per_field_postings_writers[term.field().field_id() as usize].deref_mut();
         postings_writer.subscribe(&mut self.term_index, doc, 0u32, term, &mut self.heap)
     }
 
@@ -127,12 +129,12 @@ impl MultiFieldPostingsWriter {
     pub fn serialize(
         &self,
         serializer: &mut InvertedIndexSerializer,
-    ) -> Result<HashMap<Field, HashMap<UnorderedTermId, TermOrdinal>>> {
+    ) -> Result<HashMap<Field, FnvHashMap<UnorderedTermId, TermOrdinal>>> {
         let mut term_offsets: Vec<(&[u8], Addr, UnorderedTermId)> =
             self.term_index.iter().collect();
         term_offsets.sort_unstable_by_key(|&(k, _, _)| k);
 
-        let mut unordered_term_mappings: HashMap<Field, HashMap<UnorderedTermId, TermOrdinal>> =
+        let mut unordered_term_mappings: HashMap<Field, FnvHashMap<UnorderedTermId, TermOrdinal>> =
             HashMap::new();
 
         let field_offsets = make_field_partition(&term_offsets);
@@ -147,7 +149,7 @@ impl MultiFieldPostingsWriter {
                     let unordered_term_ids = term_offsets[start..stop]
                         .iter()
                         .map(|&(_, _, bucket)| bucket);
-                    let mapping: HashMap<UnorderedTermId, TermOrdinal> = unordered_term_ids
+                    let mapping: FnvHashMap<UnorderedTermId, TermOrdinal> = unordered_term_ids
                         .enumerate()
                         .map(|(term_ord, unord_term_id)| {
                             (unord_term_id as UnorderedTermId, term_ord as TermOrdinal)
@@ -159,7 +161,7 @@ impl MultiFieldPostingsWriter {
                 FieldType::Bytes => {}
             }
 
-            let postings_writer = &self.per_field_postings_writers[field.0 as usize];
+            let postings_writer = &self.per_field_postings_writers[field.field_id() as usize];
             let mut field_serializer =
                 serializer.new_field(field, postings_writer.total_num_tokens())?;
             postings_writer.serialize(

src/postings/serializer.rs

@@ -141,10 +141,7 @@ impl<'a> FieldSerializer<'a> {
             FieldType::Str(ref text_options) => {
                 if let Some(text_indexing_options) = text_options.get_indexing_options() {
                     let index_option = text_indexing_options.index_option();
-                    (
-                        index_option.is_termfreq_enabled(),
-                        index_option.is_position_enabled(),
-                    )
+                    (index_option.has_freq(), index_option.has_positions())
                 } else {
                     (false, false)
                 }

src/postings/stacker/expull.rs

@@ -310,6 +310,7 @@ mod bench {
     use super::super::MemoryArena;
     use super::ExpUnrolledLinkedList;
     use byteorder::{NativeEndian, WriteBytesExt};
+    use std::iter;
     use test::Bencher;
 
     const NUM_STACK: usize = 10_000;
@@ -335,11 +336,10 @@ mod bench {
     fn bench_push_stack(bench: &mut Bencher) {
         bench.iter(|| {
             let mut heap = MemoryArena::new();
-            let mut stacks = Vec::with_capacity(100);
-            for _ in 0..NUM_STACK {
-                let mut stack = ExpUnrolledLinkedList::new();
-                stacks.push(stack);
-            }
+            let mut stacks: Vec<ExpUnrolledLinkedList> =
+                iter::repeat_with(ExpUnrolledLinkedList::new)
+                    .take(NUM_STACK)
+                    .collect();
             for s in 0..NUM_STACK {
                 for i in 0u32..STACK_SIZE {
                     let t = s * 392017 % NUM_STACK;

src/query/all_query.rs

@@ -130,5 +130,4 @@ mod tests {
             assert!(!scorer.advance());
         }
     }
-
 }

src/query/bitset/mod.rs

@@ -216,7 +216,6 @@ mod tests {
             assert!(!docset.advance());
         }
     }
-
 }
 
 #[cfg(all(test, feature = "unstable"))]
@@ -224,13 +223,12 @@ mod bench {
 
     use super::BitSet;
     use super::BitSetDocSet;
-    use test;
-    use tests;
-    use DocSet;
+    use crate::test;
+    use crate::tests;
+    use crate::DocSet;
 
     #[bench]
     fn bench_bitset_1pct_insert(b: &mut test::Bencher) {
-        use tests;
         let els = tests::generate_nonunique_unsorted(1_000_000u32, 10_000);
         b.iter(|| {
             let mut bitset = BitSet::with_max_value(1_000_000);
@@ -242,7 +240,6 @@ mod bench {
 
     #[bench]
     fn bench_bitset_1pct_clone(b: &mut test::Bencher) {
-        use tests;
         let els = tests::generate_nonunique_unsorted(1_000_000u32, 10_000);
         let mut bitset = BitSet::with_max_value(1_000_000);
         for el in els {

src/query/bm25.rs

@@ -137,5 +137,4 @@ mod tests {
     fn test_idf() {
         assert_nearly_equals(idf(1, 2), 0.6931472);
     }
-
 }

src/query/boolean_query/boolean_query.rs

@@ -9,7 +9,8 @@ use crate::Result;
 use crate::Searcher;
 use std::collections::BTreeSet;
 
-/// The boolean query combines a set of queries
+/// The boolean query returns a set of documents
+/// that matches the Boolean combination of constituent subqueries.
 ///
 /// The documents matched by the boolean query are
 /// those which
@@ -19,6 +20,113 @@ use std::collections::BTreeSet;
 /// `MustNot` occurence.
 /// * match at least one of the subqueries that is not
 /// a `MustNot` occurence.
+///
+///
+/// You can combine other query types and their `Occur`ances into one `BooleanQuery`
+///
+/// ```rust
+///use tantivy::collector::Count;
+///use tantivy::doc;
+///use tantivy::query::{BooleanQuery, Occur, PhraseQuery, Query, TermQuery};
+///use tantivy::schema::{IndexRecordOption, Schema, TEXT};
+///use tantivy::Term;
+///use tantivy::{Index, Result};
+///
+///fn main() -> Result<()> {
+///    let mut schema_builder = Schema::builder();
+///    let title = schema_builder.add_text_field("title", TEXT);
+///    let body = schema_builder.add_text_field("body", TEXT);
+///    let schema = schema_builder.build();
+///    let index = Index::create_in_ram(schema);
+///    {
+///        let mut index_writer = index.writer(3_000_000)?;
+///        index_writer.add_document(doc!(
+///            title => "The Name of the Wind",
+///        ));
+///        index_writer.add_document(doc!(
+///            title => "The Diary of Muadib",
+///        ));
+///        index_writer.add_document(doc!(
+///            title => "A Dairy Cow",
+///            body => "hidden",
+///        ));
+///        index_writer.add_document(doc!(
+///            title => "A Dairy Cow",
+///            body => "found",
+///        ));
+///        index_writer.add_document(doc!(
+///            title => "The Diary of a Young Girl",
+///        ));
+///        index_writer.commit().unwrap();
+///    }
+///
+///    let reader = index.reader()?;
+///    let searcher = reader.searcher();
+///
+///    // Make TermQuery's for "girl" and "diary" in the title
+///    let girl_term_query: Box<dyn Query> = Box::new(TermQuery::new(
+///        Term::from_field_text(title, "girl"),
+///        IndexRecordOption::Basic,
+///    ));
+///    let diary_term_query: Box<dyn Query> = Box::new(TermQuery::new(
+///        Term::from_field_text(title, "diary"),
+///        IndexRecordOption::Basic,
+///    ));
+///    // A TermQuery with "found" in the body
+///    let body_term_query: Box<dyn Query> = Box::new(TermQuery::new(
+///        Term::from_field_text(body, "found"),
+///        IndexRecordOption::Basic,
+///    ));
+///    // TermQuery "diary" must and "girl" must not be present
+///    let queries_with_occurs1 = vec![
+///        (Occur::Must, diary_term_query.box_clone()),
+///        (Occur::MustNot, girl_term_query),
+///    ];
+///    // Make a BooleanQuery equivalent to
+///    // title:+diary title:-girl
+///    let diary_must_and_girl_mustnot = BooleanQuery::from(queries_with_occurs1);
+///    let count1 = searcher.search(&diary_must_and_girl_mustnot, &Count)?;
+///    assert_eq!(count1, 1);
+///
+///    // TermQuery for "cow" in the title
+///    let cow_term_query: Box<dyn Query> = Box::new(TermQuery::new(
+///        Term::from_field_text(title, "cow"),
+///        IndexRecordOption::Basic,
+///    ));
+///    // "title:diary OR title:cow"
+///    let title_diary_or_cow = BooleanQuery::from(vec![
+///        (Occur::Should, diary_term_query.box_clone()),
+///        (Occur::Should, cow_term_query),
+///    ]);
+///    let count2 = searcher.search(&title_diary_or_cow, &Count)?;
+///    assert_eq!(count2, 4);
+///
+///    // Make a `PhraseQuery` from a vector of `Term`s
+///    let phrase_query: Box<dyn Query> = Box::new(PhraseQuery::new(vec![
+///        Term::from_field_text(title, "dairy"),
+///        Term::from_field_text(title, "cow"),
+///    ]));
+///    // You can combine subqueries of different types into 1 BooleanQuery:
+///    // `TermQuery` and `PhraseQuery`
+///    // "title:diary OR "dairy cow"
+///    let term_of_phrase_query = BooleanQuery::from(vec![
+///        (Occur::Should, diary_term_query.box_clone()),
+///        (Occur::Should, phrase_query.box_clone()),
+///    ]);
+///    let count3 = searcher.search(&term_of_phrase_query, &Count)?;
+///    assert_eq!(count3, 4);
+///
+///    // You can nest one BooleanQuery inside another
+///    // body:found AND ("title:diary OR "dairy cow")
+///    let nested_query = BooleanQuery::from(vec![
+///        (Occur::Must, body_term_query),
+///        (Occur::Must, Box::new(term_of_phrase_query))
+///    ]);
+///    let count4 = searcher.search(&nested_query, &Count)?;
+///    assert_eq!(count4, 1);
+///    Ok(())
+///}
+/// ```
 #[derive(Debug)]
 pub struct BooleanQuery {
     subqueries: Vec<(Occur, Box<dyn Query>)>,

src/query/boolean_query/mod.rs

@@ -247,9 +247,7 @@ mod tests {
         let reader = index.reader().unwrap();
         let searcher = reader.searcher();
         let query_parser = QueryParser::for_index(&index, vec![title, text]);
-        let query = query_parser
-            .parse_query("Оксана Лифенко")
-            .unwrap();
+        let query = query_parser.parse_query("Оксана Лифенко").unwrap();
         let weight = query.weight(&searcher, true).unwrap();
         let mut scorer = weight.scorer(searcher.segment_reader(0u32)).unwrap();
         scorer.advance();

src/query/exclude.rs

@@ -175,5 +175,4 @@ mod tests {
             sample_skip,
         );
     }
-
 }

src/query/fuzzy_query.rs

@@ -1,14 +1,12 @@
 use crate::error::TantivyError::InvalidArgument;
 use crate::query::{AutomatonWeight, Query, Weight};
 use crate::schema::Term;
-use crate::termdict::WrappedDFA;
 use crate::Result;
 use crate::Searcher;
-use levenshtein_automata::{Distance, LevenshteinAutomatonBuilder, DFA};
+use levenshtein_automata::{LevenshteinAutomatonBuilder, DFA};
 use once_cell::sync::Lazy;
 use std::collections::HashMap;
 use std::ops::Range;
-use derive_builder::Builder;
 
 /// A range of Levenshtein distances that we will build DFAs for our terms
 /// The computation is exponential, so best keep it to low single digits
@@ -26,38 +24,6 @@ static LEV_BUILDER: Lazy<HashMap<(u8, bool), LevenshteinAutomatonBuilder>> = Laz
     lev_builder_cache
 });
 
-
-#[derive(Builder, Default, Clone, Debug)]
-pub struct FuzzyConfiguration {
-    /// How many changes are we going to allow
-    pub distance: u8,
-    /// Should a transposition cost 1 or 2?
-    #[builder(default)]
-    pub transposition_cost_one: bool,
-    #[builder(default)]
-    pub prefix: bool,
-    /// If true, only the term with a levenshtein of exactly `distance` will match.
-    /// If false, terms at a distance `<=` to `distance` will match.
-    #[builder(default)]
-    pub exact_distance: bool,
-}
-
-fn build_dfa(fuzzy_configuration: &FuzzyConfiguration, term_text: &str) -> Result<DFA> {
-    let automaton_builder = LEV_BUILDER
-        .get(&(fuzzy_configuration.distance, fuzzy_configuration.transposition_cost_one))
-        .ok_or_else(|| {
-            InvalidArgument(format!(
-                "Levenshtein distance of {} is not allowed. Choose a value in the {:?} range",
-                fuzzy_configuration.distance, VALID_LEVENSHTEIN_DISTANCE_RANGE
-            ))
-        })?;
-    if fuzzy_configuration.prefix {
-        Ok(automaton_builder.build_prefix_dfa(term_text))
-    } else {
-        Ok(automaton_builder.build_dfa(term_text))
-    }
-}
-
 /// A Fuzzy Query matches all of the documents
 /// containing a specific term that is within
 /// Levenshtein distance
@@ -75,19 +41,32 @@ fn build_dfa(fuzzy_configuration: &FuzzyConfiguration, term_text: &str) -> Resul
 ///     let index = Index::create_in_ram(schema);
 ///     {
 ///         let mut index_writer = index.writer(3_000_000)?;
-///         index_writer.add_document(doc!(title => "The Name of the Wind"));
-///         index_writer.add_document(doc!(title => "The Diary of Muadib"));
-///         index_writer.add_document(doc!(title => "A Dairy Cow"));
-///         index_writer.add_document(doc!(title => "The Diary of a Young Girl"));
+///         index_writer.add_document(doc!(
+///             title => "The Name of the Wind",
+///         ));
+///         index_writer.add_document(doc!(
+///             title => "The Diary of Muadib",
+///         ));
+///         index_writer.add_document(doc!(
+///             title => "A Dairy Cow",
+///         ));
+///         index_writer.add_document(doc!(
+///             title => "The Diary of a Young Girl",
+///         ));
 ///         index_writer.commit().unwrap();
 ///     }
 ///     let reader = index.reader()?;
 ///     let searcher = reader.searcher();
-///     let term = Term::from_field_text(title, "Diary");
-///     let query = FuzzyTermQuery::new(term, 1, true);
-///     let (top_docs, count) = searcher.search(&query, &(TopDocs::with_limit(2), Count)).unwrap();
-///     assert_eq!(count, 2);
-///     assert_eq!(top_docs.len(), 2);
+///
+///     {
+///
+///         let term = Term::from_field_text(title, "Diary");
+///         let query = FuzzyTermQuery::new(term, 1, true);
+///         let (top_docs, count) = searcher.search(&query, &(TopDocs::with_limit(2), Count)).unwrap();
+///         assert_eq!(count, 2);
+///         assert_eq!(top_docs.len(), 2);
+///     }
+///
 ///     Ok(())
 /// }
 /// ```
@@ -95,58 +74,54 @@ fn build_dfa(fuzzy_configuration: &FuzzyConfiguration, term_text: &str) -> Resul
 pub struct FuzzyTermQuery {
     /// What term are we searching
     term: Term,
-    configuration: FuzzyConfiguration
+    /// How many changes are we going to allow
+    distance: u8,
+    /// Should a transposition cost 1 or 2?
+    transposition_cost_one: bool,
+    ///
+    prefix: bool,
 }
 
 impl FuzzyTermQuery {
-    pub fn new_from_configuration(term: Term, configuration: FuzzyConfiguration) -> FuzzyTermQuery {
-        FuzzyTermQuery {
-            term,
-            configuration
-        }
-    }
-
     /// Creates a new Fuzzy Query
     pub fn new(term: Term, distance: u8, transposition_cost_one: bool) -> FuzzyTermQuery {
         FuzzyTermQuery {
             term,
-            configuration: FuzzyConfiguration {
-                distance,
-                transposition_cost_one,
-                prefix: false,
-                exact_distance: false
+            distance,
+            transposition_cost_one,
+            prefix: false,
+        }
+    }
+
+    /// Creates a new Fuzzy Query that treats transpositions as cost one rather than two
+    pub fn new_prefix(term: Term, distance: u8, transposition_cost_one: bool) -> FuzzyTermQuery {
+        FuzzyTermQuery {
+            term,
+            distance,
+            transposition_cost_one,
+            prefix: true,
+        }
+    }
+
+    fn specialized_weight(&self) -> Result<AutomatonWeight<DFA>> {
+        // LEV_BUILDER is a HashMap, whose `get` method returns an Option
+        match LEV_BUILDER.get(&(self.distance, false)) {
+            // Unwrap the option and build the Ok(AutomatonWeight)
+            Some(automaton_builder) => {
+                let automaton = automaton_builder.build_dfa(self.term.text());
+                Ok(AutomatonWeight::new(self.term.field(), automaton))
             }
+            None => Err(InvalidArgument(format!(
+                "Levenshtein distance of {} is not allowed. Choose a value in the {:?} range",
+                self.distance, VALID_LEVENSHTEIN_DISTANCE_RANGE
+            ))),
         }
     }
 }
 
 impl Query for FuzzyTermQuery {
     fn weight(&self, _searcher: &Searcher, _scoring_enabled: bool) -> Result<Box<dyn Weight>> {
-        let dfa = build_dfa(&self.configuration, self.term.text())?;
-        // TODO optimize for distance = 0 and possibly prefix
-        if self.configuration.exact_distance {
-            let target_distance = self.configuration.distance;
-            let wrapped_dfa = WrappedDFA {
-                dfa,
-                condition: move |distance: Distance| distance == Distance::Exact(target_distance),
-            };
-            Ok(Box::new(AutomatonWeight::new(
-                self.term.field(),
-                wrapped_dfa,
-            )))
-        } else {
-            let wrapped_dfa = WrappedDFA {
-                dfa,
-                condition: move |distance: Distance| match distance {
-                    Distance::Exact(_) => true,
-                    Distance::AtLeast(_) => false,
-                },
-            };
-            Ok(Box::new(AutomatonWeight::new(
-                self.term.field(),
-                wrapped_dfa,
-            )))
-        }
+        Ok(Box::new(self.specialized_weight()?))
     }
 }
 
@@ -159,7 +134,6 @@ mod test {
     use crate::tests::assert_nearly_equals;
     use crate::Index;
     use crate::Term;
-    use super::FuzzyConfigurationBuilder;
 
     #[test]
     pub fn test_fuzzy_term() {
@@ -181,6 +155,7 @@ mod test {
         let searcher = reader.searcher();
         {
             let term = Term::from_field_text(country_field, "japon");
+
             let fuzzy_query = FuzzyTermQuery::new(term, 1, true);
             let top_docs = searcher
                 .search(&fuzzy_query, &TopDocs::with_limit(2))
@@ -189,73 +164,5 @@ mod test {
             let (score, _) = top_docs[0];
             assert_nearly_equals(1f32, score);
         }
-        {
-            let term = Term::from_field_text(country_field, "japon");
-            let fuzzy_conf = FuzzyConfigurationBuilder::default()
-                .distance(2)
-                .exact_distance(true)
-                .build()
-                .unwrap();
-            let fuzzy_query = FuzzyTermQuery::new_from_configuration(term, fuzzy_conf);
-            let top_docs = searcher
-                .search(&fuzzy_query, &TopDocs::with_limit(2))
-                .unwrap();
-            assert!(top_docs.is_empty());
-        }
-        {
-            let term = Term::from_field_text(country_field, "japon");
-            let fuzzy_conf = FuzzyConfigurationBuilder::default()
-                .distance(1)
-                .exact_distance(true)
-                .build()
-                .unwrap();
-            let fuzzy_query = FuzzyTermQuery::new_from_configuration(term, fuzzy_conf);
-            let top_docs = searcher
-                .search(&fuzzy_query, &TopDocs::with_limit(2))
-                .unwrap();
-            assert_eq!(top_docs.len(), 1);
-        }
-        {
-            let term = Term::from_field_text(country_field, "jpp");
-            let fuzzy_conf = FuzzyConfigurationBuilder::default()
-                .distance(1)
-                .prefix(true)
-                .build()
-                .unwrap();
-            let fuzzy_query = FuzzyTermQuery::new_from_configuration(term, fuzzy_conf);
-            let top_docs = searcher
-                .search(&fuzzy_query, &TopDocs::with_limit(2))
-                .unwrap();
-            assert_eq!(top_docs.len(), 1);
-        }
-        {
-            let term = Term::from_field_text(country_field, "jpaan");
-            let fuzzy_conf = FuzzyConfigurationBuilder::default()
-                .distance(1)
-                .exact_distance(true)
-                .transposition_cost_one(true)
-                .build()
-                .unwrap();
-            let fuzzy_query = FuzzyTermQuery::new_from_configuration(term, fuzzy_conf);
-            let top_docs = searcher
-                .search(&fuzzy_query, &TopDocs::with_limit(2))
-                .unwrap();
-            assert_eq!(top_docs.len(), 1);
-        }
-        {
-            let term = Term::from_field_text(country_field, "jpaan");
-            let fuzzy_conf = FuzzyConfigurationBuilder::default()
-                .distance(2)
-                .exact_distance(true)
-                .transposition_cost_one(false)
-                .build()
-                .unwrap();
-            let fuzzy_query = FuzzyTermQuery::new_from_configuration(term, fuzzy_conf);
-            let top_docs = searcher
-                .search(&fuzzy_query, &TopDocs::with_limit(2))
-                .unwrap();
-            assert_eq!(top_docs.len(), 1);
-        }
     }
-
 }

src/query/intersection.rs

@@ -45,7 +45,7 @@ pub fn intersect_scorers(mut scorers: Vec<Box<dyn Scorer>>) -> Box<dyn Scorer> {
     })
 }
 
-/// Creates a `DocSet` that iterator through the intersection of two `DocSet`s.
+/// Creates a `DocSet` that iterate through the intersection of two or more `DocSet`s.
 pub struct Intersection<TDocSet: DocSet, TOtherDocSet: DocSet = Box<dyn Scorer>> {
     left: TDocSet,
     right: TDocSet,

src/query/intersection_two.rs

@@ -5,7 +5,7 @@ use Score;
 use SkipResult;
 
 
-/// Creates a `DocSet` that iterator through the intersection of two `DocSet`s.
+/// Creates a `DocSet` that iterate through the intersection of two `DocSet`s.
 pub struct IntersectionTwoTerms<TDocSet> {
     left: TDocSet,
     right: TDocSet

src/query/mod.rs

@@ -40,7 +40,7 @@ pub use self::boolean_query::BooleanQuery;
 pub use self::empty_query::{EmptyQuery, EmptyScorer, EmptyWeight};
 pub use self::exclude::Exclude;
 pub use self::explanation::Explanation;
-pub use self::fuzzy_query::{FuzzyTermQuery, FuzzyConfiguration, FuzzyConfigurationBuilder};
+pub use self::fuzzy_query::FuzzyTermQuery;
 pub use self::intersection::intersect_scorers;
 pub use self::phrase_query::PhraseQuery;
 pub use self::query::Query;

src/query/phrase_query/phrase_query.rs

@@ -40,7 +40,7 @@ impl PhraseQuery {
         PhraseQuery::new_with_offset(terms_with_offset)
     }
 
-    /// Creates a new `PhraseQuery` given a list of terms and there offsets.
+    /// Creates a new `PhraseQuery` given a list of terms and their offsets.
     ///
     /// Can be used to provide custom offset for each term.
     pub fn new_with_offset(mut terms: Vec<(usize, Term)>) -> PhraseQuery {
@@ -73,7 +73,7 @@ impl PhraseQuery {
             .collect::<Vec<Term>>()
     }
 
-    /// Returns the `PhraseWeight` for the given phrase query given a specific `searcher`.  
+    /// Returns the `PhraseWeight` for the given phrase query given a specific `searcher`.
     ///
     /// This function is the same as `.weight(...)` except it returns
     /// a specialized type `PhraseWeight` instead of a Boxed trait.

src/query/query_parser/query_parser.rs

@@ -674,13 +674,19 @@ mod test {
 
         test_parse_query_to_logical_ast_helper(
             "signed:-2324",
-            &format!("{:?}", Term::from_field_i64(Field(2u32), -2324)),
+            &format!(
+                "{:?}",
+                Term::from_field_i64(Field::from_field_id(2u32), -2324)
+            ),
             false,
         );
 
         test_parse_query_to_logical_ast_helper(
             "float:2.5",
-            &format!("{:?}", Term::from_field_f64(Field(10u32), 2.5)),
+            &format!(
+                "{:?}",
+                Term::from_field_f64(Field::from_field_id(10u32), 2.5)
+            ),
             false,
         );
     }

src/query/range_query.rs

@@ -479,5 +479,4 @@ mod tests {
             91
         );
     }
-
 }

src/query/reqopt_scorer.rs

@@ -190,5 +190,4 @@ mod tests {
             skip_docs,
         );
     }
-
 }

src/query/term_query/mod.rs

@@ -118,7 +118,7 @@ mod tests {
     #[test]
     fn test_term_query_debug() {
         let term_query = TermQuery::new(
-            Term::from_field_text(Field(1), "hello"),
+            Term::from_field_text(Field::from_field_id(1), "hello"),
             IndexRecordOption::WithFreqs,
         );
         assert_eq!(

src/query/union.rs

@@ -28,7 +28,7 @@ where
     }
 }
 
-/// Creates a `DocSet` that iterator through the intersection of two `DocSet`s.
+/// Creates a `DocSet` that iterate through the union of two or more `DocSet`s.
 pub struct Union<TScorer, TScoreCombiner = DoNothingCombiner> {
     docsets: Vec<TScorer>,
     bitsets: Box<[TinySet; HORIZON_NUM_TINYBITSETS]>,
@@ -409,20 +409,17 @@ mod tests {
             vec![1, 2, 3, 7, 8, 9, 99, 100, 101, 500, 20000],
         );
     }
-
 }
 
 #[cfg(all(test, feature = "unstable"))]
 mod bench {
 
-    use query::score_combiner::DoNothingCombiner;
-    use query::ConstScorer;
-    use query::Union;
-    use query::VecDocSet;
+    use crate::query::score_combiner::DoNothingCombiner;
+    use crate::query::{ConstScorer, Union, VecDocSet};
+    use crate::tests;
+    use crate::DocId;
+    use crate::DocSet;
     use test::Bencher;
-    use tests;
-    use DocId;
-    use DocSet;
 
     #[bench]
     fn bench_union_3_high(bench: &mut Bencher) {

src/query/vec_docset.rs

@@ -82,5 +82,4 @@ pub mod tests {
         }
         assert_eq!(postings.fill_buffer(&mut buffer[..]), 9);
     }
-
 }

src/schema/document.rs

@@ -178,5 +178,4 @@ mod tests {
         doc.add_text(text_field, "My title");
         assert_eq!(doc.field_values().len(), 1);
     }
-
 }

src/schema/field.rs

@@ -3,14 +3,22 @@ use std::io;
 use std::io::Read;
 use std::io::Write;
 
-/// `Field` is actually a `u8` identifying a `Field`
-/// The schema is in charge of holding mapping between field names
-/// to `Field` objects.
-///
-/// Because the field id is a `u8`, tantivy can only have at most `255` fields.
-/// Value 255 is reserved.
+/// `Field` is represented by an unsigned 32-bit integer type
+/// The schema holds the mapping between field names and `Field` objects.
 #[derive(Copy, Clone, Debug, PartialEq, PartialOrd, Eq, Ord, Hash, Serialize, Deserialize)]
-pub struct Field(pub u32);
+pub struct Field(u32);
+
+impl Field {
+    /// Create a new field object for the given FieldId.
+    pub fn from_field_id(field_id: u32) -> Field {
+        Field(field_id)
+    }
+
+    /// Returns a u32 identifying uniquely a field within a schema.
+    pub fn field_id(&self) -> u32 {
+        self.0
+    }
+}
 
 impl BinarySerializable for Field {
     fn serialize<W: Write>(&self, writer: &mut W) -> io::Result<()> {

src/schema/index_record_option.rs

@@ -29,22 +29,6 @@ pub enum IndexRecordOption {
 }
 
 impl IndexRecordOption {
-    /// Returns true iff the term frequency will be encoded.
-    pub fn is_termfreq_enabled(self) -> bool {
-        match self {
-            IndexRecordOption::WithFreqsAndPositions | IndexRecordOption::WithFreqs => true,
-            _ => false,
-        }
-    }
-
-    /// Returns true iff the term positions within the document are stored as well.
-    pub fn is_position_enabled(self) -> bool {
-        match self {
-            IndexRecordOption::WithFreqsAndPositions => true,
-            _ => false,
-        }
-    }
-
     /// Returns true iff this option includes encoding
     /// term frequencies.
     pub fn has_freq(self) -> bool {

src/schema/mod.rs

@@ -174,5 +174,4 @@ mod tests {
         assert!(!is_valid_field_name("シャボン玉"));
         assert!(is_valid_field_name("my_text_field"));
     }
-
 }

src/schema/schema.rs

@@ -167,7 +167,7 @@ impl SchemaBuilder {
 
     /// Adds a field entry to the schema in build.
     fn add_field(&mut self, field_entry: FieldEntry) -> Field {
-        let field = Field(self.fields.len() as u32);
+        let field = Field::from_field_id(self.fields.len() as u32);
         let field_name = field_entry.name().to_string();
         self.fields.push(field_entry);
         self.fields_map.insert(field_name, field);
@@ -223,7 +223,7 @@ pub struct Schema(Arc<InnerSchema>);
 impl Schema {
     /// Return the `FieldEntry` associated to a `Field`.
     pub fn get_field_entry(&self, field: Field) -> &FieldEntry {
-        &self.0.fields[field.0 as usize]
+        &self.0.fields[field.field_id() as usize]
     }
 
     /// Return the field name for a given `Field`.
@@ -232,8 +232,12 @@ impl Schema {
     }
 
     /// Return the list of all the `Field`s.
-    pub fn fields(&self) -> &[FieldEntry] {
-        &self.0.fields
+    pub fn fields(&self) -> impl Iterator<Item = (Field, &FieldEntry)> {
+        self.0
+            .fields
+            .iter()
+            .enumerate()
+            .map(|(field_id, field_entry)| (Field::from_field_id(field_id as u32), field_entry))
     }
 
     /// Creates a new builder.
@@ -485,13 +489,32 @@ mod tests {
 
         let schema: Schema = serde_json::from_str(expected).unwrap();
 
-        let mut fields = schema.fields().iter();
-
-        assert_eq!("title", fields.next().unwrap().name());
-        assert_eq!("author", fields.next().unwrap().name());
-        assert_eq!("count", fields.next().unwrap().name());
-        assert_eq!("popularity", fields.next().unwrap().name());
-        assert_eq!("score", fields.next().unwrap().name());
+        let mut fields = schema.fields();
+        {
+            let (field, field_entry) = fields.next().unwrap();
+            assert_eq!("title", field_entry.name());
+            assert_eq!(0, field.field_id());
+        }
+        {
+            let (field, field_entry) = fields.next().unwrap();
+            assert_eq!("author", field_entry.name());
+            assert_eq!(1, field.field_id());
+        }
+        {
+            let (field, field_entry) = fields.next().unwrap();
+            assert_eq!("count", field_entry.name());
+            assert_eq!(2, field.field_id());
+        }
+        {
+            let (field, field_entry) = fields.next().unwrap();
+            assert_eq!("popularity", field_entry.name());
+            assert_eq!(3, field.field_id());
+        }
+        {
+            let (field, field_entry) = fields.next().unwrap();
+            assert_eq!("score", field_entry.name());
+            assert_eq!(4, field.field_id());
+        }
         assert!(fields.next().is_none());
     }
 

src/schema/term.rs

@@ -22,10 +22,10 @@ impl Term {
     /// Builds a term given a field, and a i64-value
     ///
     /// Assuming the term has a field id of 1, and a i64 value of 3234,
-    /// the Term will have 8 bytes.
+    /// the Term will have 12 bytes.
     ///
     /// The first four byte are dedicated to storing the field id as a u64.
-    /// The 4 following bytes are encoding the u64 value.
+    /// The 8 following bytes are encoding the u64 value.
     pub fn from_field_i64(field: Field, val: i64) -> Term {
         let val_u64: u64 = common::i64_to_u64(val);
         Term::from_field_u64(field, val_u64)
@@ -33,11 +33,11 @@ impl Term {
 
     /// Builds a term given a field, and a f64-value
     ///
-    /// Assuming the term has a field id of 1, and a u64 value of 3234,
-    /// the Term will have 8 bytes. <= this is wrong
+    /// Assuming the term has a field id of 1, and a f64 value of 1.5,
+    /// the Term will have 12 bytes.
     ///
     /// The first four byte are dedicated to storing the field id as a u64.
-    /// The 4 following bytes are encoding the u64 value.
+    /// The 8 following bytes are encoding the f64 as a u64 value.
     pub fn from_field_f64(field: Field, val: f64) -> Term {
         let val_u64: u64 = common::f64_to_u64(val);
         Term::from_field_u64(field, val_u64)
@@ -46,10 +46,10 @@ impl Term {
     /// Builds a term given a field, and a DateTime value
     ///
     /// Assuming the term has a field id of 1, and a timestamp i64 value of 3234,
-    /// the Term will have 8 bytes.
+    /// the Term will have 12 bytes.
     ///
     /// The first four byte are dedicated to storing the field id as a u64.
-    /// The 4 following bytes are encoding the DateTime as i64 timestamp value.
+    /// The 8 following bytes are encoding the DateTime as i64 timestamp value.
     pub fn from_field_date(field: Field, val: &DateTime) -> Term {
         let val_timestamp = val.timestamp();
         Term::from_field_i64(field, val_timestamp)
@@ -82,10 +82,10 @@ impl Term {
     /// Builds a term given a field, and a u64-value
     ///
     /// Assuming the term has a field id of 1, and a u64 value of 3234,
-    /// the Term will have 8 bytes.
+    /// the Term will have 12 bytes.
     ///
     /// The first four byte are dedicated to storing the field id as a u64.
-    /// The 4 following bytes are encoding the u64 value.
+    /// The 8 following bytes are encoding the u64 value.
     pub fn from_field_u64(field: Field, val: u64) -> Term {
         let mut term = Term(vec![0u8; INT_TERM_LEN]);
         term.set_field(field);
@@ -105,7 +105,7 @@ impl Term {
         if self.0.len() < 4 {
             self.0.resize(4, 0u8);
         }
-        BigEndian::write_u32(&mut self.0[0..4], field.0);
+        BigEndian::write_u32(&mut self.0[0..4], field.field_id());
     }
 
     /// Sets a u64 value in the term.
@@ -157,7 +157,7 @@ where
 
     /// Returns the field.
     pub fn field(&self) -> Field {
-        Field(BigEndian::read_u32(&self.0.as_ref()[..4]))
+        Field::from_field_id(BigEndian::read_u32(&self.0.as_ref()[..4]))
     }
 
     /// Returns the `u64` value stored in a term.
@@ -182,7 +182,7 @@ where
     ///
     /// # Panics
     /// ... or returns an invalid value
-    /// if the term is not a `i64` field.
+    /// if the term is not a `f64` field.
     pub fn get_f64(&self) -> f64 {
         common::u64_to_f64(BigEndian::read_u64(&self.0.as_ref()[4..]))
     }
@@ -227,7 +227,7 @@ impl fmt::Debug for Term {
         write!(
             f,
             "Term(field={},bytes={:?})",
-            self.field().0,
+            self.field().field_id(),
             self.value_bytes()
         )
     }

src/store/mod.rs

@@ -120,17 +120,16 @@ pub mod tests {
             );
         }
     }
-
 }
 
 #[cfg(all(test, feature = "unstable"))]
 mod bench {
 
     use super::tests::write_lorem_ipsum_store;
-    use directory::Directory;
-    use directory::RAMDirectory;
+    use crate::directory::Directory;
+    use crate::directory::RAMDirectory;
+    use crate::store::StoreReader;
     use std::path::Path;
-    use store::StoreReader;
     use test::Bencher;
 
     #[bench]

src/store/skiplist/mod.rs

@@ -165,5 +165,4 @@ mod tests {
         assert_eq!(output.len(), 65);
         assert_eq!(output[0], 128u8 + 3u8);
     }
-
 }

src/store/writer.rs

@@ -3,6 +3,7 @@ use super::skiplist::SkipListBuilder;
 use super::StoreReader;
 use crate::common::CountingWriter;
 use crate::common::{BinarySerializable, VInt};
+use crate::directory::TerminatingWrite;
 use crate::directory::WritePtr;
 use crate::schema::Document;
 use crate::DocId;
@@ -109,6 +110,6 @@ impl StoreWriter {
         self.offset_index_writer.write(&mut self.writer)?;
         header_offset.serialize(&mut self.writer)?;
         self.doc.serialize(&mut self.writer)?;
-        self.writer.flush()
+        self.writer.terminate()
     }
 }

src/termdict/mod.rs

@@ -31,43 +31,14 @@ mod termdict;
 pub use self::merger::TermMerger;
 pub use self::streamer::{TermStreamer, TermStreamerBuilder};
 pub use self::termdict::{TermDictionary, TermDictionaryBuilder};
-use levenshtein_automata::{Distance, DFA, SINK_STATE};
-use tantivy_fst::Automaton;
-
-pub(crate) struct WrappedDFA<Cond> {
-    pub dfa: DFA,
-    pub condition: Cond,
-}
-
-impl<Cond: Fn(Distance) -> bool> Automaton for WrappedDFA<Cond> {
-    type State = u32;
-
-    fn start(&self) -> Self::State {
-        self.dfa.initial_state()
-    }
-
-    fn is_match(&self, state: &Self::State) -> bool {
-        let distance = self.dfa.distance(*state);
-        (self.condition)(distance)
-    }
-
-    fn can_match(&self, state: &Self::State) -> bool {
-        *state != SINK_STATE
-    }
-
-    fn accept(&self, state: &Self::State, byte: u8) -> Self::State {
-        self.dfa.transition(*state, byte)
-    }
-}
 
 #[cfg(test)]
 mod tests {
-    use super::{TermDictionary, TermDictionaryBuilder, TermStreamer, WrappedDFA};
+    use super::{TermDictionary, TermDictionaryBuilder, TermStreamer};
     use crate::core::Index;
     use crate::directory::{Directory, RAMDirectory, ReadOnlySource};
     use crate::postings::TermInfo;
     use crate::schema::{Document, FieldType, Schema, TEXT};
-    use levenshtein_automata::Distance;
     use std::path::PathBuf;
     use std::str;
 
@@ -452,14 +423,9 @@ mod tests {
 
         // We can now build an entire dfa.
         let lev_automaton_builder = LevenshteinAutomatonBuilder::new(2, true);
-        let wrapped_dfa = WrappedDFA {
-            dfa: lev_automaton_builder.build_dfa("Spaen"),
-            condition: |distance| match distance {
-                Distance::Exact(_) => true,
-                Distance::AtLeast(_) => false,
-            },
-        };
-        let mut range = term_dict.search(wrapped_dfa).into_stream();
+        let automaton = lev_automaton_builder.build_dfa("Spaen");
+
+        let mut range = term_dict.search(automaton).into_stream();
 
         // get the first finding
         assert!(range.advance());

src/termdict/term_info_store.rs

@@ -328,5 +328,4 @@ mod tests {
             assert_eq!(term_info_store.get(i as u64), term_infos[i]);
         }
     }
-
 }

src/tokenizer/lower_caser.rs

@@ -98,10 +98,6 @@ mod tests {
     #[test]
     fn test_lowercaser() {
         assert_eq!(lowercase_helper("Tree"), vec!["tree".to_string()]);
-        assert_eq!(
-            lowercase_helper("Русский"),
-            vec!["русский".to_string()]
-        );
+        assert_eq!(lowercase_helper("Русский"), vec!["русский".to_string()]);
     }
-
 }

src/tokenizer/mod.rs

@@ -281,5 +281,4 @@ pub mod tests {
             assert!(tokens.is_empty());
         }
     }
-
 }

src/tokenizer/ngram_tokenizer.rs

@@ -460,5 +460,4 @@ mod tests {
         assert_eq!(it.next(), Some((8, 9)));
         assert_eq!(it.next(), None);
     }
-
 }

src/tokenizer/token_stream_chain.rs

@@ -97,5 +97,4 @@ mod tests {
 
         assert!(!token_chain.advance());
     }
-
 }

tests/failpoints/mod.rs

@@ -1,7 +1,6 @@
 use fail;
-use std::io::Write;
 use std::path::Path;
-use tantivy::directory::{Directory, ManagedDirectory, RAMDirectory};
+use tantivy::directory::{Directory, ManagedDirectory, RAMDirectory, TerminatingWrite};
 use tantivy::doc;
 use tantivy::schema::{Schema, TEXT};
 use tantivy::{Index, Term};
@@ -17,7 +16,7 @@ fn test_failpoints_managed_directory_gc_if_delete_fails() {
     managed_directory
         .open_write(test_path)
         .unwrap()
-        .flush()
+        .terminate()
         .unwrap();
     assert!(managed_directory.exists(test_path));
     // triggering gc and setting the delete operation to fail.