Update field.rs

Removing the contract about the order of the field, and the
field id allocation.

.travis.yml

@@ -41,8 +41,8 @@ matrix:
     - env: TARGET=x86_64-unknown-linux-gnu CODECOV=1 #UPLOAD_DOCS=1
     # - env: TARGET=x86_64-unknown-linux-musl CODECOV=1
     # OSX
-    - env: TARGET=x86_64-apple-darwin
-      os: osx
+    #- env: TARGET=x86_64-apple-darwin
+    #  os: osx
 
 before_install:
   - set -e

CHANGELOG.md

@@ -7,7 +7,8 @@ Tantivy 0.11.0
     - Better handling of whitespaces.
 - Closes #498 - add support for Elastic-style unbounded range queries for alphanumeric types eg. "title:>hello", "weight:>=70.5", "height:<200" (@petr-tik)
 - API change around `Box<BoxableTokenizer>`. See detail in #629
-- Avoid rebuilding Regex automaton whenever a regex query is reused. #630 (@brainlock)
+- Avoid rebuilding Regex automaton whenever a regex query is reused. #639 (@brainlock)
+- Add footer with some metadata to index files. #605 (@fdb-hiroshima)
  
 ## How to update?
 
@@ -15,6 +16,12 @@ Tantivy 0.11.0
 - Regex are now compiled when the `RegexQuery` instance is built. As a result, it can now return
 an error and handling the `Result` is required.
 
+
+Tantivy 0.10.2
+=====================
+
+- Closes #656. Solving memory leak.
+
 Tantivy 0.10.1
 =====================
 

Cargo.toml

@@ -15,6 +15,7 @@ edition = "2018"
 [dependencies]
 base64 = "0.10.0"
 byteorder = "1.0"
+crc32fast = "1.2.0"
 once_cell = "1.0"
 regex ={version = "1.3.0", default-features = false, features = ["std"]}
 tantivy-fst = "0.1"

README.md

@@ -21,9 +21,9 @@
 [![Become a patron](https://c5.patreon.com/external/logo/become_a_patron_button.png)](https://www.patreon.com/fulmicoton)
 
 
-**Tantivy** is a **full text search engine library** written in rust.
+**Tantivy** is a **full text search engine library** written in Rust.
 
-It is closer to [Apache Lucene](https://lucene.apache.org/) than to [Elasticsearch](https://www.elastic.co/products/elasticsearch) and [Apache Solr](https://lucene.apache.org/solr/) in the sense it is not
+It is closer to [Apache Lucene](https://lucene.apache.org/) than to [Elasticsearch](https://www.elastic.co/products/elasticsearch) or [Apache Solr](https://lucene.apache.org/solr/) in the sense it is not
 an off-the-shelf search engine server, but rather a crate that can be used
 to build such a search engine.
 
@@ -31,7 +31,7 @@ Tantivy is, in fact, strongly inspired by Lucene's design.
 
 # Benchmark
 
-Tantivy is typically faster than Lucene, but the results will depend on 
+Tantivy is typically faster than Lucene, but the results depend on 
 the nature of the queries in your workload.
 
 The following [benchmark](https://tantivy-search.github.io/bench/) break downs 
@@ -40,19 +40,19 @@ performance for different type of queries / collection.
 # Features
 
 - Full-text search
-- Configurable tokenizer. (stemming available for 17 latin languages. Third party support for Chinese ([tantivy-jieba](https://crates.io/crates/tantivy-jieba) and [cang-jie](https://crates.io/crates/cang-jie)) and [Japanese](https://crates.io/crates/tantivy-tokenizer-tiny-segmenter)
+- Configurable tokenizer (stemming available for 17 Latin languages with third party support for Chinese ([tantivy-jieba](https://crates.io/crates/tantivy-jieba) and [cang-jie](https://crates.io/crates/cang-jie)) and [Japanese](https://crates.io/crates/tantivy-tokenizer-tiny-segmenter))
 - Fast (check out the :racehorse: :sparkles: [benchmark](https://tantivy-search.github.io/bench/) :sparkles: :racehorse:)
 - Tiny startup time (<10ms), perfect for command line tools
-- BM25 scoring (the same as lucene)
-- Natural query language `(michael AND jackson) OR "king of pop"`
-- Phrase queries search (`"michael jackson"`)
+- BM25 scoring (the same as Lucene)
+- Natural query language (e.g. `(michael AND jackson) OR "king of pop"`)
+- Phrase queries search (e.g. `"michael jackson"`)
 - Incremental indexing
 - Multithreaded indexing (indexing English Wikipedia takes < 3 minutes on my desktop)
 - Mmap directory
-- SIMD integer compression when the platform/CPU includes the SSE2 instruction set.
-- Single valued and multivalued u64, i64 and f64 fast fields (equivalent of doc values in Lucene)
+- SIMD integer compression when the platform/CPU includes the SSE2 instruction set
+- Single valued and multivalued u64, i64, and f64 fast fields (equivalent of doc values in Lucene)
 - `&[u8]` fast fields
-- Text, i64, u64, f64, dates and hierarchical facet fields
+- Text, i64, u64, f64, dates, and hierarchical facet fields
 - LZ4 compressed document store
 - Range queries
 - Faceted search
@@ -61,43 +61,42 @@ performance for different type of queries / collection.
 
 # Non-features
 
-- Distributed search is out of the scope of tantivy. That being said, tantivy is meant as a
+- Distributed search is out of the scope of Tantivy. That being said, Tantivy is a
 library upon which one could build a distributed search. Serializable/mergeable collector state for instance, 
-are within the scope of tantivy.
+are within the scope of Tantivy.
 
 # Supported OS and compiler
 
-Tantivy works on stable rust (>= 1.27) and supports Linux, MacOS and Windows.
+Tantivy works on stable Rust (>= 1.27) and supports Linux, MacOS, and Windows.
 
 # Getting started
 
-- [tantivy's simple search example](https://tantivy-search.github.io/examples/basic_search.html)
-- [tantivy-cli and its tutorial](https://github.com/tantivy-search/tantivy-cli).
-`tantivy-cli` is an actual command line interface that makes it easy for you to create a search engine,
-index documents and search via the CLI or a small server with a REST API.
-It will walk you through getting a wikipedia search engine up and running in a few minutes.
-- [reference doc for the last released version](https://docs.rs/tantivy/)
+- [Tantivy's simple search example](https://tantivy-search.github.io/examples/basic_search.html)
+- [tantivy-cli and its tutorial](https://github.com/tantivy-search/tantivy-cli) - `tantivy-cli` is an actual command line interface that makes it easy for you to create a search engine,
+index documents, and search via the CLI or a small server with a REST API.
+It walks you through getting a wikipedia search engine up and running in a few minutes.
+- [Reference doc for the last released version](https://docs.rs/tantivy/)
 
 # How can I support this project?
 
 There are many ways to support this project. 
 
-- Use tantivy and tell us about your experience on [gitter](https://gitter.im/tantivy-search/tantivy) or by email (paul.masurel@gmail.com)
+- Use Tantivy and tell us about your experience on [Gitter](https://gitter.im/tantivy-search/tantivy) or by email (paul.masurel@gmail.com)
 - Report bugs
 - Write a blog post
 - Help with documentation by asking questions or submitting PRs
-- Contribute code (you can join [our gitter](https://gitter.im/tantivy-search/tantivy) )
-- Talk about tantivy around you
+- Contribute code (you can join [our Gitter](https://gitter.im/tantivy-search/tantivy))
+- Talk about Tantivy around you
 - Drop a word on on [![Say Thanks!](https://img.shields.io/badge/Say%20Thanks-!-1EAEDB.svg)](https://saythanks.io/to/fulmicoton) or even [![Become a patron](https://c5.patreon.com/external/logo/become_a_patron_button.png)](https://www.patreon.com/fulmicoton)
 
 # Contributing code
 
-We use the GitHub Pull Request workflow - reference a GitHub ticket and/or include a comprehensive commit message when opening a PR.
+We use the GitHub Pull Request workflow: reference a GitHub ticket and/or include a comprehensive commit message when opening a PR.
 
 ## Clone and build locally
 
-Tantivy compiles on stable rust but requires `Rust >= 1.27`.
-To check out and run tests, you can simply run :
+Tantivy compiles on stable Rust but requires `Rust >= 1.27`.
+To check out and run tests, you can simply run:
 
 ```bash
     git clone https://github.com/tantivy-search/tantivy.git
@@ -108,7 +107,7 @@ To check out and run tests, you can simply run :
 ## Run tests
 
 Some tests will not run with just `cargo test` because of `fail-rs`.
-To run the tests exhaustively, run `./run-tests.sh`
+To run the tests exhaustively, run `./run-tests.sh`.
 
 ## Debug
 
@@ -116,13 +115,13 @@ You might find it useful to step through the programme with a debugger.
 
 ### A failing test
 
-Make sure you haven't run `cargo clean` after the most recent `cargo test` or `cargo build` to guarantee that `target/` dir exists. Use this bash script to find the most name of the most recent debug build of tantivy and run it under rust-gdb.
+Make sure you haven't run `cargo clean` after the most recent `cargo test` or `cargo build` to guarantee that the `target/` directory exists. Use this bash script to find the name of the most recent debug build of Tantivy and run it under `rust-gdb`:
 
 ```bash
 find target/debug/ -maxdepth 1 -executable -type f -name "tantivy*" -printf '%TY-%Tm-%Td %TT %p\n' | sort -r | cut -d " " -f 3 | xargs -I RECENT_DBG_TANTIVY rust-gdb RECENT_DBG_TANTIVY
 ```
 
-Now that you are in rust-gdb, you can set breakpoints on lines and methods that match your source-code and run the debug executable with flags that you normally pass to `cargo test` to like this
+Now that you are in `rust-gdb`, you can set breakpoints on lines and methods that match your source code and run the debug executable with flags that you normally pass to `cargo test` like this:
 
 ```bash
 $gdb run --test-threads 1 --test $NAME_OF_TEST
@@ -130,7 +129,7 @@ $gdb run --test-threads 1 --test $NAME_OF_TEST
 
 ### An example
 
-By default, rustc compiles everything in the `examples/` dir in debug mode. This makes it easy for you to make examples to reproduce bugs.
+By default, `rustc` compiles everything in the `examples/` directory in debug mode. This makes it easy for you to make examples to reproduce bugs:
 
 ```bash
 rust-gdb target/debug/examples/$EXAMPLE_NAME

query-grammar/src/occur.rs

@@ -2,7 +2,7 @@ use std::fmt;
 use std::fmt::Write;
 
 /// Defines whether a term in a query must be present,
-/// should be present or must not be present.
+/// should be present or must be not present.
 #[derive(Debug, Clone, Hash, Copy, Eq, PartialEq)]
 pub enum Occur {
     /// For a given document to be considered for scoring,

src/collector/count_collector.rs

@@ -123,5 +123,4 @@ mod tests {
             assert_eq!(count_collector.harvest(), 2);
         }
     }
-
 }

src/collector/facet_collector.rs

@@ -515,7 +515,7 @@ mod tests {
     #[should_panic(expected = "Tried to add a facet which is a descendant of \
                                an already added facet.")]
     fn test_misused_facet_collector() {
-        let mut facet_collector = FacetCollector::for_field(Field(0));
+        let mut facet_collector = FacetCollector::for_field(Field::from_field_id(0));
         facet_collector.add_facet(Facet::from("/country"));
         facet_collector.add_facet(Facet::from("/country/europe"));
     }
@@ -546,7 +546,7 @@ mod tests {
 
     #[test]
     fn test_non_used_facet_collector() {
-        let mut facet_collector = FacetCollector::for_field(Field(0));
+        let mut facet_collector = FacetCollector::for_field(Field::from_field_id(0));
         facet_collector.add_facet(Facet::from("/country"));
         facet_collector.add_facet(Facet::from("/countryeurope"));
     }
@@ -599,19 +599,18 @@ mod tests {
             );
         }
     }
-
 }
 
 #[cfg(all(test, feature = "unstable"))]
 mod bench {
 
-    use collector::FacetCollector;
-    use query::AllQuery;
-    use rand::{thread_rng, Rng};
-    use schema::Facet;
-    use schema::Schema;
+    use crate::collector::FacetCollector;
+    use crate::query::AllQuery;
+    use crate::schema::{Facet, Schema};
+    use crate::Index;
+    use rand::seq::SliceRandom;
+    use rand::thread_rng;
     use test::Bencher;
-    use Index;
 
     #[bench]
     fn bench_facet_collector(b: &mut Bencher) {
@@ -628,7 +627,7 @@ mod bench {
             }
         }
         // 40425 docs
-        thread_rng().shuffle(&mut docs[..]);
+        docs[..].shuffle(&mut thread_rng());
 
         let mut index_writer = index.writer_with_num_threads(1, 3_000_000).unwrap();
         for doc in docs {
@@ -637,7 +636,7 @@ mod bench {
         index_writer.commit().unwrap();
         let reader = index.reader().unwrap();
         b.iter(|| {
-            let searcher = index.searcher();
+            let searcher = reader.searcher();
             let facet_collector = FacetCollector::for_field(facet_field);
             searcher.search(&AllQuery, &facet_collector).unwrap();
         });

src/collector/top_score_collector.rs

@@ -551,7 +551,7 @@ mod tests {
             ));
         });
         let searcher = index.reader().unwrap().searcher();
-        let top_collector = TopDocs::with_limit(4).order_by_u64_field(Field(2));
+        let top_collector = TopDocs::with_limit(4).order_by_u64_field(Field::from_field_id(2));
         let segment_reader = searcher.segment_reader(0u32);
         top_collector
             .for_segment(0, segment_reader)
@@ -592,5 +592,4 @@ mod tests {
         let query = query_parser.parse_query(query).unwrap();
         (index, query)
     }
-
 }

src/common/composite_file.rs

@@ -2,7 +2,7 @@ use crate::common::BinarySerializable;
 use crate::common::CountingWriter;
 use crate::common::VInt;
 use crate::directory::ReadOnlySource;
-use crate::directory::WritePtr;
+use crate::directory::{TerminatingWrite, WritePtr};
 use crate::schema::Field;
 use crate::space_usage::FieldUsage;
 use crate::space_usage::PerFieldSpaceUsage;
@@ -42,7 +42,7 @@ pub struct CompositeWrite<W = WritePtr> {
     offsets: HashMap<FileAddr, u64>,
 }
 
-impl<W: Write> CompositeWrite<W> {
+impl<W: TerminatingWrite + Write> CompositeWrite<W> {
     /// Crate a new API writer that writes a composite file
     /// in a given write.
     pub fn wrap(w: W) -> CompositeWrite<W> {
@@ -91,8 +91,7 @@ impl<W: Write> CompositeWrite<W> {
 
         let footer_len = (self.write.written_bytes() - footer_offset) as u32;
         footer_len.serialize(&mut self.write)?;
-        self.write.flush()?;
-        Ok(())
+        self.write.terminate()
     }
 }
 
@@ -200,13 +199,13 @@ mod test {
             let w = directory.open_write(path).unwrap();
             let mut composite_write = CompositeWrite::wrap(w);
             {
-                let mut write_0 = composite_write.for_field(Field(0u32));
+                let mut write_0 = composite_write.for_field(Field::from_field_id(0u32));
                 VInt(32431123u64).serialize(&mut write_0).unwrap();
                 write_0.flush().unwrap();
             }
 
             {
-                let mut write_4 = composite_write.for_field(Field(4u32));
+                let mut write_4 = composite_write.for_field(Field::from_field_id(4u32));
                 VInt(2).serialize(&mut write_4).unwrap();
                 write_4.flush().unwrap();
             }
@@ -216,14 +215,18 @@ mod test {
             let r = directory.open_read(path).unwrap();
             let composite_file = CompositeFile::open(&r).unwrap();
             {
-                let file0 = composite_file.open_read(Field(0u32)).unwrap();
+                let file0 = composite_file
+                    .open_read(Field::from_field_id(0u32))
+                    .unwrap();
                 let mut file0_buf = file0.as_slice();
                 let payload_0 = VInt::deserialize(&mut file0_buf).unwrap().0;
                 assert_eq!(file0_buf.len(), 0);
                 assert_eq!(payload_0, 32431123u64);
             }
             {
-                let file4 = composite_file.open_read(Field(4u32)).unwrap();
+                let file4 = composite_file
+                    .open_read(Field::from_field_id(4u32))
+                    .unwrap();
                 let mut file4_buf = file4.as_slice();
                 let payload_4 = VInt::deserialize(&mut file4_buf).unwrap().0;
                 assert_eq!(file4_buf.len(), 0);
@@ -231,5 +234,4 @@ mod test {
             }
         }
     }
-
 }

src/common/counting_writer.rs

@@ -1,3 +1,5 @@
+use crate::directory::AntiCallToken;
+use crate::directory::TerminatingWrite;
 use std::io;
 use std::io::Write;
 
@@ -42,6 +44,13 @@ impl<W: Write> Write for CountingWriter<W> {
     }
 }
 
+impl<W: TerminatingWrite> TerminatingWrite for CountingWriter<W> {
+    fn terminate_ref(&mut self, token: AntiCallToken) -> io::Result<()> {
+        self.flush()?;
+        self.underlying.terminate_ref(token)
+    }
+}
+
 #[cfg(test)]
 mod test {
 

src/common/serialize.rs

@@ -199,10 +199,7 @@ pub mod test {
     fn test_serialize_string() {
         assert_eq!(serialize_test(String::from("")), 1);
         assert_eq!(serialize_test(String::from("ぽよぽよ")), 1 + 3 * 4);
-        assert_eq!(
-            serialize_test(String::from("富士さん見える。")),
-            1 + 3 * 8
-        );
+        assert_eq!(serialize_test(String::from("富士さん見える。")), 1 + 3 * 8);
     }
 
     #[test]

src/core/index.rs

@@ -26,9 +26,10 @@ use crate::IndexWriter;
 use crate::Result;
 use num_cpus;
 use std::borrow::BorrowMut;
+use std::collections::HashSet;
 use std::fmt;
 #[cfg(feature = "mmap")]
-use std::path::Path;
+use std::path::{Path, PathBuf};
 use std::sync::Arc;
 
 fn load_metas(directory: &dyn Directory, inventory: &SegmentMetaInventory) -> Result<IndexMeta> {
@@ -368,6 +369,11 @@ impl Index {
             .map(SegmentMeta::id)
             .collect())
     }
+
+    /// Returns the set of corrupted files
+    pub fn validate_checksum(&self) -> Result<HashSet<PathBuf>> {
+        self.directory.list_damaged().map_err(Into::into)
+    }
 }
 
 impl fmt::Debug for Index {
@@ -595,5 +601,4 @@ mod tests {
         assert_eq!(searcher.num_docs(), 8_000);
         assert!(mem_right_after_merge_finished < mem_right_after_commit);
     }
-
 }

src/directory/directory.rs

@@ -118,6 +118,8 @@ pub trait Directory: DirectoryClone + fmt::Debug + Send + Sync + 'static {
     ///
     /// Specifically, subsequent writes or flushes should
     /// have no effect on the returned `ReadOnlySource` object.
+    ///
+    /// You should only use this to read files create with [`open_write`]
     fn open_read(&self, path: &Path) -> result::Result<ReadOnlySource, OpenReadError>;
 
     /// Removes a file
@@ -157,6 +159,8 @@ pub trait Directory: DirectoryClone + fmt::Debug + Send + Sync + 'static {
     /// atomic_write.
     ///
     /// This should only be used for small files.
+    ///
+    /// You should only use this to read files create with [`atomic_write`]
     fn atomic_read(&self, path: &Path) -> Result<Vec<u8>, OpenReadError>;
 
     /// Atomically replace the content of a file with data.

src/directory/footer.rs

@@ -0,0 +1,213 @@
+use crate::directory::read_only_source::ReadOnlySource;
+use crate::directory::{AntiCallToken, TerminatingWrite};
+use byteorder::{ByteOrder, LittleEndian};
+use crc32fast::Hasher;
+use std::io;
+use std::io::Write;
+
+const COMMON_FOOTER_SIZE: usize = 4 * 5;
+
+#[derive(Debug, Clone, PartialEq)]
+pub struct Footer {
+    pub tantivy_version: (u32, u32, u32),
+    pub meta: String,
+    pub versioned_footer: VersionedFooter,
+}
+
+impl Footer {
+    pub fn new(versioned_footer: VersionedFooter) -> Self {
+        let tantivy_version = (
+            env!("CARGO_PKG_VERSION_MAJOR").parse().unwrap(),
+            env!("CARGO_PKG_VERSION_MINOR").parse().unwrap(),
+            env!("CARGO_PKG_VERSION_PATCH").parse().unwrap(),
+        );
+        Footer {
+            tantivy_version,
+            meta: format!(
+                "tantivy {}.{}.{}, index v{}",
+                tantivy_version.0,
+                tantivy_version.1,
+                tantivy_version.2,
+                versioned_footer.version()
+            ),
+            versioned_footer,
+        }
+    }
+
+    pub fn to_bytes(&self) -> Vec<u8> {
+        let mut res = self.versioned_footer.to_bytes();
+        res.extend_from_slice(self.meta.as_bytes());
+        let len = res.len();
+        res.resize(len + COMMON_FOOTER_SIZE, 0);
+        let mut common_footer = &mut res[len..];
+        LittleEndian::write_u32(&mut common_footer, self.meta.len() as u32);
+        LittleEndian::write_u32(&mut common_footer[4..], self.tantivy_version.0);
+        LittleEndian::write_u32(&mut common_footer[8..], self.tantivy_version.1);
+        LittleEndian::write_u32(&mut common_footer[12..], self.tantivy_version.2);
+        LittleEndian::write_u32(&mut common_footer[16..], (len + COMMON_FOOTER_SIZE) as u32);
+        res
+    }
+
+    pub fn from_bytes(data: &[u8]) -> Result<Self, io::Error> {
+        let len = data.len();
+        if len < COMMON_FOOTER_SIZE + 4 {
+            // 4 bytes for index version, stored in versioned footer
+            return Err(io::Error::new(
+                    io::ErrorKind::UnexpectedEof,
+                    format!("File corrupted. The footer len must be over 24, while the entire file len is {}", len)
+                    )
+            );
+        }
+
+        let size = LittleEndian::read_u32(&data[len - 4..]) as usize;
+        if len < size as usize {
+            return Err(io::Error::new(
+                io::ErrorKind::UnexpectedEof,
+                format!(
+                    "File corrupted. The footer len is {}, while the entire file len is {}",
+                    size, len
+                ),
+            ));
+        }
+        let footer = &data[len - size as usize..];
+        let meta_len = LittleEndian::read_u32(&footer[size - 20..]) as usize;
+        let tantivy_major = LittleEndian::read_u32(&footer[size - 16..]);
+        let tantivy_minor = LittleEndian::read_u32(&footer[size - 12..]);
+        let tantivy_patch = LittleEndian::read_u32(&footer[size - 8..]);
+        Ok(Footer {
+            tantivy_version: (tantivy_major, tantivy_minor, tantivy_patch),
+            meta: String::from_utf8_lossy(&footer[size - meta_len - 20..size - 20]).into_owned(),
+            versioned_footer: VersionedFooter::from_bytes(&footer[..size - meta_len - 20])?,
+        })
+    }
+
+    pub fn extract_footer(source: ReadOnlySource) -> Result<(Footer, ReadOnlySource), io::Error> {
+        let footer = Footer::from_bytes(source.as_slice())?;
+        let reader = source.slice_to(source.as_slice().len() - footer.size());
+        Ok((footer, reader))
+    }
+
+    pub fn size(&self) -> usize {
+        self.versioned_footer.size() as usize + self.meta.len() + 20
+    }
+}
+
+#[derive(Debug, Clone, PartialEq)]
+pub enum VersionedFooter {
+    UnknownVersion { version: u32, size: u32 },
+    V0(u32), // crc
+}
+
+impl VersionedFooter {
+    pub fn to_bytes(&self) -> Vec<u8> {
+        match self {
+            VersionedFooter::V0(crc) => {
+                let mut res = vec![0; 8];
+                LittleEndian::write_u32(&mut res, 0);
+                LittleEndian::write_u32(&mut res[4..], *crc);
+                res
+            }
+            VersionedFooter::UnknownVersion { .. } => {
+                panic!("Unsupported index should never get serialized");
+            }
+        }
+    }
+
+    pub fn from_bytes(footer: &[u8]) -> Result<Self, io::Error> {
+        assert!(footer.len() >= 4);
+        let version = LittleEndian::read_u32(footer);
+        match version {
+            0 => {
+                if footer.len() == 8 {
+                    Ok(VersionedFooter::V0(LittleEndian::read_u32(&footer[4..])))
+                } else {
+                    Err(io::Error::new(
+                        io::ErrorKind::UnexpectedEof,
+                        format!(
+                            "File corrupted. The versioned footer len is {}, while it should be 8",
+                            footer.len()
+                        ),
+                    ))
+                }
+            }
+            version => Ok(VersionedFooter::UnknownVersion {
+                version,
+                size: footer.len() as u32,
+            }),
+        }
+    }
+
+    pub fn size(&self) -> u32 {
+        match self {
+            VersionedFooter::V0(_) => 8,
+            VersionedFooter::UnknownVersion { size, .. } => *size,
+        }
+    }
+
+    pub fn version(&self) -> u32 {
+        match self {
+            VersionedFooter::V0(_) => 0,
+            VersionedFooter::UnknownVersion { version, .. } => *version,
+        }
+    }
+
+    pub fn crc(&self) -> Option<u32> {
+        match self {
+            VersionedFooter::V0(crc) => Some(*crc),
+            VersionedFooter::UnknownVersion { .. } => None,
+        }
+    }
+}
+
+pub(crate) struct FooterProxy<W: TerminatingWrite> {
+    /// always Some except after terminate call
+    hasher: Option<Hasher>,
+    /// always Some except after terminate call
+    writer: Option<W>,
+}
+
+impl<W: TerminatingWrite> FooterProxy<W> {
+    pub fn new(writer: W) -> Self {
+        FooterProxy {
+            hasher: Some(Hasher::new()),
+            writer: Some(writer),
+        }
+    }
+}
+
+impl<W: TerminatingWrite> Write for FooterProxy<W> {
+    fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
+        let count = self.writer.as_mut().unwrap().write(buf)?;
+        self.hasher.as_mut().unwrap().update(&buf[..count]);
+        Ok(count)
+    }
+
+    fn flush(&mut self) -> io::Result<()> {
+        self.writer.as_mut().unwrap().flush()
+    }
+}
+
+impl<W: TerminatingWrite> TerminatingWrite for FooterProxy<W> {
+    fn terminate_ref(&mut self, _: AntiCallToken) -> io::Result<()> {
+        let crc = self.hasher.take().unwrap().finalize();
+
+        let footer = Footer::new(VersionedFooter::V0(crc)).to_bytes();
+        let mut writer = self.writer.take().unwrap();
+        writer.write_all(&footer)?;
+        writer.terminate()
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use crate::directory::footer::{Footer, VersionedFooter};
+
+    #[test]
+    fn test_serialize_deserialize_footer() {
+        let crc = 123456;
+        let footer = Footer::new(VersionedFooter::V0(crc));
+        let footer_bytes = footer.to_bytes();
+
+        assert_eq!(Footer::from_bytes(&footer_bytes).unwrap(), footer);
+    }
+}

src/directory/managed_directory.rs

@@ -1,5 +1,6 @@
 use crate::core::MANAGED_FILEPATH;
 use crate::directory::error::{DeleteError, IOError, LockError, OpenReadError, OpenWriteError};
+use crate::directory::footer::{Footer, FooterProxy};
 use crate::directory::DirectoryLock;
 use crate::directory::Lock;
 use crate::directory::META_LOCK;
@@ -8,6 +9,7 @@ use crate::directory::{WatchCallback, WatchHandle};
 use crate::error::DataCorruption;
 use crate::Directory;
 use crate::Result;
+use crc32fast::Hasher;
 use serde_json;
 use std::collections::HashSet;
 use std::io;
@@ -207,17 +209,59 @@ impl ManagedDirectory {
         }
         Ok(())
     }
+
+    /// Verify checksum of a managed file
+    pub fn validate_checksum(&self, path: &Path) -> result::Result<bool, OpenReadError> {
+        let reader = self.directory.open_read(path)?;
+        let (footer, data) = Footer::extract_footer(reader)
+            .map_err(|err| IOError::with_path(path.to_path_buf(), err))?;
+        let mut hasher = Hasher::new();
+        hasher.update(data.as_slice());
+        let crc = hasher.finalize();
+        Ok(footer
+            .versioned_footer
+            .crc()
+            .map(|v| v == crc)
+            .unwrap_or(false))
+    }
+
+    /// List files for which checksum does not match content
+    pub fn list_damaged(&self) -> result::Result<HashSet<PathBuf>, OpenReadError> {
+        let mut hashset = HashSet::new();
+        let managed_paths = self
+            .meta_informations
+            .read()
+            .expect("Managed directory rlock poisoned in list damaged.")
+            .managed_paths
+            .clone();
+
+        for path in managed_paths.into_iter() {
+            if !self.validate_checksum(&path)? {
+                hashset.insert(path);
+            }
+        }
+        Ok(hashset)
+    }
 }
 
 impl Directory for ManagedDirectory {
     fn open_read(&self, path: &Path) -> result::Result<ReadOnlySource, OpenReadError> {
-        self.directory.open_read(path)
+        let read_only_source = self.directory.open_read(path)?;
+        let (_footer, reader) = Footer::extract_footer(read_only_source)
+            .map_err(|err| IOError::with_path(path.to_path_buf(), err))?;
+        Ok(reader)
     }
 
     fn open_write(&mut self, path: &Path) -> result::Result<WritePtr, OpenWriteError> {
         self.register_file_as_managed(path)
             .map_err(|e| IOError::with_path(path.to_owned(), e))?;
-        self.directory.open_write(path)
+        Ok(io::BufWriter::new(Box::new(FooterProxy::new(
+            self.directory
+                .open_write(path)?
+                .into_inner()
+                .map_err(|_| ())
+                .expect("buffer should be empty"),
+        ))))
     }
 
     fn atomic_write(&mut self, path: &Path, data: &[u8]) -> io::Result<()> {
@@ -259,8 +303,9 @@ impl Clone for ManagedDirectory {
 #[cfg(test)]
 mod tests_mmap_specific {
 
-    use crate::directory::{Directory, ManagedDirectory, MmapDirectory};
+    use crate::directory::{Directory, ManagedDirectory, MmapDirectory, TerminatingWrite};
     use std::collections::HashSet;
+    use std::fs::OpenOptions;
     use std::io::Write;
     use std::path::{Path, PathBuf};
     use tempfile::TempDir;
@@ -275,8 +320,8 @@ mod tests_mmap_specific {
         {
             let mmap_directory = MmapDirectory::open(&tempdir_path).unwrap();
             let mut managed_directory = ManagedDirectory::wrap(mmap_directory).unwrap();
-            let mut write_file = managed_directory.open_write(test_path1).unwrap();
-            write_file.flush().unwrap();
+            let write_file = managed_directory.open_write(test_path1).unwrap();
+            write_file.terminate().unwrap();
             managed_directory
                 .atomic_write(test_path2, &[0u8, 1u8])
                 .unwrap();
@@ -310,9 +355,9 @@ mod tests_mmap_specific {
 
         let mmap_directory = MmapDirectory::open(&tempdir_path).unwrap();
         let mut managed_directory = ManagedDirectory::wrap(mmap_directory).unwrap();
-        managed_directory
-            .atomic_write(test_path1, &vec![0u8, 1u8])
-            .unwrap();
+        let mut write = managed_directory.open_write(test_path1).unwrap();
+        write.write_all(&[0u8, 1u8]).unwrap();
+        write.terminate().unwrap();
         assert!(managed_directory.exists(test_path1));
 
         let _mmap_read = managed_directory.open_read(test_path1).unwrap();
@@ -331,4 +376,38 @@ mod tests_mmap_specific {
         }
     }
 
+    #[test]
+    fn test_checksum() {
+        let test_path1: &'static Path = Path::new("some_path_for_test");
+        let test_path2: &'static Path = Path::new("other_test_path");
+
+        let tempdir = TempDir::new().unwrap();
+        let tempdir_path = PathBuf::from(tempdir.path());
+
+        let mmap_directory = MmapDirectory::open(&tempdir_path).unwrap();
+        let mut managed_directory = ManagedDirectory::wrap(mmap_directory).unwrap();
+        let mut write = managed_directory.open_write(test_path1).unwrap();
+        write.write_all(&[0u8, 1u8]).unwrap();
+        write.terminate().unwrap();
+
+        let mut write = managed_directory.open_write(test_path2).unwrap();
+        write.write_all(&[3u8, 4u8, 5u8]).unwrap();
+        write.terminate().unwrap();
+
+        assert!(managed_directory.list_damaged().unwrap().is_empty());
+
+        let mut corrupted_path = tempdir_path.clone();
+        corrupted_path.push(test_path2);
+        let mut file = OpenOptions::new()
+            .write(true)
+            .open(&corrupted_path)
+            .unwrap();
+        file.write_all(&[255u8]).unwrap();
+        file.flush().unwrap();
+        drop(file);
+
+        let damaged = managed_directory.list_damaged().unwrap();
+        assert_eq!(damaged.len(), 1);
+        assert!(damaged.contains(test_path2));
+    }
 }

src/directory/mmap_directory.rs

@@ -11,6 +11,7 @@ use crate::directory::error::{
     DeleteError, IOError, OpenDirectoryError, OpenReadError, OpenWriteError,
 };
 use crate::directory::read_only_source::BoxedData;
+use crate::directory::AntiCallToken;
 use crate::directory::Directory;
 use crate::directory::DirectoryLock;
 use crate::directory::Lock;
@@ -18,7 +19,7 @@ use crate::directory::ReadOnlySource;
 use crate::directory::WatchCallback;
 use crate::directory::WatchCallbackList;
 use crate::directory::WatchHandle;
-use crate::directory::WritePtr;
+use crate::directory::{TerminatingWrite, WritePtr};
 use atomicwrites;
 use memmap::Mmap;
 use std::collections::HashMap;
@@ -141,42 +142,28 @@ impl MmapCache {
     }
 }
 
-struct InnerWatcherWrapper {
-    _watcher: Mutex<notify::RecommendedWatcher>,
-    watcher_router: WatchCallbackList,
-}
-
-impl InnerWatcherWrapper {
-    pub fn new(path: &Path) -> Result<(Self, Receiver<notify::RawEvent>), notify::Error> {
-        let (tx, watcher_recv): (Sender<RawEvent>, Receiver<RawEvent>) = channel();
-        // We need to initialize the
-        let mut watcher = notify::raw_watcher(tx)?;
-        watcher.watch(path, RecursiveMode::Recursive)?;
-        let inner = InnerWatcherWrapper {
-            _watcher: Mutex::new(watcher),
-            watcher_router: Default::default(),
-        };
-        Ok((inner, watcher_recv))
-    }
-}
-
-#[derive(Clone)]
 struct WatcherWrapper {
-    inner: Arc<InnerWatcherWrapper>,
+    _watcher: Mutex<notify::RecommendedWatcher>,
+    watcher_router: Arc<WatchCallbackList>,
 }
 
 impl WatcherWrapper {
     pub fn new(path: &Path) -> Result<Self, OpenDirectoryError> {
-        let (inner, watcher_recv) = InnerWatcherWrapper::new(path).map_err(|err| match err {
-            notify::Error::PathNotFound => OpenDirectoryError::DoesNotExist(path.to_owned()),
-            _ => {
-                panic!("Unknown error while starting watching directory {:?}", path);
-            }
-        })?;
-        let watcher_wrapper = WatcherWrapper {
-            inner: Arc::new(inner),
-        };
-        let watcher_wrapper_clone = watcher_wrapper.clone();
+        let (tx, watcher_recv): (Sender<RawEvent>, Receiver<RawEvent>) = channel();
+        // We need to initialize the
+        let watcher = notify::raw_watcher(tx)
+            .and_then(|mut watcher| {
+                watcher.watch(path, RecursiveMode::Recursive)?;
+                Ok(watcher)
+            })
+            .map_err(|err| match err {
+                notify::Error::PathNotFound => OpenDirectoryError::DoesNotExist(path.to_owned()),
+                _ => {
+                    panic!("Unknown error while starting watching directory {:?}", path);
+                }
+            })?;
+        let watcher_router: Arc<WatchCallbackList> = Default::default();
+        let watcher_router_clone = watcher_router.clone();
         thread::Builder::new()
             .name("meta-file-watch-thread".to_string())
             .spawn(move || {
@@ -187,7 +174,7 @@ impl WatcherWrapper {
                             // We might want to be more accurate than this at one point.
                             if let Some(filename) = changed_path.file_name() {
                                 if filename == *META_FILEPATH {
-                                    watcher_wrapper_clone.inner.watcher_router.broadcast();
+                                    watcher_router_clone.broadcast();
                                 }
                             }
                         }
@@ -200,13 +187,15 @@ impl WatcherWrapper {
                         }
                     }
                 }
-            })
-            .expect("Failed to spawn thread to watch meta.json");
-        Ok(watcher_wrapper)
+            })?;
+        Ok(WatcherWrapper {
+            _watcher: Mutex::new(watcher),
+            watcher_router,
+        })
     }
 
     pub fn watch(&mut self, watch_callback: WatchCallback) -> WatchHandle {
-        self.inner.watcher_router.subscribe(watch_callback)
+        self.watcher_router.subscribe(watch_callback)
     }
 }
 
@@ -412,6 +401,12 @@ impl Seek for SafeFileWriter {
     }
 }
 
+impl TerminatingWrite for SafeFileWriter {
+    fn terminate_ref(&mut self, _: AntiCallToken) -> io::Result<()> {
+        self.flush()
+    }
+}
+
 impl Directory for MmapDirectory {
     fn open_read(&self, path: &Path) -> result::Result<ReadOnlySource, OpenReadError> {
         debug!("Open Read {:?}", path);

src/directory/mod.rs

@@ -9,6 +9,7 @@ mod mmap_directory;
 
 mod directory;
 mod directory_lock;
+mod footer;
 mod managed_directory;
 mod ram_directory;
 mod read_only_source;
@@ -24,18 +25,49 @@ pub use self::ram_directory::RAMDirectory;
 pub use self::read_only_source::ReadOnlySource;
 pub(crate) use self::watch_event_router::WatchCallbackList;
 pub use self::watch_event_router::{WatchCallback, WatchHandle};
-use std::io::{BufWriter, Write};
+use std::io::{self, BufWriter, Write};
 
 #[cfg(feature = "mmap")]
 pub use self::mmap_directory::MmapDirectory;
 
 pub use self::managed_directory::ManagedDirectory;
 
+/// Struct used to prevent from calling [`terminate_ref`](trait.TerminatingWrite#method.terminate_ref) directly
+pub struct AntiCallToken(());
+
+/// Trait used to indicate when no more write need to be done on a writer
+pub trait TerminatingWrite: Write {
+    /// Indicate that the writer will no longer be used. Internally call terminate_ref.
+    fn terminate(mut self) -> io::Result<()>
+    where
+        Self: Sized,
+    {
+        self.terminate_ref(AntiCallToken(()))
+    }
+
+    /// You should implement this function to define custom behavior.
+    /// This function should flush any buffer it may hold.
+    fn terminate_ref(&mut self, _: AntiCallToken) -> io::Result<()>;
+}
+
+impl<W: TerminatingWrite + ?Sized> TerminatingWrite for Box<W> {
+    fn terminate_ref(&mut self, token: AntiCallToken) -> io::Result<()> {
+        self.as_mut().terminate_ref(token)
+    }
+}
+
+impl<W: TerminatingWrite> TerminatingWrite for BufWriter<W> {
+    fn terminate_ref(&mut self, a: AntiCallToken) -> io::Result<()> {
+        self.flush()?;
+        self.get_mut().terminate_ref(a)
+    }
+}
+
 /// Write object for Directory.
 ///
 /// `WritePtr` are required to implement both Write
 /// and Seek.
-pub type WritePtr = BufWriter<Box<dyn Write>>;
+pub type WritePtr = BufWriter<Box<dyn TerminatingWrite>>;
 
 #[cfg(test)]
 mod tests;

src/directory/ram_directory.rs

@@ -1,8 +1,9 @@
 use crate::core::META_FILEPATH;
 use crate::directory::error::{DeleteError, OpenReadError, OpenWriteError};
+use crate::directory::AntiCallToken;
 use crate::directory::WatchCallbackList;
-use crate::directory::WritePtr;
 use crate::directory::{Directory, ReadOnlySource, WatchCallback, WatchHandle};
+use crate::directory::{TerminatingWrite, WritePtr};
 use fail::fail_point;
 use std::collections::HashMap;
 use std::fmt;
@@ -71,6 +72,12 @@ impl Write for VecWriter {
     }
 }
 
+impl TerminatingWrite for VecWriter {
+    fn terminate_ref(&mut self, _: AntiCallToken) -> io::Result<()> {
+        self.flush()
+    }
+}
+
 #[derive(Default)]
 struct InnerDirectory {
     fs: HashMap<PathBuf, ReadOnlySource>,

src/directory/tests.rs

@@ -127,7 +127,7 @@ fn test_watch(directory: &mut dyn Directory) {
         assert!(directory
             .atomic_write(Path::new("meta.json"), b"random_test_data_2")
             .is_ok());
-        for _ in 0..100 {
+        for _ in 0..1_000 {
             if counter.load(Ordering::SeqCst) > i {
                 break;
             }

src/directory/watch_event_router.rs

@@ -152,5 +152,4 @@ mod tests {
         thread::sleep(Duration::from_millis(WAIT_TIME));
         assert_eq!(2, counter.load(Ordering::SeqCst));
     }
-
 }

src/fastfield/mod.rs

@@ -429,7 +429,6 @@ mod tests {
             }
         }
     }
-
 }
 
 #[cfg(all(test, feature = "unstable"))]
@@ -437,9 +436,9 @@ mod bench {
     use super::tests::FIELD;
     use super::tests::{generate_permutation, SCHEMA};
     use super::*;
-    use common::CompositeFile;
-    use directory::{Directory, RAMDirectory, WritePtr};
-    use fastfield::FastFieldReader;
+    use crate::common::CompositeFile;
+    use crate::directory::{Directory, RAMDirectory, WritePtr};
+    use crate::fastfield::FastFieldReader;
     use std::collections::HashMap;
     use std::path::Path;
     use test::{self, Bencher};
@@ -537,5 +536,4 @@ mod bench {
             });
         }
     }
-
 }

src/fastfield/readers.rs

@@ -59,8 +59,7 @@ impl FastFieldReaders {
             fast_bytes: Default::default(),
             fast_fields_composite: fast_fields_composite.clone(),
         };
-        for (field_id, field_entry) in schema.fields().iter().enumerate() {
-            let field = Field(field_id as u32);
+        for (field, field_entry) in schema.fields() {
             let field_type = field_entry.field_type();
             if field_type == &FieldType::Bytes {
                 let idx_reader = fast_fields_composite

src/fastfield/writer.rs

@@ -24,8 +24,7 @@ impl FastFieldsWriter {
         let mut multi_values_writers = Vec::new();
         let mut bytes_value_writers = Vec::new();
 
-        for (field_id, field_entry) in schema.fields().iter().enumerate() {
-            let field = Field(field_id as u32);
+        for (field, field_entry) in schema.fields() {
             let default_value = match *field_entry.field_type() {
                 FieldType::I64(_) => common::i64_to_u64(0i64),
                 FieldType::F64(_) => common::f64_to_u64(0.0f64),

src/fieldnorm/writer.rs

@@ -22,11 +22,14 @@ impl FieldNormsWriter {
     pub(crate) fn fields_with_fieldnorm(schema: &Schema) -> Vec<Field> {
         schema
             .fields()
-            .iter()
-            .enumerate()
-            .filter(|&(_, field_entry)| field_entry.is_indexed())
-            .map(|(field, _)| Field(field as u32))
-            .collect::<Vec<Field>>()
+            .filter_map(|(field, field_entry)| {
+                if field_entry.is_indexed() {
+                    Some(field)
+                } else {
+                    None
+                }
+            })
+            .collect::<Vec<_>>()
     }
 
     /// Initialize with state for tracking the field norm fields
@@ -35,7 +38,7 @@ impl FieldNormsWriter {
         let fields = FieldNormsWriter::fields_with_fieldnorm(schema);
         let max_field = fields
             .iter()
-            .map(|field| field.0)
+            .map(Field::field_id)
             .max()
             .map(|max_field_id| max_field_id as usize + 1)
             .unwrap_or(0);
@@ -50,8 +53,8 @@ impl FieldNormsWriter {
     ///
     /// Will extend with 0-bytes for documents that have not been seen.
     pub fn fill_up_to_max_doc(&mut self, max_doc: DocId) {
-        for &field in self.fields.iter() {
-            self.fieldnorms_buffer[field.0 as usize].resize(max_doc as usize, 0u8);
+        for field in self.fields.iter() {
+            self.fieldnorms_buffer[field.field_id() as usize].resize(max_doc as usize, 0u8);
         }
     }
 
@@ -64,7 +67,7 @@ impl FieldNormsWriter {
     /// * field     - the field being set
     /// * fieldnorm - the number of terms present in document `doc` in field `field`
     pub fn record(&mut self, doc: DocId, field: Field, fieldnorm: u32) {
-        let fieldnorm_buffer: &mut Vec<u8> = &mut self.fieldnorms_buffer[field.0 as usize];
+        let fieldnorm_buffer: &mut Vec<u8> = &mut self.fieldnorms_buffer[field.field_id() as usize];
         assert!(
             fieldnorm_buffer.len() <= doc as usize,
             "Cannot register a given fieldnorm twice"
@@ -77,7 +80,7 @@ impl FieldNormsWriter {
     /// Serialize the seen fieldnorm values to the serializer for all fields.
     pub fn serialize(&self, fieldnorms_serializer: &mut FieldNormsSerializer) -> io::Result<()> {
         for &field in self.fields.iter() {
-            let fieldnorm_values: &[u8] = &self.fieldnorms_buffer[field.0 as usize][..];
+            let fieldnorm_values: &[u8] = &self.fieldnorms_buffer[field.field_id() as usize][..];
             fieldnorms_serializer.serialize_field(field, fieldnorm_values)?;
         }
         Ok(())

src/indexer/delete_queue.rs

@@ -258,7 +258,7 @@ mod tests {
         let delete_queue = DeleteQueue::new();
 
         let make_op = |i: usize| {
-            let field = Field(1u32);
+            let field = Field::from_field_id(1u32);
             DeleteOperation {
                 opstamp: i as u64,
                 term: Term::from_field_u64(field, i as u64),

src/indexer/index_writer.rs

@@ -8,6 +8,7 @@ use crate::core::SegmentId;
 use crate::core::SegmentMeta;
 use crate::core::SegmentReader;
 use crate::directory::DirectoryLock;
+use crate::directory::TerminatingWrite;
 use crate::docset::DocSet;
 use crate::error::TantivyError;
 use crate::fastfield::write_delete_bitset;
@@ -168,6 +169,7 @@ pub(crate) fn advance_deletes(
             segment = segment.with_delete_meta(num_deleted_docs as u32, target_opstamp);
             let mut delete_file = segment.open_write(SegmentComponent::DELETE)?;
             write_delete_bitset(&delete_bitset, &mut delete_file)?;
+            delete_file.terminate()?;
         }
     }
     segment_entry.set_meta(segment.meta().clone());
@@ -1177,5 +1179,4 @@ mod tests {
         assert!(clear_again.is_ok());
         assert!(commit_again.is_ok());
     }
-
 }

src/indexer/merger.rs

@@ -190,8 +190,7 @@ impl IndexMerger {
         fast_field_serializer: &mut FastFieldSerializer,
         mut term_ord_mappings: HashMap<Field, TermOrdinalMapping>,
     ) -> Result<()> {
-        for (field_id, field_entry) in self.schema.fields().iter().enumerate() {
-            let field = Field(field_id as u32);
+        for (field, field_entry) in self.schema.fields() {
             let field_type = field_entry.field_type();
             match *field_type {
                 FieldType::HierarchicalFacet => {
@@ -649,15 +648,12 @@ impl IndexMerger {
         serializer: &mut InvertedIndexSerializer,
     ) -> Result<HashMap<Field, TermOrdinalMapping>> {
         let mut term_ordinal_mappings = HashMap::new();
-        for (field_ord, field_entry) in self.schema.fields().iter().enumerate() {
+        for (field, field_entry) in self.schema.fields() {
             if field_entry.is_indexed() {
-                let indexed_field = Field(field_ord as u32);
-                if let Some(term_ordinal_mapping) = self.write_postings_for_field(
-                    indexed_field,
-                    field_entry.field_type(),
-                    serializer,
-                )? {
-                    term_ordinal_mappings.insert(indexed_field, term_ordinal_mapping);
+                if let Some(term_ordinal_mapping) =
+                    self.write_postings_for_field(field, field_entry.field_type(), serializer)?
+                {
+                    term_ordinal_mappings.insert(field, term_ordinal_mapping);
                 }
             }
         }

src/indexer/segment_register.rs

@@ -134,5 +134,4 @@ mod tests {
         }
         assert_eq!(segment_ids(&segment_register), vec![segment_id_merged]);
     }
-
 }

src/indexer/segment_writer.rs

@@ -6,11 +6,11 @@ use crate::fieldnorm::FieldNormsWriter;
 use crate::indexer::segment_serializer::SegmentSerializer;
 use crate::postings::compute_table_size;
 use crate::postings::MultiFieldPostingsWriter;
-use crate::schema::FieldEntry;
 use crate::schema::FieldType;
 use crate::schema::Schema;
 use crate::schema::Term;
 use crate::schema::Value;
+use crate::schema::{Field, FieldEntry};
 use crate::tokenizer::BoxedTokenizer;
 use crate::tokenizer::FacetTokenizer;
 use crate::tokenizer::{TokenStream, Tokenizer};
@@ -70,12 +70,10 @@ impl SegmentWriter {
         let table_num_bits = initial_table_size(memory_budget)?;
         let segment_serializer = SegmentSerializer::for_segment(&mut segment)?;
         let multifield_postings = MultiFieldPostingsWriter::new(schema, table_num_bits);
-        let tokenizers =
-            schema
-                .fields()
-                .iter()
-                .map(FieldEntry::field_type)
-                .map(|field_type| match *field_type {
+        let tokenizers = schema
+            .fields()
+            .map(
+                |(_, field_entry): (Field, &FieldEntry)| match field_entry.field_type() {
                     FieldType::Str(ref text_options) => text_options
                         .get_indexing_options()
                         .and_then(|text_index_option| {
@@ -83,8 +81,9 @@ impl SegmentWriter {
                             segment.index().tokenizers().get(tokenizer_name)
                         }),
                     _ => None,
-                })
-                .collect();
+                },
+            )
+            .collect();
         Ok(SegmentWriter {
             max_doc: 0,
             multifield_postings,
@@ -160,7 +159,7 @@ impl SegmentWriter {
                 }
                 FieldType::Str(_) => {
                     let num_tokens = if let Some(ref mut tokenizer) =
-                        self.tokenizers[field.0 as usize]
+                        self.tokenizers[field.field_id() as usize]
                     {
                         let texts: Vec<&str> = field_values
                             .iter()
@@ -296,5 +295,4 @@ mod tests {
         assert_eq!(initial_table_size(10_000_000).unwrap(), 17);
         assert_eq!(initial_table_size(1_000_000_000).unwrap(), 19);
     }
-
 }

src/lib.rs

@@ -212,15 +212,13 @@ pub type Score = f32;
 pub type SegmentLocalId = u32;
 
 impl DocAddress {
-    /// Return the segment ordinal.
-    /// The segment ordinal is an id identifying the segment
-    /// hosting the document. It is only meaningful, in the context
-    /// of a searcher.
+    /// Return the segment ordinal id that identifies the segment
+    /// hosting the document in the `Searcher` it is called from.
     pub fn segment_ord(self) -> SegmentLocalId {
         self.0
     }
 
-    /// Return the segment local `DocId`
+    /// Return the segment-local `DocId`
     pub fn doc(self) -> DocId {
         self.1
     }
@@ -229,11 +227,11 @@ impl DocAddress {
 /// `DocAddress` contains all the necessary information
 /// to identify a document given a `Searcher` object.
 ///
-/// It consists in an id identifying its segment, and
-/// its segment-local `DocId`.
+/// It consists of an id identifying its segment, and
+/// a segment-local `DocId`.
 ///
 /// The id used for the segment is actually an ordinal
-/// in the list of segment hold by a `Searcher`.
+/// in the list of `Segment`s held by a `Searcher`.
 #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
 pub struct DocAddress(pub SegmentLocalId, pub DocId);
 

src/postings/compression/mod.rs

@@ -274,13 +274,15 @@ pub mod tests {
 mod bench {
 
     use super::*;
+    use rand::rngs::StdRng;
+    use rand::Rng;
     use rand::SeedableRng;
-    use rand::{Rng, XorShiftRng};
     use test::Bencher;
 
     fn generate_array_with_seed(n: usize, ratio: f64, seed_val: u8) -> Vec<u32> {
-        let seed: &[u8; 16] = &[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, seed_val];
-        let mut rng: XorShiftRng = XorShiftRng::from_seed(*seed);
+        let mut seed: [u8; 32] = [0; 32];
+        seed[31] = seed_val;
+        let mut rng = StdRng::from_seed(seed);
         (0u32..).filter(|_| rng.gen_bool(ratio)).take(n).collect()
     }
 

src/postings/mod.rs

@@ -356,9 +356,9 @@ pub mod tests {
 
     #[test]
     fn test_skip_next() {
-        let term_0 = Term::from_field_u64(Field(0), 0);
-        let term_1 = Term::from_field_u64(Field(0), 1);
-        let term_2 = Term::from_field_u64(Field(0), 2);
+        let term_0 = Term::from_field_u64(Field::from_field_id(0), 0);
+        let term_1 = Term::from_field_u64(Field::from_field_id(0), 1);
+        let term_2 = Term::from_field_u64(Field::from_field_id(0), 2);
 
         let num_docs = 300u32;
 
@@ -511,19 +511,19 @@ pub mod tests {
     }
 
     pub static TERM_A: Lazy<Term> = Lazy::new(|| {
-        let field = Field(0);
+        let field = Field::from_field_id(0);
         Term::from_field_text(field, "a")
     });
     pub static TERM_B: Lazy<Term> = Lazy::new(|| {
-        let field = Field(0);
+        let field = Field::from_field_id(0);
         Term::from_field_text(field, "b")
     });
     pub static TERM_C: Lazy<Term> = Lazy::new(|| {
-        let field = Field(0);
+        let field = Field::from_field_id(0);
         Term::from_field_text(field, "c")
     });
     pub static TERM_D: Lazy<Term> = Lazy::new(|| {
-        let field = Field(0);
+        let field = Field::from_field_id(0);
         Term::from_field_text(field, "d")
     });
 
@@ -622,23 +622,23 @@ pub mod tests {
             assert!(!postings_unopt.advance());
         }
     }
-
 }
 
 #[cfg(all(test, feature = "unstable"))]
 mod bench {
 
     use super::tests::*;
-    use docset::SkipResult;
-    use query::Intersection;
-    use schema::IndexRecordOption;
+    use crate::docset::SkipResult;
+    use crate::query::Intersection;
+    use crate::schema::IndexRecordOption;
+    use crate::tests;
+    use crate::DocSet;
     use test::{self, Bencher};
-    use tests;
-    use DocSet;
 
     #[bench]
     fn bench_segment_postings(b: &mut Bencher) {
-        let searcher = INDEX.searcher();
+        let reader = INDEX.reader().unwrap();
+        let searcher = reader.searcher();
         let segment_reader = searcher.segment_reader(0);
 
         b.iter(|| {
@@ -652,7 +652,8 @@ mod bench {
 
     #[bench]
     fn bench_segment_intersection(b: &mut Bencher) {
-        let searcher = INDEX.searcher();
+        let reader = INDEX.reader().unwrap();
+        let searcher = reader.searcher();
         let segment_reader = searcher.segment_reader(0);
         b.iter(|| {
             let segment_postings_a = segment_reader
@@ -682,7 +683,8 @@ mod bench {
     }
 
     fn bench_skip_next(p: f64, b: &mut Bencher) {
-        let searcher = INDEX.searcher();
+        let reader = INDEX.reader().unwrap();
+        let searcher = reader.searcher();
         let segment_reader = searcher.segment_reader(0);
         let docs = tests::sample(segment_reader.num_docs(), p);
 
@@ -737,7 +739,8 @@ mod bench {
 
     #[bench]
     fn bench_iterate_segment_postings(b: &mut Bencher) {
-        let searcher = INDEX.searcher();
+        let reader = INDEX.reader().unwrap();
+        let searcher = reader.searcher();
         let segment_reader = searcher.segment_reader(0);
         b.iter(|| {
             let n: u32 = test::black_box(17);

src/postings/postings_writer.rs

@@ -61,12 +61,12 @@ fn make_field_partition(
         .iter()
         .map(|(key, _, _)| Term::wrap(key).field())
         .enumerate();
-    let mut prev_field = Field(u32::max_value());
+    let mut prev_field_opt = None;
     let mut fields = vec![];
     let mut offsets = vec![];
     for (offset, field) in term_offsets_it {
-        if field != prev_field {
-            prev_field = field;
+        if Some(field) != prev_field_opt {
+            prev_field_opt = Some(field);
             fields.push(field);
             offsets.push(offset);
         }
@@ -86,8 +86,7 @@ impl MultiFieldPostingsWriter {
         let term_index = TermHashMap::new(table_bits);
         let per_field_postings_writers: Vec<_> = schema
             .fields()
-            .iter()
-            .map(|field_entry| posting_from_field_entry(field_entry))
+            .map(|(_, field_entry)| posting_from_field_entry(field_entry))
             .collect();
         MultiFieldPostingsWriter {
             heap: MemoryArena::new(),
@@ -107,7 +106,8 @@ impl MultiFieldPostingsWriter {
         field: Field,
         token_stream: &mut dyn TokenStream,
     ) -> u32 {
-        let postings_writer = self.per_field_postings_writers[field.0 as usize].deref_mut();
+        let postings_writer =
+            self.per_field_postings_writers[field.field_id() as usize].deref_mut();
         postings_writer.index_text(
             &mut self.term_index,
             doc,
@@ -118,7 +118,8 @@ impl MultiFieldPostingsWriter {
     }
 
     pub fn subscribe(&mut self, doc: DocId, term: &Term) -> UnorderedTermId {
-        let postings_writer = self.per_field_postings_writers[term.field().0 as usize].deref_mut();
+        let postings_writer =
+            self.per_field_postings_writers[term.field().field_id() as usize].deref_mut();
         postings_writer.subscribe(&mut self.term_index, doc, 0u32, term, &mut self.heap)
     }
 
@@ -160,7 +161,7 @@ impl MultiFieldPostingsWriter {
                 FieldType::Bytes => {}
             }
 
-            let postings_writer = &self.per_field_postings_writers[field.0 as usize];
+            let postings_writer = &self.per_field_postings_writers[field.field_id() as usize];
             let mut field_serializer =
                 serializer.new_field(field, postings_writer.total_num_tokens())?;
             postings_writer.serialize(

src/postings/stacker/expull.rs

@@ -310,6 +310,7 @@ mod bench {
     use super::super::MemoryArena;
     use super::ExpUnrolledLinkedList;
     use byteorder::{NativeEndian, WriteBytesExt};
+    use std::iter;
     use test::Bencher;
 
     const NUM_STACK: usize = 10_000;
@@ -335,11 +336,10 @@ mod bench {
     fn bench_push_stack(bench: &mut Bencher) {
         bench.iter(|| {
             let mut heap = MemoryArena::new();
-            let mut stacks = Vec::with_capacity(100);
-            for _ in 0..NUM_STACK {
-                let mut stack = ExpUnrolledLinkedList::new();
-                stacks.push(stack);
-            }
+            let mut stacks: Vec<ExpUnrolledLinkedList> =
+                iter::repeat_with(ExpUnrolledLinkedList::new)
+                    .take(NUM_STACK)
+                    .collect();
             for s in 0..NUM_STACK {
                 for i in 0u32..STACK_SIZE {
                     let t = s * 392017 % NUM_STACK;

src/query/all_query.rs

@@ -130,5 +130,4 @@ mod tests {
             assert!(!scorer.advance());
         }
     }
-
 }

src/query/bitset/mod.rs

@@ -216,7 +216,6 @@ mod tests {
             assert!(!docset.advance());
         }
     }
-
 }
 
 #[cfg(all(test, feature = "unstable"))]
@@ -224,13 +223,12 @@ mod bench {
 
     use super::BitSet;
     use super::BitSetDocSet;
-    use test;
-    use tests;
-    use DocSet;
+    use crate::test;
+    use crate::tests;
+    use crate::DocSet;
 
     #[bench]
     fn bench_bitset_1pct_insert(b: &mut test::Bencher) {
-        use tests;
         let els = tests::generate_nonunique_unsorted(1_000_000u32, 10_000);
         b.iter(|| {
             let mut bitset = BitSet::with_max_value(1_000_000);
@@ -242,7 +240,6 @@ mod bench {
 
     #[bench]
     fn bench_bitset_1pct_clone(b: &mut test::Bencher) {
-        use tests;
         let els = tests::generate_nonunique_unsorted(1_000_000u32, 10_000);
         let mut bitset = BitSet::with_max_value(1_000_000);
         for el in els {

src/query/bm25.rs

@@ -137,5 +137,4 @@ mod tests {
     fn test_idf() {
         assert_nearly_equals(idf(1, 2), 0.6931472);
     }
-
 }

src/query/boolean_query/boolean_query.rs

@@ -9,7 +9,8 @@ use crate::Result;
 use crate::Searcher;
 use std::collections::BTreeSet;
 
-/// The boolean query combines a set of queries
+/// The boolean query returns a set of documents
+/// that matches the Boolean combination of constituent subqueries.
 ///
 /// The documents matched by the boolean query are
 /// those which
@@ -19,6 +20,113 @@ use std::collections::BTreeSet;
 /// `MustNot` occurence.
 /// * match at least one of the subqueries that is not
 /// a `MustNot` occurence.
+///
+///
+/// You can combine other query types and their `Occur`ances into one `BooleanQuery`
+///
+/// ```rust
+///use tantivy::collector::Count;
+///use tantivy::doc;
+///use tantivy::query::{BooleanQuery, Occur, PhraseQuery, Query, TermQuery};
+///use tantivy::schema::{IndexRecordOption, Schema, TEXT};
+///use tantivy::Term;
+///use tantivy::{Index, Result};
+///
+///fn main() -> Result<()> {
+///    let mut schema_builder = Schema::builder();
+///    let title = schema_builder.add_text_field("title", TEXT);
+///    let body = schema_builder.add_text_field("body", TEXT);
+///    let schema = schema_builder.build();
+///    let index = Index::create_in_ram(schema);
+///    {
+///        let mut index_writer = index.writer(3_000_000)?;
+///        index_writer.add_document(doc!(
+///            title => "The Name of the Wind",
+///        ));
+///        index_writer.add_document(doc!(
+///            title => "The Diary of Muadib",
+///        ));
+///        index_writer.add_document(doc!(
+///            title => "A Dairy Cow",
+///            body => "hidden",
+///        ));
+///        index_writer.add_document(doc!(
+///            title => "A Dairy Cow",
+///            body => "found",
+///        ));
+///        index_writer.add_document(doc!(
+///            title => "The Diary of a Young Girl",
+///        ));
+///        index_writer.commit().unwrap();
+///    }
+///
+///    let reader = index.reader()?;
+///    let searcher = reader.searcher();
+///
+///    // Make TermQuery's for "girl" and "diary" in the title
+///    let girl_term_query: Box<dyn Query> = Box::new(TermQuery::new(
+///        Term::from_field_text(title, "girl"),
+///        IndexRecordOption::Basic,
+///    ));
+///    let diary_term_query: Box<dyn Query> = Box::new(TermQuery::new(
+///        Term::from_field_text(title, "diary"),
+///        IndexRecordOption::Basic,
+///    ));
+///    // A TermQuery with "found" in the body
+///    let body_term_query: Box<dyn Query> = Box::new(TermQuery::new(
+///        Term::from_field_text(body, "found"),
+///        IndexRecordOption::Basic,
+///    ));
+///    // TermQuery "diary" must and "girl" must not be present
+///    let queries_with_occurs1 = vec![
+///        (Occur::Must, diary_term_query.box_clone()),
+///        (Occur::MustNot, girl_term_query),
+///    ];
+///    // Make a BooleanQuery equivalent to
+///    // title:+diary title:-girl
+///    let diary_must_and_girl_mustnot = BooleanQuery::from(queries_with_occurs1);
+///    let count1 = searcher.search(&diary_must_and_girl_mustnot, &Count)?;
+///    assert_eq!(count1, 1);
+///
+///    // TermQuery for "cow" in the title
+///    let cow_term_query: Box<dyn Query> = Box::new(TermQuery::new(
+///        Term::from_field_text(title, "cow"),
+///        IndexRecordOption::Basic,
+///    ));
+///    // "title:diary OR title:cow"
+///    let title_diary_or_cow = BooleanQuery::from(vec![
+///        (Occur::Should, diary_term_query.box_clone()),
+///        (Occur::Should, cow_term_query),
+///    ]);
+///    let count2 = searcher.search(&title_diary_or_cow, &Count)?;
+///    assert_eq!(count2, 4);
+///
+///    // Make a `PhraseQuery` from a vector of `Term`s
+///    let phrase_query: Box<dyn Query> = Box::new(PhraseQuery::new(vec![
+///        Term::from_field_text(title, "dairy"),
+///        Term::from_field_text(title, "cow"),
+///    ]));
+///    // You can combine subqueries of different types into 1 BooleanQuery:
+///    // `TermQuery` and `PhraseQuery`
+///    // "title:diary OR "dairy cow"
+///    let term_of_phrase_query = BooleanQuery::from(vec![
+///        (Occur::Should, diary_term_query.box_clone()),
+///        (Occur::Should, phrase_query.box_clone()),
+///    ]);
+///    let count3 = searcher.search(&term_of_phrase_query, &Count)?;
+///    assert_eq!(count3, 4);
+///
+///    // You can nest one BooleanQuery inside another
+///    // body:found AND ("title:diary OR "dairy cow")
+///    let nested_query = BooleanQuery::from(vec![
+///        (Occur::Must, body_term_query),
+///        (Occur::Must, Box::new(term_of_phrase_query))
+///    ]);
+///    let count4 = searcher.search(&nested_query, &Count)?;
+///    assert_eq!(count4, 1);
+///    Ok(())
+///}
+/// ```
 #[derive(Debug)]
 pub struct BooleanQuery {
     subqueries: Vec<(Occur, Box<dyn Query>)>,

src/query/boolean_query/mod.rs

@@ -247,9 +247,7 @@ mod tests {
         let reader = index.reader().unwrap();
         let searcher = reader.searcher();
         let query_parser = QueryParser::for_index(&index, vec![title, text]);
-        let query = query_parser
-            .parse_query("Оксана Лифенко")
-            .unwrap();
+        let query = query_parser.parse_query("Оксана Лифенко").unwrap();
         let weight = query.weight(&searcher, true).unwrap();
         let mut scorer = weight.scorer(searcher.segment_reader(0u32)).unwrap();
         scorer.advance();

src/query/exclude.rs

@@ -175,5 +175,4 @@ mod tests {
             sample_skip,
         );
     }
-
 }

src/query/phrase_query/phrase_query.rs

@@ -40,7 +40,7 @@ impl PhraseQuery {
         PhraseQuery::new_with_offset(terms_with_offset)
     }
 
-    /// Creates a new `PhraseQuery` given a list of terms and there offsets.
+    /// Creates a new `PhraseQuery` given a list of terms and their offsets.
     ///
     /// Can be used to provide custom offset for each term.
     pub fn new_with_offset(mut terms: Vec<(usize, Term)>) -> PhraseQuery {
@@ -73,7 +73,7 @@ impl PhraseQuery {
             .collect::<Vec<Term>>()
     }
 
-    /// Returns the `PhraseWeight` for the given phrase query given a specific `searcher`.  
+    /// Returns the `PhraseWeight` for the given phrase query given a specific `searcher`.
     ///
     /// This function is the same as `.weight(...)` except it returns
     /// a specialized type `PhraseWeight` instead of a Boxed trait.

src/query/query_parser/query_parser.rs

@@ -674,13 +674,19 @@ mod test {
 
         test_parse_query_to_logical_ast_helper(
             "signed:-2324",
-            &format!("{:?}", Term::from_field_i64(Field(2u32), -2324)),
+            &format!(
+                "{:?}",
+                Term::from_field_i64(Field::from_field_id(2u32), -2324)
+            ),
             false,
         );
 
         test_parse_query_to_logical_ast_helper(
             "float:2.5",
-            &format!("{:?}", Term::from_field_f64(Field(10u32), 2.5)),
+            &format!(
+                "{:?}",
+                Term::from_field_f64(Field::from_field_id(10u32), 2.5)
+            ),
             false,
         );
     }

src/query/range_query.rs

@@ -479,5 +479,4 @@ mod tests {
             91
         );
     }
-
 }

src/query/reqopt_scorer.rs

@@ -190,5 +190,4 @@ mod tests {
             skip_docs,
         );
     }
-
 }

src/query/term_query/mod.rs

@@ -118,7 +118,7 @@ mod tests {
     #[test]
     fn test_term_query_debug() {
         let term_query = TermQuery::new(
-            Term::from_field_text(Field(1), "hello"),
+            Term::from_field_text(Field::from_field_id(1), "hello"),
             IndexRecordOption::WithFreqs,
         );
         assert_eq!(

src/query/union.rs

@@ -409,20 +409,17 @@ mod tests {
             vec![1, 2, 3, 7, 8, 9, 99, 100, 101, 500, 20000],
         );
     }
-
 }
 
 #[cfg(all(test, feature = "unstable"))]
 mod bench {
 
-    use query::score_combiner::DoNothingCombiner;
-    use query::ConstScorer;
-    use query::Union;
-    use query::VecDocSet;
+    use crate::query::score_combiner::DoNothingCombiner;
+    use crate::query::{ConstScorer, Union, VecDocSet};
+    use crate::tests;
+    use crate::DocId;
+    use crate::DocSet;
     use test::Bencher;
-    use tests;
-    use DocId;
-    use DocSet;
 
     #[bench]
     fn bench_union_3_high(bench: &mut Bencher) {

src/query/vec_docset.rs

@@ -82,5 +82,4 @@ pub mod tests {
         }
         assert_eq!(postings.fill_buffer(&mut buffer[..]), 9);
     }
-
 }

src/schema/document.rs

@@ -178,5 +178,4 @@ mod tests {
         doc.add_text(text_field, "My title");
         assert_eq!(doc.field_values().len(), 1);
     }
-
 }

src/schema/field.rs

@@ -3,14 +3,22 @@ use std::io;
 use std::io::Read;
 use std::io::Write;
 
-/// `Field` is actually a `u8` identifying a `Field`
-/// The schema is in charge of holding mapping between field names
-/// to `Field` objects.
-///
-/// Because the field id is a `u8`, tantivy can only have at most `255` fields.
-/// Value 255 is reserved.
+/// `Field` is represented by an unsigned 32-bit integer type
+/// The schema holds the mapping between field names and `Field` objects.
 #[derive(Copy, Clone, Debug, PartialEq, PartialOrd, Eq, Ord, Hash, Serialize, Deserialize)]
-pub struct Field(pub u32);
+pub struct Field(u32);
+
+impl Field {
+    /// Create a new field object for the given FieldId.
+    pub fn from_field_id(field_id: u32) -> Field {
+        Field(field_id)
+    }
+
+    /// Returns a u32 identifying uniquely a field within a schema.
+    pub fn field_id(&self) -> u32 {
+        self.0
+    }
+}
 
 impl BinarySerializable for Field {
     fn serialize<W: Write>(&self, writer: &mut W) -> io::Result<()> {

src/schema/mod.rs

@@ -174,5 +174,4 @@ mod tests {
         assert!(!is_valid_field_name("シャボン玉"));
         assert!(is_valid_field_name("my_text_field"));
     }
-
 }

src/schema/schema.rs

@@ -167,7 +167,7 @@ impl SchemaBuilder {
 
     /// Adds a field entry to the schema in build.
     fn add_field(&mut self, field_entry: FieldEntry) -> Field {
-        let field = Field(self.fields.len() as u32);
+        let field = Field::from_field_id(self.fields.len() as u32);
         let field_name = field_entry.name().to_string();
         self.fields.push(field_entry);
         self.fields_map.insert(field_name, field);
@@ -223,7 +223,7 @@ pub struct Schema(Arc<InnerSchema>);
 impl Schema {
     /// Return the `FieldEntry` associated to a `Field`.
     pub fn get_field_entry(&self, field: Field) -> &FieldEntry {
-        &self.0.fields[field.0 as usize]
+        &self.0.fields[field.field_id() as usize]
     }
 
     /// Return the field name for a given `Field`.
@@ -232,8 +232,12 @@ impl Schema {
     }
 
     /// Return the list of all the `Field`s.
-    pub fn fields(&self) -> &[FieldEntry] {
-        &self.0.fields
+    pub fn fields(&self) -> impl Iterator<Item = (Field, &FieldEntry)> {
+        self.0
+            .fields
+            .iter()
+            .enumerate()
+            .map(|(field_id, field_entry)| (Field::from_field_id(field_id as u32), field_entry))
     }
 
     /// Creates a new builder.
@@ -485,13 +489,32 @@ mod tests {
 
         let schema: Schema = serde_json::from_str(expected).unwrap();
 
-        let mut fields = schema.fields().iter();
-
-        assert_eq!("title", fields.next().unwrap().name());
-        assert_eq!("author", fields.next().unwrap().name());
-        assert_eq!("count", fields.next().unwrap().name());
-        assert_eq!("popularity", fields.next().unwrap().name());
-        assert_eq!("score", fields.next().unwrap().name());
+        let mut fields = schema.fields();
+        {
+            let (field, field_entry) = fields.next().unwrap();
+            assert_eq!("title", field_entry.name());
+            assert_eq!(0, field.field_id());
+        }
+        {
+            let (field, field_entry) = fields.next().unwrap();
+            assert_eq!("author", field_entry.name());
+            assert_eq!(1, field.field_id());
+        }
+        {
+            let (field, field_entry) = fields.next().unwrap();
+            assert_eq!("count", field_entry.name());
+            assert_eq!(2, field.field_id());
+        }
+        {
+            let (field, field_entry) = fields.next().unwrap();
+            assert_eq!("popularity", field_entry.name());
+            assert_eq!(3, field.field_id());
+        }
+        {
+            let (field, field_entry) = fields.next().unwrap();
+            assert_eq!("score", field_entry.name());
+            assert_eq!(4, field.field_id());
+        }
         assert!(fields.next().is_none());
     }
 

src/schema/term.rs

@@ -105,7 +105,7 @@ impl Term {
         if self.0.len() < 4 {
             self.0.resize(4, 0u8);
         }
-        BigEndian::write_u32(&mut self.0[0..4], field.0);
+        BigEndian::write_u32(&mut self.0[0..4], field.field_id());
     }
 
     /// Sets a u64 value in the term.
@@ -157,7 +157,7 @@ where
 
     /// Returns the field.
     pub fn field(&self) -> Field {
-        Field(BigEndian::read_u32(&self.0.as_ref()[..4]))
+        Field::from_field_id(BigEndian::read_u32(&self.0.as_ref()[..4]))
     }
 
     /// Returns the `u64` value stored in a term.
@@ -227,7 +227,7 @@ impl fmt::Debug for Term {
         write!(
             f,
             "Term(field={},bytes={:?})",
-            self.field().0,
+            self.field().field_id(),
             self.value_bytes()
         )
     }

src/store/mod.rs

@@ -120,17 +120,16 @@ pub mod tests {
             );
         }
     }
-
 }
 
 #[cfg(all(test, feature = "unstable"))]
 mod bench {
 
     use super::tests::write_lorem_ipsum_store;
-    use directory::Directory;
-    use directory::RAMDirectory;
+    use crate::directory::Directory;
+    use crate::directory::RAMDirectory;
+    use crate::store::StoreReader;
     use std::path::Path;
-    use store::StoreReader;
     use test::Bencher;
 
     #[bench]

src/store/skiplist/mod.rs

@@ -165,5 +165,4 @@ mod tests {
         assert_eq!(output.len(), 65);
         assert_eq!(output[0], 128u8 + 3u8);
     }
-
 }

src/store/writer.rs

@@ -3,6 +3,7 @@ use super::skiplist::SkipListBuilder;
 use super::StoreReader;
 use crate::common::CountingWriter;
 use crate::common::{BinarySerializable, VInt};
+use crate::directory::TerminatingWrite;
 use crate::directory::WritePtr;
 use crate::schema::Document;
 use crate::DocId;
@@ -109,6 +110,6 @@ impl StoreWriter {
         self.offset_index_writer.write(&mut self.writer)?;
         header_offset.serialize(&mut self.writer)?;
         self.doc.serialize(&mut self.writer)?;
-        self.writer.flush()
+        self.writer.terminate()
     }
 }

src/termdict/term_info_store.rs

@@ -328,5 +328,4 @@ mod tests {
             assert_eq!(term_info_store.get(i as u64), term_infos[i]);
         }
     }
-
 }

src/tokenizer/lower_caser.rs

@@ -98,10 +98,6 @@ mod tests {
     #[test]
     fn test_lowercaser() {
         assert_eq!(lowercase_helper("Tree"), vec!["tree".to_string()]);
-        assert_eq!(
-            lowercase_helper("Русский"),
-            vec!["русский".to_string()]
-        );
+        assert_eq!(lowercase_helper("Русский"), vec!["русский".to_string()]);
     }
-
 }

src/tokenizer/mod.rs

@@ -281,5 +281,4 @@ pub mod tests {
             assert!(tokens.is_empty());
         }
     }
-
 }

src/tokenizer/ngram_tokenizer.rs

@@ -460,5 +460,4 @@ mod tests {
         assert_eq!(it.next(), Some((8, 9)));
         assert_eq!(it.next(), None);
     }
-
 }

src/tokenizer/token_stream_chain.rs

@@ -97,5 +97,4 @@ mod tests {
 
         assert!(!token_chain.advance());
     }
-
 }

tests/failpoints/mod.rs

@@ -1,7 +1,6 @@
 use fail;
-use std::io::Write;
 use std::path::Path;
-use tantivy::directory::{Directory, ManagedDirectory, RAMDirectory};
+use tantivy::directory::{Directory, ManagedDirectory, RAMDirectory, TerminatingWrite};
 use tantivy::doc;
 use tantivy::schema::{Schema, TEXT};
 use tantivy::{Index, Term};
@@ -17,7 +16,7 @@ fn test_failpoints_managed_directory_gc_if_delete_fails() {
     managed_directory
         .open_write(test_path)
         .unwrap()
-        .flush()
+        .terminate()
         .unwrap();
     assert!(managed_directory.exists(test_path));
     // triggering gc and setting the delete operation to fail.