Update field.rs

Update delete_queue.rs
Refactoring around Field
2025-12-28 13:02:55 +00:00 · 2019-10-25 09:05:56 +09:00 · 2019-10-25 09:02:57 +09:00 · 2019-10-24 10:03:59 +09:00 · 2019-10-22 09:34:10 +09:00 · 2019-10-21 10:50:53 +09:00
108 changed files with 1798 additions and 1117 deletions
--- a/.travis.yml
+++ b/.travis.yml
@@ -47,6 +47,7 @@ matrix:
 before_install:
  - set -e
  - rustup self update
+  - rustup component add rustfmt

 install:
  - sh ci/install.sh
@@ -60,6 +61,7 @@ before_script:

 script:
  - bash ci/script.sh
+  - cargo fmt --all -- --check

 before_deploy:
  - sh ci/before_deploy.sh
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,25 @@ Tantivy 0.11.0
 =====================

 - Added f64 field. Internally reuse u64 code the same way i64 does (@fdb-hiroshima)
+- Various bugfixes in the query parser.
+    - Better handling of hyphens in query parser. (#609)
+    - Better handling of whitespaces.
+- Closes #498 - add support for Elastic-style unbounded range queries for alphanumeric types eg. "title:>hello", "weight:>=70.5", "height:<200" (@petr-tik)
+- API change around `Box<BoxableTokenizer>`. See detail in #629
+- Avoid rebuilding Regex automaton whenever a regex query is reused. #639 (@brainlock)
+- Add footer with some metadata to index files. #605 (@fdb-hiroshima)
+ 
+## How to update?
+
+- `Box<dyn BoxableTokenizer>` has been replaced by a `BoxedTokenizer` struct.
+- Regex are now compiled when the `RegexQuery` instance is built. As a result, it can now return
+an error and handling the `Result` is required.
+
+
+Tantivy 0.10.2
+=====================
+
+- Closes #656. Solving memory leak.

 Tantivy 0.10.1
 =====================
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "tantivy"
-version = "0.10.1"
+version = "0.11.0"
 authors = ["Paul Masurel <paul.masurel@gmail.com>"]
 license = "MIT"
 categories = ["database-implementations", "data-structures"]
@@ -15,8 +15,9 @@ edition = "2018"
 [dependencies]
 base64 = "0.10.0"
 byteorder = "1.0"
-once_cell = "0.2"
-regex = "1.0"
+crc32fast = "1.2.0"
+once_cell = "1.0"
+regex ={version = "1.3.0", default-features = false, features = ["std"]}
 tantivy-fst = "0.1"
 memmap = {version = "0.7", optional=true}
 lz4 = {version="1.20", optional=true}
@@ -24,8 +25,6 @@ snap = {version="0.2"}
 atomicwrites = {version="0.2.2", optional=true}
 tempfile = "3.0"
 log = "0.4"
-combine = ">=3.6.0,<4.0.0"
-tempdir = "0.3"
 serde = "1.0"
 serde_derive = "1.0"
 serde_json = "1.0"
@@ -36,13 +35,14 @@ levenshtein_automata = {version="0.1", features=["fst_automaton"]}
 notify = {version="4", optional=true}
 bit-set = "0.5"
 uuid = { version = "0.7.2", features = ["v4", "serde"] }
-crossbeam = "0.5"
+crossbeam = "0.7"
 futures = "0.1"
 futures-cpupool = "0.1"
 owning_ref = "0.4"
 stable_deref_trait = "1.0.0"
 rust-stemmers = "1.1"
 downcast-rs = { version="1.0" }
+tantivy-query-grammar = { path="./query-grammar" }
 bitpacking = {version="0.8", default-features = false, features=["bitpacker4x"]}
 census = "0.2"
 fnv = "1.0.6"
@@ -81,13 +81,16 @@ failpoints = ["fail/failpoints"]
 unstable = [] # useful for benches.
 wasm-bindgen = ["uuid/wasm-bindgen"]

+[workspace]
+members = ["query-grammar"]
+
 [badges]
 travis-ci = { repository = "tantivy-search/tantivy" }

 [dev-dependencies.fail]
+version = "0.3"
 features = ["failpoints"]

-
 # Following the "fail" crate best practises, we isolate
 # tests that define specific behavior in fail check points
 # in a different binary.
@@ -98,4 +101,4 @@ features = ["failpoints"]
 [[test]]
 name = "failpoints"
 path = "tests/failpoints/mod.rs"
-required-features = ["fail/failpoints"]
+required-features = ["fail/failpoints"]
--- a/3
+++ b/3
@@ -0,0 +1,3 @@
+test:
+	echo "Run test only... No examples."
+	cargo test --tests --lib
--- a/README.md
+++ b/README.md
@@ -21,9 +21,9 @@
 [![Become a patron](https://c5.patreon.com/external/logo/become_a_patron_button.png)](https://www.patreon.com/fulmicoton)


-**Tantivy** is a **full text search engine library** written in rust.
+**Tantivy** is a **full text search engine library** written in Rust.

-It is closer to [Apache Lucene](https://lucene.apache.org/) than to [Elasticsearch](https://www.elastic.co/products/elasticsearch) and [Apache Solr](https://lucene.apache.org/solr/) in the sense it is not
+It is closer to [Apache Lucene](https://lucene.apache.org/) than to [Elasticsearch](https://www.elastic.co/products/elasticsearch) or [Apache Solr](https://lucene.apache.org/solr/) in the sense it is not
 an off-the-shelf search engine server, but rather a crate that can be used
 to build such a search engine.

@@ -31,7 +31,7 @@ Tantivy is, in fact, strongly inspired by Lucene's design.

 # Benchmark

-Tantivy is typically faster than Lucene, but the results will depend on 
+Tantivy is typically faster than Lucene, but the results depend on 
 the nature of the queries in your workload.

 The following [benchmark](https://tantivy-search.github.io/bench/) break downs 
@@ -40,19 +40,19 @@ performance for different type of queries / collection.
 # Features

 - Full-text search
- Configurable tokenizer. (stemming available for 17 latin languages. Third party support for Chinese ([tantivy-jieba](https://crates.io/crates/tantivy-jieba) and [cang-jie](https://crates.io/crates/cang-jie)) and [Japanese](https://crates.io/crates/tantivy-tokenizer-tiny-segmenter)
+- Configurable tokenizer (stemming available for 17 Latin languages with third party support for Chinese ([tantivy-jieba](https://crates.io/crates/tantivy-jieba) and [cang-jie](https://crates.io/crates/cang-jie)) and [Japanese](https://crates.io/crates/tantivy-tokenizer-tiny-segmenter))
 - Fast (check out the :racehorse: :sparkles: [benchmark](https://tantivy-search.github.io/bench/) :sparkles: :racehorse:)
 - Tiny startup time (<10ms), perfect for command line tools
- BM25 scoring (the same as lucene)
- Natural query language `(michael AND jackson) OR "king of pop"`
- Phrase queries search (`"michael jackson"`)
+- BM25 scoring (the same as Lucene)
+- Natural query language (e.g. `(michael AND jackson) OR "king of pop"`)
+- Phrase queries search (e.g. `"michael jackson"`)
 - Incremental indexing
 - Multithreaded indexing (indexing English Wikipedia takes < 3 minutes on my desktop)
 - Mmap directory
- SIMD integer compression when the platform/CPU includes the SSE2 instruction set.
- Single valued and multivalued u64, i64 and f64 fast fields (equivalent of doc values in Lucene)
+- SIMD integer compression when the platform/CPU includes the SSE2 instruction set
+- Single valued and multivalued u64, i64, and f64 fast fields (equivalent of doc values in Lucene)
 - `&[u8]` fast fields
- Text, i64, u64, f64, dates and hierarchical facet fields
+- Text, i64, u64, f64, dates, and hierarchical facet fields
 - LZ4 compressed document store
 - Range queries
 - Faceted search
@@ -61,43 +61,42 @@ performance for different type of queries / collection.

 # Non-features

- Distributed search is out of the scope of tantivy. That being said, tantivy is meant as a
+- Distributed search is out of the scope of Tantivy. That being said, Tantivy is a
 library upon which one could build a distributed search. Serializable/mergeable collector state for instance, 
-are within the scope of tantivy.
+are within the scope of Tantivy.

 # Supported OS and compiler

-Tantivy works on stable rust (>= 1.27) and supports Linux, MacOS and Windows.
+Tantivy works on stable Rust (>= 1.27) and supports Linux, MacOS, and Windows.

 # Getting started

- [tantivy's simple search example](https://tantivy-search.github.io/examples/basic_search.html)
- [tantivy-cli and its tutorial](https://github.com/tantivy-search/tantivy-cli).
-`tantivy-cli` is an actual command line interface that makes it easy for you to create a search engine,
-index documents and search via the CLI or a small server with a REST API.
-It will walk you through getting a wikipedia search engine up and running in a few minutes.
- [reference doc for the last released version](https://docs.rs/tantivy/)
+- [Tantivy's simple search example](https://tantivy-search.github.io/examples/basic_search.html)
+- [tantivy-cli and its tutorial](https://github.com/tantivy-search/tantivy-cli) - `tantivy-cli` is an actual command line interface that makes it easy for you to create a search engine,
+index documents, and search via the CLI or a small server with a REST API.
+It walks you through getting a wikipedia search engine up and running in a few minutes.
+- [Reference doc for the last released version](https://docs.rs/tantivy/)

 # How can I support this project?

 There are many ways to support this project. 

- Use tantivy and tell us about your experience on [gitter](https://gitter.im/tantivy-search/tantivy) or by email (paul.masurel@gmail.com)
+- Use Tantivy and tell us about your experience on [Gitter](https://gitter.im/tantivy-search/tantivy) or by email (paul.masurel@gmail.com)
 - Report bugs
 - Write a blog post
 - Help with documentation by asking questions or submitting PRs
- Contribute code (you can join [our gitter](https://gitter.im/tantivy-search/tantivy) )
- Talk about tantivy around you
+- Contribute code (you can join [our Gitter](https://gitter.im/tantivy-search/tantivy))
+- Talk about Tantivy around you
 - Drop a word on on [![Say Thanks!](https://img.shields.io/badge/Say%20Thanks-!-1EAEDB.svg)](https://saythanks.io/to/fulmicoton) or even [![Become a patron](https://c5.patreon.com/external/logo/become_a_patron_button.png)](https://www.patreon.com/fulmicoton)

 # Contributing code

-We use the GitHub Pull Request workflow - reference a GitHub ticket and/or include a comprehensive commit message when opening a PR.
+We use the GitHub Pull Request workflow: reference a GitHub ticket and/or include a comprehensive commit message when opening a PR.

 ## Clone and build locally

-Tantivy compiles on stable rust but requires `Rust >= 1.27`.
-To check out and run tests, you can simply run :
+Tantivy compiles on stable Rust but requires `Rust >= 1.27`.
+To check out and run tests, you can simply run:

 ```bash
    git clone https://github.com/tantivy-search/tantivy.git
@@ -108,7 +107,7 @@ To check out and run tests, you can simply run :
 ## Run tests

 Some tests will not run with just `cargo test` because of `fail-rs`.
-To run the tests exhaustively, run `./run-tests.sh`
+To run the tests exhaustively, run `./run-tests.sh`.

 ## Debug

@@ -116,13 +115,13 @@ You might find it useful to step through the programme with a debugger.

 ### A failing test

-Make sure you haven't run `cargo clean` after the most recent `cargo test` or `cargo build` to guarantee that `target/` dir exists. Use this bash script to find the most name of the most recent debug build of tantivy and run it under rust-gdb.
+Make sure you haven't run `cargo clean` after the most recent `cargo test` or `cargo build` to guarantee that the `target/` directory exists. Use this bash script to find the name of the most recent debug build of Tantivy and run it under `rust-gdb`:

 ```bash
 find target/debug/ -maxdepth 1 -executable -type f -name "tantivy*" -printf '%TY-%Tm-%Td %TT %p\n' | sort -r | cut -d " " -f 3 | xargs -I RECENT_DBG_TANTIVY rust-gdb RECENT_DBG_TANTIVY
 ```

-Now that you are in rust-gdb, you can set breakpoints on lines and methods that match your source-code and run the debug executable with flags that you normally pass to `cargo test` to like this
+Now that you are in `rust-gdb`, you can set breakpoints on lines and methods that match your source code and run the debug executable with flags that you normally pass to `cargo test` like this:

 ```bash
 $gdb run --test-threads 1 --test $NAME_OF_TEST
@@ -130,7 +129,7 @@ $gdb run --test-threads 1 --test $NAME_OF_TEST

 ### An example

-By default, rustc compiles everything in the `examples/` dir in debug mode. This makes it easy for you to make examples to reproduce bugs.
+By default, `rustc` compiles everything in the `examples/` directory in debug mode. This makes it easy for you to make examples to reproduce bugs:

 ```bash
 rust-gdb target/debug/examples/$EXAMPLE_NAME
--- a/ci/script.sh
+++ b/ci/script.sh
@@ -7,7 +7,7 @@ set -ex
 main() {
    if [ ! -z $CODECOV ]; then
        echo "Codecov"
-        cargo build --verbose && cargo coverage --verbose && bash <(curl -s https://codecov.io/bash) -s target/kcov
+        cargo build --verbose && cargo coverage --verbose --all && bash <(curl -s https://codecov.io/bash) -s target/kcov
    else
        echo "Build"
        cross build --target $TARGET
@@ -15,7 +15,8 @@ main() {
            return
        fi
        echo "Test"
-        cross test --target $TARGET --no-default-features --features mmap -- --test-threads 1
+        cross test --target $TARGET --no-default-features --features mmap
+        cross test --target $TARGET --no-default-features --features mmap query-grammar
    fi
    for example in $(ls examples/*.rs)
    do
--- a/examples/basic_search.rs
+++ b/examples/basic_search.rs
@@ -5,26 +5,23 @@
 //
 // We will :
 // - define our schema
-// = create an index in a directory
-// - index few documents in our index
-// - search for the best document matchings "sea whale"
-// - retrieve the best document original content.
+// - create an index in a directory
+// - index a few documents into our index
+// - search for the best document matching a basic query
+// - retrieve the best document's original content.

 // ---
 // Importing tantivy...
-#[macro_use]
-extern crate tantivy;
 use tantivy::collector::TopDocs;
 use tantivy::query::QueryParser;
 use tantivy::schema::*;
-use tantivy::Index;
-use tantivy::ReloadPolicy;
-use tempdir::TempDir;
+use tantivy::{doc, Index, ReloadPolicy};
+use tempfile::TempDir;

 fn main() -> tantivy::Result<()> {
    // Let's create a temporary directory for the
    // sake of this example
-    let index_path = TempDir::new("tantivy_example_dir")?;
+    let index_path = TempDir::new()?;

    // # Defining the schema
    //
@@ -33,7 +30,7 @@ fn main() -> tantivy::Result<()> {
    // and for each field, its type and "the way it should
    // be indexed".

-    // first we need to define a schema ...
+    // First we need to define a schema ...
    let mut schema_builder = Schema::builder();

    // Our first field is title.
@@ -48,7 +45,7 @@ fn main() -> tantivy::Result<()> {
    //
    // `STORED` means that the field will also be saved
    // in a compressed, row-oriented key-value store.
-    // This store is useful to reconstruct the
+    // This store is useful for reconstructing the
    // documents that were selected during the search phase.
    schema_builder.add_text_field("title", TEXT | STORED);

@@ -57,8 +54,7 @@ fn main() -> tantivy::Result<()> {
    // need to be able to be able to retrieve it
    // for our application.
    //
-    // We can make our index lighter and
-    // by omitting `STORED` flag.
+    // We can make our index lighter by omitting the `STORED` flag.
    schema_builder.add_text_field("body", TEXT);

    let schema = schema_builder.build();
@@ -71,7 +67,7 @@ fn main() -> tantivy::Result<()> {
    // with our schema in the directory.
    let index = Index::create_in_dir(&index_path, schema.clone())?;

-    // To insert document we need an index writer.
+    // To insert a document we will need an index writer.
    // There must be only one writer at a time.
    // This single `IndexWriter` is already
    // multithreaded.
@@ -149,8 +145,8 @@ fn main() -> tantivy::Result<()> {
    // At this point our documents are not searchable.
    //
    //
-    // We need to call .commit() explicitly to force the
-    // index_writer to finish processing the documents in the queue,
+    // We need to call `.commit()` explicitly to force the
+    // `index_writer` to finish processing the documents in the queue,
    // flush the current index to the disk, and advertise
    // the existence of new documents.
    //
@@ -162,14 +158,14 @@ fn main() -> tantivy::Result<()> {
    // persistently indexed.
    //
    // In the scenario of a crash or a power failure,
-    // tantivy behaves as if has rolled back to its last
+    // tantivy behaves as if it has rolled back to its last
    // commit.

    // # Searching
    //
    // ### Searcher
    //
-    // A reader is required to get search the index.
+    // A reader is required first in order to search an index.
    // It acts as a `Searcher` pool that reloads itself,
    // depending on a `ReloadPolicy`.
    //
@@ -185,7 +181,7 @@ fn main() -> tantivy::Result<()> {

    // We now need to acquire a searcher.
    //
-    // A searcher points to snapshotted, immutable version of the index.
+    // A searcher points to a snapshotted, immutable version of the index.
    //
    // Some search experience might require more than
    // one query. Using the same searcher ensures that all of these queries will run on the
@@ -205,7 +201,7 @@ fn main() -> tantivy::Result<()> {
    // in both title and body.
    let query_parser = QueryParser::for_index(&index, vec![title, body]);

-    // QueryParser may fail if the query is not in the right
+    // `QueryParser` may fail if the query is not in the right
    // format. For user facing applications, this can be a problem.
    // A ticket has been opened regarding this problem.
    let query = query_parser.parse_query("sea whale")?;
@@ -221,7 +217,7 @@ fn main() -> tantivy::Result<()> {
    //
    // We are not interested in all of the documents but
    // only in the top 10. Keeping track of our top 10 best documents
-    // is the role of the TopDocs.
+    // is the role of the `TopDocs` collector.

    // We can now perform our query.
    let top_docs = searcher.search(&query, &TopDocs::with_limit(10))?;
--- a/examples/custom_collector.rs
+++ b/examples/custom_collector.rs
@@ -9,15 +9,12 @@

 // ---
 // Importing tantivy...
-#[macro_use]
-extern crate tantivy;
 use tantivy::collector::{Collector, SegmentCollector};
 use tantivy::fastfield::FastFieldReader;
 use tantivy::query::QueryParser;
 use tantivy::schema::Field;
 use tantivy::schema::{Schema, FAST, INDEXED, TEXT};
-use tantivy::SegmentReader;
-use tantivy::{Index, TantivyError};
+use tantivy::{doc, Index, SegmentReader, TantivyError};

 #[derive(Default)]
 struct Stats {
--- a/examples/custom_tokenizer.rs
+++ b/examples/custom_tokenizer.rs
@@ -2,14 +2,11 @@
 //
 // In this example, we'll see how to define a tokenizer pipeline
 // by aligning a bunch of `TokenFilter`.
-
-#[macro_use]
-extern crate tantivy;
 use tantivy::collector::TopDocs;
 use tantivy::query::QueryParser;
 use tantivy::schema::*;
 use tantivy::tokenizer::NgramTokenizer;
-use tantivy::Index;
+use tantivy::{doc, Index};

 fn main() -> tantivy::Result<()> {
    // # Defining the schema
--- a/examples/deleting_updating_documents.rs
+++ b/examples/deleting_updating_documents.rs
@@ -8,13 +8,10 @@
 //
 // ---
 // Importing tantivy...
-#[macro_use]
-extern crate tantivy;
 use tantivy::collector::TopDocs;
 use tantivy::query::TermQuery;
 use tantivy::schema::*;
-use tantivy::Index;
-use tantivy::IndexReader;
+use tantivy::{doc, Index, IndexReader};

 // A simple helper function to fetch a single document
 // given its id from our index.
--- a/examples/faceted_search.rs
+++ b/examples/faceted_search.rs
@@ -12,17 +12,16 @@

 // ---
 // Importing tantivy...
-#[macro_use]
-extern crate tantivy;
 use tantivy::collector::FacetCollector;
 use tantivy::query::AllQuery;
 use tantivy::schema::*;
-use tantivy::Index;
+use tantivy::{doc, Index};
+use tempfile::TempDir;

 fn main() -> tantivy::Result<()> {
    // Let's create a temporary directory for the
    // sake of this example
-    let index_path = TempDir::new("tantivy_facet_example_dir")?;
+    let index_path = TempDir::new()?;
    let mut schema_builder = Schema::builder();

    schema_builder.add_text_field("name", TEXT | STORED);
@@ -74,5 +73,3 @@ fn main() -> tantivy::Result<()> {

    Ok(())
 }
-
-use tempdir::TempDir;
--- a/examples/integer_range_search.rs
+++ b/examples/integer_range_search.rs
@@ -2,14 +2,10 @@
 //
 // Below is an example of creating an indexed integer field in your schema
 // You can use RangeQuery to get a Count of all occurrences in a given range.
-
-#[macro_use]
-extern crate tantivy;
 use tantivy::collector::Count;
 use tantivy::query::RangeQuery;
 use tantivy::schema::{Schema, INDEXED};
-use tantivy::Index;
-use tantivy::Result;
+use tantivy::{doc, Index, Result};

 fn run() -> Result<()> {
    // For the sake of simplicity, this schema will only have 1 field
--- a/examples/iterating_docs_and_positions.rs
+++ b/examples/iterating_docs_and_positions.rs
@@ -9,11 +9,8 @@

 // ---
 // Importing tantivy...
-#[macro_use]
-extern crate tantivy;
 use tantivy::schema::*;
-use tantivy::Index;
-use tantivy::{DocId, DocSet, Postings};
+use tantivy::{doc, DocId, DocSet, Index, Postings};

 fn main() -> tantivy::Result<()> {
    // We first create a schema for the sake of the
--- a/examples/multiple_producer.rs
+++ b/examples/multiple_producer.rs
@@ -25,14 +25,11 @@

 // ---
 // Importing tantivy...
-#[macro_use]
-extern crate tantivy;
 use std::sync::{Arc, RwLock};
 use std::thread;
 use std::time::Duration;
 use tantivy::schema::{Schema, STORED, TEXT};
-use tantivy::Opstamp;
-use tantivy::{Index, IndexWriter};
+use tantivy::{doc, Index, IndexWriter, Opstamp};

 fn main() -> tantivy::Result<()> {
    // # Defining the schema
@@ -49,10 +46,9 @@ fn main() -> tantivy::Result<()> {
    thread::spawn(move || {
        // we index 100 times the document... for the sake of the example.
        for i in 0..100 {
-            let opstamp = {
-                // A read lock is sufficient here.
-                let index_writer_rlock = index_writer_clone_1.read().unwrap();
-                index_writer_rlock.add_document(
+            let opstamp = index_writer_clone_1
+                .read().unwrap() //< A read lock is sufficient here.
+                .add_document(
                    doc!(
                        title => "Of Mice and Men",
                        body => "A few miles south of Soledad, the Salinas River drops in close to the hillside \
@@ -63,8 +59,7 @@ fn main() -> tantivy::Result<()> {
                        fresh and green with every spring, carrying in their lower leaf junctures the \
                        debris of the winter’s flooding; and sycamores with mottled, white, recumbent \
                        limbs and branches that arch over the pool"
-                    ))
-            };
+                    ));
            println!("add doc {} from thread 1 - opstamp {}", i, opstamp);
            thread::sleep(Duration::from_millis(20));
        }
--- a/examples/snippet.rs
+++ b/examples/snippet.rs
@@ -7,19 +7,16 @@

 // ---
 // Importing tantivy...
-#[macro_use]
-extern crate tantivy;
 use tantivy::collector::TopDocs;
 use tantivy::query::QueryParser;
 use tantivy::schema::*;
-use tantivy::Index;
-use tantivy::{Snippet, SnippetGenerator};
-use tempdir::TempDir;
+use tantivy::{doc, Index, Snippet, SnippetGenerator};
+use tempfile::TempDir;

 fn main() -> tantivy::Result<()> {
    // Let's create a temporary directory for the
    // sake of this example
-    let index_path = TempDir::new("tantivy_example_dir")?;
+    let index_path = TempDir::new()?;

    // # Defining the schema
    let mut schema_builder = Schema::builder();
--- a/examples/stop_words.rs
+++ b/examples/stop_words.rs
@@ -11,13 +11,11 @@

 // ---
 // Importing tantivy...
-#[macro_use]
-extern crate tantivy;
 use tantivy::collector::TopDocs;
 use tantivy::query::QueryParser;
 use tantivy::schema::*;
 use tantivy::tokenizer::*;
-use tantivy::Index;
+use tantivy::{doc, Index};

 fn main() -> tantivy::Result<()> {
    // this example assumes you understand the content in `basic_search`
--- a/query-grammar/Cargo.toml
+++ b/query-grammar/Cargo.toml
@@ -0,0 +1,16 @@
+[package]
+name = "tantivy-query-grammar"
+version = "0.11.0"
+authors = ["Paul Masurel <paul.masurel@gmail.com>"]
+license = "MIT"
+categories = ["database-implementations", "data-structures"]
+description = """Search engine library"""
+documentation = "https://tantivy-search.github.io/tantivy/tantivy/index.html"
+homepage = "https://github.com/tantivy-search/tantivy"
+repository = "https://github.com/tantivy-search/tantivy"
+readme = "README.md"
+keywords = ["search", "information", "retrieval"]
+edition = "2018"
+
+[dependencies]
+combine = ">=3.6.0,<4.0.0"
--- a/query-grammar/src/lib.rs
+++ b/query-grammar/src/lib.rs
@@ -0,0 +1,17 @@
+#![recursion_limit = "100"]
+
+mod occur;
+mod query_grammar;
+mod user_input_ast;
+use combine::parser::Parser;
+
+pub use crate::occur::Occur;
+use crate::query_grammar::parse_to_ast;
+pub use crate::user_input_ast::{UserInputAST, UserInputBound, UserInputLeaf, UserInputLiteral};
+
+pub struct Error;
+
+pub fn parse_query(query: &str) -> Result<UserInputAST, Error> {
+    let (user_input_ast, _remaining) = parse_to_ast().parse(query).map_err(|_| Error)?;
+    Ok(user_input_ast)
+}
--- a/query-grammar/src/occur.rs
+++ b/query-grammar/src/occur.rs
@@ -1,5 +1,8 @@
+use std::fmt;
+use std::fmt::Write;
+
 /// Defines whether a term in a query must be present,
-/// should be present or must not be present.
+/// should be present or must be not present.
 #[derive(Debug, Clone, Hash, Copy, Eq, PartialEq)]
 pub enum Occur {
    /// For a given document to be considered for scoring,
@@ -18,32 +21,38 @@ impl Occur {
    /// - `Should` => '?',
    /// - `Must` => '+'
    /// - `Not` => '-'
-    pub fn to_char(self) -> char {
+    fn to_char(self) -> char {
        match self {
            Occur::Should => '?',
            Occur::Must => '+',
            Occur::MustNot => '-',
        }
    }
-}

-/// Compose two occur values.
-pub fn compose_occur(left: Occur, right: Occur) -> Occur {
-    match left {
-        Occur::Should => right,
-        Occur::Must => {
-            if right == Occur::MustNot {
-                Occur::MustNot
-            } else {
-                Occur::Must
+    /// Compose two occur values.
+    pub fn compose(left: Occur, right: Occur) -> Occur {
+        match left {
+            Occur::Should => right,
+            Occur::Must => {
+                if right == Occur::MustNot {
+                    Occur::MustNot
+                } else {
+                    Occur::Must
+                }
            }
-        }
-        Occur::MustNot => {
-            if right == Occur::MustNot {
-                Occur::Must
-            } else {
-                Occur::MustNot
+            Occur::MustNot => {
+                if right == Occur::MustNot {
+                    Occur::Must
+                } else {
+                    Occur::MustNot
+                }
            }
        }
    }
 }
+
+impl fmt::Display for Occur {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        f.write_char(self.to_char())
+    }
+}
--- a/query-grammar/src/query_grammar.rs
+++ b/query-grammar/src/query_grammar.rs
@@ -0,0 +1,380 @@
+use super::user_input_ast::*;
+use crate::Occur;
+use combine::char::*;
+use combine::error::StreamError;
+use combine::stream::StreamErrorFor;
+use combine::*;
+
+parser! {
+    fn field[I]()(I) -> String
+    where [I: Stream<Item = char>] {
+        (
+            letter(),
+            many(satisfy(|c: char| c.is_alphanumeric() || c == '_')),
+        ).skip(char(':')).map(|(s1, s2): (char, String)| format!("{}{}", s1, s2))
+    }
+}
+
+parser! {
+    fn word[I]()(I) -> String
+    where [I: Stream<Item = char>] {
+        (
+            satisfy(|c: char| !c.is_whitespace() && !['-', '`', ':', '{', '}', '"', '[', ']', '(',')'].contains(&c) ),
+            many(satisfy(|c: char| !c.is_whitespace() && ![':', '{', '}', '"', '[', ']', '(',')'].contains(&c)))
+        )
+        .map(|(s1, s2): (char, String)| format!("{}{}", s1, s2))
+        .and_then(|s: String|
+           match s.as_str() {
+             "OR" => Err(StreamErrorFor::<I>::unexpected_static_message("OR")),
+             "AND" => Err(StreamErrorFor::<I>::unexpected_static_message("AND")),
+             "NOT" => Err(StreamErrorFor::<I>::unexpected_static_message("NOT")),
+             _ => Ok(s)
+           })
+    }
+}
+
+parser! {
+    fn literal[I]()(I) -> UserInputLeaf
+    where [I: Stream<Item = char>]
+    {
+        let term_val = || {
+            let phrase = char('"').with(many1(satisfy(|c| c != '"'))).skip(char('"'));
+            phrase.or(word())
+        };
+        let term_val_with_field = negative_number().or(term_val());
+        let term_query =
+            (field(), term_val_with_field)
+            .map(|(field_name, phrase)| UserInputLiteral {
+                field_name: Some(field_name),
+                phrase,
+            });
+        let term_default_field = term_val().map(|phrase| UserInputLiteral {
+            field_name: None,
+            phrase,
+        });
+        attempt(term_query)
+            .or(term_default_field)
+            .map(UserInputLeaf::from)
+    }
+}
+
+parser! {
+    fn negative_number[I]()(I) -> String
+    where [I: Stream<Item = char>]
+    {
+        (char('-'), many1(satisfy(char::is_numeric)),
+         optional((char('.'), many1(satisfy(char::is_numeric)))))
+            .map(|(s1, s2, s3): (char, String, Option<(char, String)>)| {
+                if let Some(('.', s3)) = s3 {
+                    format!("{}{}.{}", s1, s2, s3)
+                } else {
+                    format!("{}{}", s1, s2)
+                }
+            })
+    }
+}
+
+parser! {
+    fn spaces1[I]()(I) -> ()
+    where [I: Stream<Item = char>] {
+        skip_many1(space())
+    }
+}
+
+parser! {
+    /// Function that parses a range out of a Stream
+    /// Supports ranges like:
+    /// [5 TO 10], {5 TO 10}, [* TO 10], [10 TO *], {10 TO *], >5, <=10
+    /// [a TO *], [a TO c], [abc TO bcd}
+    fn range[I]()(I) -> UserInputLeaf
+    where [I: Stream<Item = char>] {
+        let range_term_val = || {
+            word().or(negative_number()).or(char('*').with(value("*".to_string())))
+        };
+
+        // check for unbounded range in the form of <5, <=10, >5, >=5
+        let elastic_unbounded_range = (choice([attempt(string(">=")),
+                                               attempt(string("<=")),
+                                               attempt(string("<")),
+                                               attempt(string(">"))])
+                                       .skip(spaces()),
+                                       range_term_val()).
+            map(|(comparison_sign, bound): (&str, String)|
+                match comparison_sign {
+                    ">=" => (UserInputBound::Inclusive(bound), UserInputBound::Unbounded),
+                    "<=" => (UserInputBound::Unbounded, UserInputBound::Inclusive(bound)),
+                    "<" => (UserInputBound::Unbounded, UserInputBound::Exclusive(bound)),
+                    ">" => (UserInputBound::Exclusive(bound), UserInputBound::Unbounded),
+                    // default case
+                    _ => (UserInputBound::Unbounded, UserInputBound::Unbounded)
+                });
+        let lower_bound = (one_of("{[".chars()), range_term_val())
+            .map(|(boundary_char, lower_bound): (char, String)|
+                 if lower_bound == "*" {
+                     UserInputBound::Unbounded
+                 } else if boundary_char == '{' {
+                         UserInputBound::Exclusive(lower_bound)
+                 } else {
+                     UserInputBound::Inclusive(lower_bound)
+                 });
+        let upper_bound = (range_term_val(), one_of("}]".chars()))
+            .map(|(higher_bound, boundary_char): (String, char)|
+                 if higher_bound == "*" {
+                     UserInputBound::Unbounded
+                 } else if boundary_char == '}' {
+                     UserInputBound::Exclusive(higher_bound)
+                 } else {
+                     UserInputBound::Inclusive(higher_bound)
+                 });
+         // return only lower and upper
+        let lower_to_upper = (lower_bound.
+                                    skip((spaces(),
+                                          string("TO"),
+                                          spaces())),
+                                    upper_bound);
+
+        (optional(field()).skip(spaces()),
+         // try elastic first, if it matches, the range is unbounded
+         attempt(elastic_unbounded_range).or(lower_to_upper))
+            .map(|(field, (lower, upper))|
+                 // Construct the leaf from extracted field (optional)
+                 // and bounds
+                 UserInputLeaf::Range {
+                     field,
+                     lower,
+                     upper
+        })
+    }
+}
+
+fn negate(expr: UserInputAST) -> UserInputAST {
+    expr.unary(Occur::MustNot)
+}
+
+fn must(expr: UserInputAST) -> UserInputAST {
+    expr.unary(Occur::Must)
+}
+
+parser! {
+    fn leaf[I]()(I) -> UserInputAST
+    where [I: Stream<Item = char>] {
+            char('-').with(leaf()).map(negate)
+        .or(char('+').with(leaf()).map(must))
+        .or(char('(').with(ast()).skip(char(')')))
+        .or(char('*').map(|_| UserInputAST::from(UserInputLeaf::All)))
+        .or(attempt(string("NOT").skip(spaces1()).with(leaf()).map(negate)))
+        .or(attempt(range().map(UserInputAST::from)))
+        .or(literal().map(UserInputAST::from))
+    }
+}
+
+#[derive(Clone, Copy)]
+enum BinaryOperand {
+    Or,
+    And,
+}
+
+parser! {
+    fn binary_operand[I]()(I) -> BinaryOperand
+    where [I: Stream<Item = char>]
+    {
+       string("AND").with(value(BinaryOperand::And))
+       .or(string("OR").with(value(BinaryOperand::Or)))
+    }
+}
+
+fn aggregate_binary_expressions(
+    left: UserInputAST,
+    others: Vec<(BinaryOperand, UserInputAST)>,
+) -> UserInputAST {
+    let mut dnf: Vec<Vec<UserInputAST>> = vec![vec![left]];
+    for (operator, operand_ast) in others {
+        match operator {
+            BinaryOperand::And => {
+                if let Some(last) = dnf.last_mut() {
+                    last.push(operand_ast);
+                }
+            }
+            BinaryOperand::Or => {
+                dnf.push(vec![operand_ast]);
+            }
+        }
+    }
+    if dnf.len() == 1 {
+        UserInputAST::and(dnf.into_iter().next().unwrap()) //< safe
+    } else {
+        let conjunctions = dnf.into_iter().map(UserInputAST::and).collect();
+        UserInputAST::or(conjunctions)
+    }
+}
+
+parser! {
+    pub fn ast[I]()(I) -> UserInputAST
+    where [I: Stream<Item = char>]
+    {
+        let operand_leaf = (binary_operand().skip(spaces()), leaf().skip(spaces()));
+        let boolean_expr = (leaf().skip(spaces().silent()), many1(operand_leaf)).map(
+            |(left, right)| aggregate_binary_expressions(left,right));
+        let whitespace_separated_leaves = many1(leaf().skip(spaces().silent()))
+        .map(|subqueries: Vec<UserInputAST>|
+            if subqueries.len() == 1 {
+                subqueries.into_iter().next().unwrap()
+            } else {
+                UserInputAST::Clause(subqueries.into_iter().collect())
+            });
+        let expr = attempt(boolean_expr).or(whitespace_separated_leaves);
+        spaces().with(expr).skip(spaces())
+    }
+}
+
+parser! {
+    pub fn parse_to_ast[I]()(I) -> UserInputAST
+    where [I: Stream<Item = char>]
+    {
+        spaces().with(optional(ast()).skip(eof())).map(|opt_ast| opt_ast.unwrap_or_else(UserInputAST::empty_query))
+    }
+}
+
+#[cfg(test)]
+mod test {
+
+    use super::*;
+
+    fn test_parse_query_to_ast_helper(query: &str, expected: &str) {
+        let query = parse_to_ast().parse(query).unwrap().0;
+        let query_str = format!("{:?}", query);
+        assert_eq!(query_str, expected);
+    }
+
+    fn test_is_parse_err(query: &str) {
+        assert!(parse_to_ast().parse(query).is_err());
+    }
+
+    #[test]
+    fn test_parse_empty_to_ast() {
+        test_parse_query_to_ast_helper("", "<emptyclause>");
+    }
+
+    #[test]
+    fn test_parse_query_to_ast_hyphen() {
+        test_parse_query_to_ast_helper("\"www-form-encoded\"", "\"www-form-encoded\"");
+        test_parse_query_to_ast_helper("www-form-encoded", "\"www-form-encoded\"");
+        test_parse_query_to_ast_helper("www-form-encoded", "\"www-form-encoded\"");
+    }
+
+    #[test]
+    fn test_parse_query_to_ast_not_op() {
+        assert_eq!(
+            format!("{:?}", parse_to_ast().parse("NOT")),
+            "Err(UnexpectedParse)"
+        );
+        test_parse_query_to_ast_helper("NOTa", "\"NOTa\"");
+        test_parse_query_to_ast_helper("NOT a", "-(\"a\")");
+    }
+
+    #[test]
+    fn test_parse_query_to_ast_binary_op() {
+        test_parse_query_to_ast_helper("a AND b", "(+(\"a\") +(\"b\"))");
+        test_parse_query_to_ast_helper("a OR b", "(?(\"a\") ?(\"b\"))");
+        test_parse_query_to_ast_helper("a OR b AND c", "(?(\"a\") ?((+(\"b\") +(\"c\"))))");
+        test_parse_query_to_ast_helper("a AND b         AND c", "(+(\"a\") +(\"b\") +(\"c\"))");
+        assert_eq!(
+            format!("{:?}", parse_to_ast().parse("a OR b aaa")),
+            "Err(UnexpectedParse)"
+        );
+        assert_eq!(
+            format!("{:?}", parse_to_ast().parse("a AND b aaa")),
+            "Err(UnexpectedParse)"
+        );
+        assert_eq!(
+            format!("{:?}", parse_to_ast().parse("aaa a OR b ")),
+            "Err(UnexpectedParse)"
+        );
+        assert_eq!(
+            format!("{:?}", parse_to_ast().parse("aaa ccc a OR b ")),
+            "Err(UnexpectedParse)"
+        );
+    }
+
+    #[test]
+    fn test_parse_elastic_query_ranges() {
+        test_parse_query_to_ast_helper("title: >a", "title:{\"a\" TO \"*\"}");
+        test_parse_query_to_ast_helper("title:>=a", "title:[\"a\" TO \"*\"}");
+        test_parse_query_to_ast_helper("title: <a", "title:{\"*\" TO \"a\"}");
+        test_parse_query_to_ast_helper("title:<=a", "title:{\"*\" TO \"a\"]");
+        test_parse_query_to_ast_helper("title:<=bsd", "title:{\"*\" TO \"bsd\"]");
+
+        test_parse_query_to_ast_helper("weight: >70", "weight:{\"70\" TO \"*\"}");
+        test_parse_query_to_ast_helper("weight:>=70", "weight:[\"70\" TO \"*\"}");
+        test_parse_query_to_ast_helper("weight: <70", "weight:{\"*\" TO \"70\"}");
+        test_parse_query_to_ast_helper("weight:<=70", "weight:{\"*\" TO \"70\"]");
+        test_parse_query_to_ast_helper("weight: >60.7", "weight:{\"60.7\" TO \"*\"}");
+
+        test_parse_query_to_ast_helper("weight: <= 70", "weight:{\"*\" TO \"70\"]");
+
+        test_parse_query_to_ast_helper("weight: <= 70.5", "weight:{\"*\" TO \"70.5\"]");
+    }
+
+    #[test]
+    fn test_range_parser() {
+        // testing the range() parser separately
+        let res = range().parse("title: <hello").unwrap().0;
+        let expected = UserInputLeaf::Range {
+            field: Some("title".to_string()),
+            lower: UserInputBound::Unbounded,
+            upper: UserInputBound::Exclusive("hello".to_string()),
+        };
+        let res2 = range().parse("title:{* TO hello}").unwrap().0;
+        assert_eq!(res, expected);
+        assert_eq!(res2, expected);
+        let expected_weight = UserInputLeaf::Range {
+            field: Some("weight".to_string()),
+            lower: UserInputBound::Inclusive("71.2".to_string()),
+            upper: UserInputBound::Unbounded,
+        };
+
+        let res3 = range().parse("weight: >=71.2").unwrap().0;
+        let res4 = range().parse("weight:[71.2 TO *}").unwrap().0;
+        assert_eq!(res3, expected_weight);
+        assert_eq!(res4, expected_weight);
+    }
+
+    #[test]
+    fn test_parse_query_to_triming_spaces() {
+        test_parse_query_to_ast_helper("   abc", "\"abc\"");
+        test_parse_query_to_ast_helper("abc ", "\"abc\"");
+        test_parse_query_to_ast_helper("(  a OR abc)", "(?(\"a\") ?(\"abc\"))");
+        test_parse_query_to_ast_helper("(a  OR abc)", "(?(\"a\") ?(\"abc\"))");
+        test_parse_query_to_ast_helper("(a OR  abc)", "(?(\"a\") ?(\"abc\"))");
+        test_parse_query_to_ast_helper("a OR abc ", "(?(\"a\") ?(\"abc\"))");
+        test_parse_query_to_ast_helper("(a OR abc )", "(?(\"a\") ?(\"abc\"))");
+        test_parse_query_to_ast_helper("(a OR  abc) ", "(?(\"a\") ?(\"abc\"))");
+    }
+
+    #[test]
+    fn test_parse_query_to_ast() {
+        test_parse_query_to_ast_helper("abc", "\"abc\"");
+        test_parse_query_to_ast_helper("a b", "(\"a\" \"b\")");
+        test_parse_query_to_ast_helper("+(a b)", "+((\"a\" \"b\"))");
+        test_parse_query_to_ast_helper("+d", "+(\"d\")");
+        test_parse_query_to_ast_helper("+(a b) +d", "(+((\"a\" \"b\")) +(\"d\"))");
+        test_parse_query_to_ast_helper("(+a +b) d", "((+(\"a\") +(\"b\")) \"d\")");
+        test_parse_query_to_ast_helper("(+a)", "+(\"a\")");
+        test_parse_query_to_ast_helper("(+a +b)", "(+(\"a\") +(\"b\"))");
+        test_parse_query_to_ast_helper("abc:toto", "abc:\"toto\"");
+        test_parse_query_to_ast_helper("abc:1.1", "abc:\"1.1\"");
+        test_parse_query_to_ast_helper("+abc:toto", "+(abc:\"toto\")");
+        test_parse_query_to_ast_helper("(+abc:toto -titi)", "(+(abc:\"toto\") -(\"titi\"))");
+        test_parse_query_to_ast_helper("-abc:toto", "-(abc:\"toto\")");
+        test_parse_query_to_ast_helper("abc:a b", "(abc:\"a\" \"b\")");
+        test_parse_query_to_ast_helper("abc:\"a b\"", "abc:\"a b\"");
+        test_parse_query_to_ast_helper("foo:[1 TO 5]", "foo:[\"1\" TO \"5\"]");
+        test_parse_query_to_ast_helper("[1 TO 5]", "[\"1\" TO \"5\"]");
+        test_parse_query_to_ast_helper("foo:{a TO z}", "foo:{\"a\" TO \"z\"}");
+        test_parse_query_to_ast_helper("foo:[1 TO toto}", "foo:[\"1\" TO \"toto\"}");
+        test_parse_query_to_ast_helper("foo:[* TO toto}", "foo:{\"*\" TO \"toto\"}");
+        test_parse_query_to_ast_helper("foo:[1 TO *}", "foo:[\"1\" TO \"*\"}");
+        test_parse_query_to_ast_helper("foo:[1.1 TO *}", "foo:[\"1.1\" TO \"*\"}");
+        test_is_parse_err("abc +    ");
+    }
+}
--- a/src/query/query_parser/user_input_ast.rs
+++ b/src/query/query_parser/user_input_ast.rs
@@ -1,8 +1,9 @@
 use std::fmt;
 use std::fmt::{Debug, Formatter};

-use crate::query::Occur;
+use crate::Occur;

+#[derive(PartialEq)]
 pub enum UserInputLeaf {
    Literal(UserInputLiteral),
    All,
@@ -35,6 +36,7 @@ impl Debug for UserInputLeaf {
    }
 }

+#[derive(PartialEq)]
 pub struct UserInputLiteral {
    pub field_name: Option<String>,
    pub phrase: String,
@@ -49,9 +51,11 @@ impl fmt::Debug for UserInputLiteral {
    }
 }

+#[derive(PartialEq)]
 pub enum UserInputBound {
    Inclusive(String),
    Exclusive(String),
+    Unbounded,
 }

 impl UserInputBound {
@@ -59,6 +63,7 @@ impl UserInputBound {
        match *self {
            UserInputBound::Inclusive(ref word) => write!(formatter, "[\"{}\"", word),
            UserInputBound::Exclusive(ref word) => write!(formatter, "{{\"{}\"", word),
+            UserInputBound::Unbounded => write!(formatter, "{{\"*\""),
        }
    }

@@ -66,6 +71,7 @@ impl UserInputBound {
        match *self {
            UserInputBound::Inclusive(ref word) => write!(formatter, "\"{}\"]", word),
            UserInputBound::Exclusive(ref word) => write!(formatter, "\"{}\"}}", word),
+            UserInputBound::Unbounded => write!(formatter, "\"*\"}}"),
        }
    }

@@ -73,6 +79,7 @@ impl UserInputBound {
        match *self {
            UserInputBound::Inclusive(ref contents) => contents,
            UserInputBound::Exclusive(ref contents) => contents,
+            UserInputBound::Unbounded => &"*",
        }
    }
 }
@@ -80,9 +87,6 @@ impl UserInputBound {
 pub enum UserInputAST {
    Clause(Vec<UserInputAST>),
    Unary(Occur, Box<UserInputAST>),
-    //    Not(Box<UserInputAST>),
-    //    Should(Box<UserInputAST>),
-    //    Must(Box<UserInputAST>),
    Leaf(Box<UserInputLeaf>),
 }

@@ -92,7 +96,7 @@ impl UserInputAST {
    }

    fn compose(occur: Occur, asts: Vec<UserInputAST>) -> UserInputAST {
-        assert!(occur != Occur::MustNot);
+        assert_ne!(occur, Occur::MustNot);
        assert!(!asts.is_empty());
        if asts.len() == 1 {
            asts.into_iter().next().unwrap() //< safe
@@ -105,6 +109,10 @@ impl UserInputAST {
        }
    }

+    pub fn empty_query() -> UserInputAST {
+        UserInputAST::Clause(Vec::default())
+    }
+
    pub fn and(asts: Vec<UserInputAST>) -> UserInputAST {
        UserInputAST::compose(Occur::Must, asts)
    }
@@ -114,42 +122,6 @@ impl UserInputAST {
    }
 }

-/*
-impl UserInputAST {
-
-    fn compose_occur(self, occur: Occur) -> UserInputAST {
-        match self {
-            UserInputAST::Not(other) => {
-                let new_occur = compose_occur(Occur::MustNot, occur);
-                other.simplify()
-            }
-            _ => {
-                self
-            }
-        }
-    }
-
-    pub fn simplify(self) -> UserInputAST {
-        match self {
-            UserInputAST::Clause(els) => {
-                if els.len() == 1 {
-                    return els.into_iter().next().unwrap();
-                } else {
-                    return self;
-                }
-            }
-            UserInputAST::Not(els) => {
-                if els.len() == 1 {
-                    return els.into_iter().next().unwrap();
-                } else {
-                    return self;
-                }
-            }
-        }
-    }
-}
-*/
-
 impl From<UserInputLiteral> for UserInputLeaf {
    fn from(literal: UserInputLiteral) -> UserInputLeaf {
        UserInputLeaf::Literal(literal)
@@ -179,7 +151,7 @@ impl fmt::Debug for UserInputAST {
                Ok(())
            }
            UserInputAST::Unary(ref occur, ref subquery) => {
-                write!(formatter, "{}({:?})", occur.to_char(), subquery)
+                write!(formatter, "{}({:?})", occur, subquery)
            }
            UserInputAST::Leaf(ref subquery) => write!(formatter, "{:?}", subquery),
        }
--- a/src/collector/count_collector.rs
+++ b/src/collector/count_collector.rs
@@ -10,12 +10,10 @@ use crate::SegmentReader;
 /// documents match the query.
 ///
 /// ```rust
-/// #[macro_use]
-/// extern crate tantivy;
-/// use tantivy::schema::{Schema, TEXT};
-/// use tantivy::{Index, Result};
 /// use tantivy::collector::Count;
 /// use tantivy::query::QueryParser;
+/// use tantivy::schema::{Schema, TEXT};
+/// use tantivy::{doc, Index, Result};
 ///
 /// # fn main() { example().unwrap(); }
 /// fn example() -> Result<()> {
@@ -125,5 +123,4 @@ mod tests {
            assert_eq!(count_collector.harvest(), 2);
        }
    }
-
 }
--- a/src/collector/facet_collector.rs
+++ b/src/collector/facet_collector.rs
@@ -81,12 +81,10 @@ fn facet_depth(facet_bytes: &[u8]) -> usize {
 ///
 ///
 /// ```rust
-/// #[macro_use]
-/// extern crate tantivy;
-/// use tantivy::schema::{Facet, Schema, TEXT};
-/// use tantivy::{Index, Result};
 /// use tantivy::collector::FacetCollector;
 /// use tantivy::query::AllQuery;
+/// use tantivy::schema::{Facet, Schema, TEXT};
+/// use tantivy::{doc, Index, Result};
 ///
 /// # fn main() { example().unwrap(); }
 /// fn example() -> Result<()> {
@@ -517,7 +515,7 @@ mod tests {
    #[should_panic(expected = "Tried to add a facet which is a descendant of \
                               an already added facet.")]
    fn test_misused_facet_collector() {
-        let mut facet_collector = FacetCollector::for_field(Field(0));
+        let mut facet_collector = FacetCollector::for_field(Field::from_field_id(0));
        facet_collector.add_facet(Facet::from("/country"));
        facet_collector.add_facet(Facet::from("/country/europe"));
    }
@@ -548,7 +546,7 @@ mod tests {

    #[test]
    fn test_non_used_facet_collector() {
-        let mut facet_collector = FacetCollector::for_field(Field(0));
+        let mut facet_collector = FacetCollector::for_field(Field::from_field_id(0));
        facet_collector.add_facet(Facet::from("/country"));
        facet_collector.add_facet(Facet::from("/countryeurope"));
    }
@@ -601,19 +599,18 @@ mod tests {
            );
        }
    }
-
 }

 #[cfg(all(test, feature = "unstable"))]
 mod bench {

-    use collector::FacetCollector;
-    use query::AllQuery;
-    use rand::{thread_rng, Rng};
-    use schema::Facet;
-    use schema::Schema;
+    use crate::collector::FacetCollector;
+    use crate::query::AllQuery;
+    use crate::schema::{Facet, Schema};
+    use crate::Index;
+    use rand::seq::SliceRandom;
+    use rand::thread_rng;
    use test::Bencher;
-    use Index;

    #[bench]
    fn bench_facet_collector(b: &mut Bencher) {
@@ -630,7 +627,7 @@ mod bench {
            }
        }
        // 40425 docs
-        thread_rng().shuffle(&mut docs[..]);
+        docs[..].shuffle(&mut thread_rng());

        let mut index_writer = index.writer_with_num_threads(1, 3_000_000).unwrap();
        for doc in docs {
@@ -639,7 +636,7 @@ mod bench {
        index_writer.commit().unwrap();
        let reader = index.reader().unwrap();
        b.iter(|| {
-            let searcher = index.searcher();
+            let searcher = reader.searcher();
            let facet_collector = FacetCollector::for_field(facet_field);
            searcher.search(&AllQuery, &facet_collector).unwrap();
        });
--- a/src/collector/mod.rs
+++ b/src/collector/mod.rs
@@ -35,7 +35,6 @@ The resulting `Fruit` will then be a typed tuple with each collector's original
 in their respective position.

 ```rust
-# extern crate tantivy;
 # use tantivy::schema::*;
 # use tantivy::*;
 # use tantivy::query::*;
--- a/src/collector/multi_collector.rs
+++ b/src/collector/multi_collector.rs
@@ -105,12 +105,10 @@ impl<TFruit: Fruit> FruitHandle<TFruit> {
 /// [Combining several collectors section of the collector documentation](./index.html#combining-several-collectors).
 ///
 /// ```rust
-/// #[macro_use]
-/// extern crate tantivy;
-/// use tantivy::schema::{Schema, TEXT};
-/// use tantivy::{Index, Result};
 /// use tantivy::collector::{Count, TopDocs, MultiCollector};
 /// use tantivy::query::QueryParser;
+/// use tantivy::schema::{Schema, TEXT};
+/// use tantivy::{doc, Index, Result};
 ///
 /// # fn main() { example().unwrap(); }
 /// fn example() -> Result<()> {
--- a/src/collector/top_score_collector.rs
+++ b/src/collector/top_score_collector.rs
@@ -23,13 +23,10 @@ use std::fmt;
 /// is `O(n log K)`.
 ///
 /// ```rust
-/// #[macro_use]
-/// extern crate tantivy;
-/// use tantivy::DocAddress;
-/// use tantivy::schema::{Schema, TEXT};
-/// use tantivy::{Index, Result};
 /// use tantivy::collector::TopDocs;
 /// use tantivy::query::QueryParser;
+/// use tantivy::schema::{Schema, TEXT};
+/// use tantivy::{doc, DocAddress, Index, Result};
 ///
 /// # fn main() { example().unwrap(); }
 /// fn example() -> Result<()> {
@@ -87,10 +84,8 @@ impl TopDocs {
    /// Set top-K to rank documents by a given fast field.
    ///
    /// ```rust
-    /// #[macro_use]
-    /// extern crate tantivy;
    /// # use tantivy::schema::{Schema, FAST, TEXT};
-    /// # use tantivy::{Index, Result, DocAddress};
+    /// # use tantivy::{doc, Index, Result, DocAddress};
    /// # use tantivy::query::{Query, QueryParser};
    /// use tantivy::Searcher;
    /// use tantivy::collector::TopDocs;
@@ -128,7 +123,7 @@ impl TopDocs {
    /// ///
    /// /// `field` is required to be a FAST field.
    /// fn docs_sorted_by_rating(searcher: &Searcher,
-    ///                          query: &Query,
+    ///                          query: &dyn Query,
    ///                          sort_by_field: Field)
    ///     -> Result<Vec<(u64, DocAddress)>> {
    ///
@@ -197,10 +192,8 @@ impl TopDocs {
    /// learning-to-rank model over various features
    ///
    /// ```rust
-    /// #[macro_use]
-    /// extern crate tantivy;
    /// # use tantivy::schema::{Schema, FAST, TEXT};
-    /// # use tantivy::{Index, DocAddress, DocId, Score};
+    /// # use tantivy::{doc, Index, DocAddress, DocId, Score};
    /// # use tantivy::query::QueryParser;
    /// use tantivy::SegmentReader;
    /// use tantivy::collector::TopDocs;
@@ -302,10 +295,8 @@ impl TopDocs {
    /// # Example
    ///
    /// ```rust
-    /// # #[macro_use]
-    /// # extern crate tantivy;
    /// # use tantivy::schema::{Schema, FAST, TEXT};
-    /// # use tantivy::{Index, DocAddress, DocId};
+    /// # use tantivy::{doc, Index, DocAddress, DocId};
    /// # use tantivy::query::QueryParser;
    /// use tantivy::SegmentReader;
    /// use tantivy::collector::TopDocs;
@@ -560,7 +551,7 @@ mod tests {
            ));
        });
        let searcher = index.reader().unwrap().searcher();
-        let top_collector = TopDocs::with_limit(4).order_by_u64_field(Field(2));
+        let top_collector = TopDocs::with_limit(4).order_by_u64_field(Field::from_field_id(2));
        let segment_reader = searcher.segment_reader(0u32);
        top_collector
            .for_segment(0, segment_reader)
@@ -591,7 +582,7 @@ mod tests {
        query_field: Field,
        schema: Schema,
        mut doc_adder: impl FnMut(&mut IndexWriter) -> (),
-    ) -> (Index, Box<Query>) {
+    ) -> (Index, Box<dyn Query>) {
        let index = Index::create_in_ram(schema);

        let mut index_writer = index.writer_with_num_threads(1, 3_000_000).unwrap();
@@ -601,5 +592,4 @@ mod tests {
        let query = query_parser.parse_query(query).unwrap();
        (index, query)
    }
-
 }
--- a/src/common/composite_file.rs
+++ b/src/common/composite_file.rs
@@ -2,7 +2,7 @@ use crate::common::BinarySerializable;
 use crate::common::CountingWriter;
 use crate::common::VInt;
 use crate::directory::ReadOnlySource;
-use crate::directory::WritePtr;
+use crate::directory::{TerminatingWrite, WritePtr};
 use crate::schema::Field;
 use crate::space_usage::FieldUsage;
 use crate::space_usage::PerFieldSpaceUsage;
@@ -42,7 +42,7 @@ pub struct CompositeWrite<W = WritePtr> {
    offsets: HashMap<FileAddr, u64>,
 }

-impl<W: Write> CompositeWrite<W> {
+impl<W: TerminatingWrite + Write> CompositeWrite<W> {
    /// Crate a new API writer that writes a composite file
    /// in a given write.
    pub fn wrap(w: W) -> CompositeWrite<W> {
@@ -91,8 +91,7 @@ impl<W: Write> CompositeWrite<W> {

        let footer_len = (self.write.written_bytes() - footer_offset) as u32;
        footer_len.serialize(&mut self.write)?;
-        self.write.flush()?;
-        Ok(())
+        self.write.terminate()
    }
 }

@@ -200,13 +199,13 @@ mod test {
            let w = directory.open_write(path).unwrap();
            let mut composite_write = CompositeWrite::wrap(w);
            {
-                let mut write_0 = composite_write.for_field(Field(0u32));
+                let mut write_0 = composite_write.for_field(Field::from_field_id(0u32));
                VInt(32431123u64).serialize(&mut write_0).unwrap();
                write_0.flush().unwrap();
            }

            {
-                let mut write_4 = composite_write.for_field(Field(4u32));
+                let mut write_4 = composite_write.for_field(Field::from_field_id(4u32));
                VInt(2).serialize(&mut write_4).unwrap();
                write_4.flush().unwrap();
            }
@@ -216,14 +215,18 @@ mod test {
            let r = directory.open_read(path).unwrap();
            let composite_file = CompositeFile::open(&r).unwrap();
            {
-                let file0 = composite_file.open_read(Field(0u32)).unwrap();
+                let file0 = composite_file
+                    .open_read(Field::from_field_id(0u32))
+                    .unwrap();
                let mut file0_buf = file0.as_slice();
                let payload_0 = VInt::deserialize(&mut file0_buf).unwrap().0;
                assert_eq!(file0_buf.len(), 0);
                assert_eq!(payload_0, 32431123u64);
            }
            {
-                let file4 = composite_file.open_read(Field(4u32)).unwrap();
+                let file4 = composite_file
+                    .open_read(Field::from_field_id(4u32))
+                    .unwrap();
                let mut file4_buf = file4.as_slice();
                let payload_4 = VInt::deserialize(&mut file4_buf).unwrap().0;
                assert_eq!(file4_buf.len(), 0);
@@ -231,5 +234,4 @@ mod test {
            }
        }
    }
-
 }
--- a/src/common/counting_writer.rs
+++ b/src/common/counting_writer.rs
@@ -1,3 +1,5 @@
+use crate::directory::AntiCallToken;
+use crate::directory::TerminatingWrite;
 use std::io;
 use std::io::Write;

@@ -42,6 +44,13 @@ impl<W: Write> Write for CountingWriter<W> {
    }
 }

+impl<W: TerminatingWrite> TerminatingWrite for CountingWriter<W> {
+    fn terminate_ref(&mut self, token: AntiCallToken) -> io::Result<()> {
+        self.flush()?;
+        self.underlying.terminate_ref(token)
+    }
+}
+
 #[cfg(test)]
 mod test {

--- a/src/common/mod.rs
+++ b/src/common/mod.rs
@@ -124,26 +124,24 @@ pub fn f64_to_u64(val: f64) -> u64 {
 /// Reverse the mapping given by [`i64_to_u64`](./fn.i64_to_u64.html).
 #[inline(always)]
 pub fn u64_to_f64(val: u64) -> f64 {
-    f64::from_bits(
-        if val & HIGHEST_BIT != 0 {
-            val ^ HIGHEST_BIT
-        } else {
-            !val
-        }
-    )
+    f64::from_bits(if val & HIGHEST_BIT != 0 {
+        val ^ HIGHEST_BIT
+    } else {
+        !val
+    })
 }

 #[cfg(test)]
 pub(crate) mod test {

    pub use super::serialize::test::fixed_size_test;
-    use super::{compute_num_bits, i64_to_u64, u64_to_i64, f64_to_u64, u64_to_f64};
+    use super::{compute_num_bits, f64_to_u64, i64_to_u64, u64_to_f64, u64_to_i64};
    use std::f64;

    fn test_i64_converter_helper(val: i64) {
        assert_eq!(u64_to_i64(i64_to_u64(val)), val);
    }
-    
+
    fn test_f64_converter_helper(val: f64) {
        assert_eq!(u64_to_f64(f64_to_u64(val)), val);
    }
@@ -172,7 +170,8 @@ pub(crate) mod test {

    #[test]
    fn test_f64_order() {
-        assert!(!(f64_to_u64(f64::NEG_INFINITY)..f64_to_u64(f64::INFINITY)).contains(&f64_to_u64(f64::NAN))); //nan is not a number
+        assert!(!(f64_to_u64(f64::NEG_INFINITY)..f64_to_u64(f64::INFINITY))
+            .contains(&f64_to_u64(f64::NAN))); //nan is not a number
        assert!(f64_to_u64(1.5) > f64_to_u64(1.0)); //same exponent, different mantissa
        assert!(f64_to_u64(2.0) > f64_to_u64(1.0)); //same mantissa, different exponent
        assert!(f64_to_u64(2.0) > f64_to_u64(1.5)); //different exponent and mantissa
--- a/src/common/serialize.rs
+++ b/src/common/serialize.rs
@@ -199,10 +199,7 @@ pub mod test {
    fn test_serialize_string() {
        assert_eq!(serialize_test(String::from("")), 1);
        assert_eq!(serialize_test(String::from("ぽよぽよ")), 1 + 3 * 4);
-        assert_eq!(
-            serialize_test(String::from("富士さん見える。")),
-            1 + 3 * 8
-        );
+        assert_eq!(serialize_test(String::from("富士さん見える。")), 1 + 3 * 8);
    }

    #[test]
--- a/src/core/index.rs
+++ b/src/core/index.rs
@@ -26,9 +26,10 @@ use crate::IndexWriter;
 use crate::Result;
 use num_cpus;
 use std::borrow::BorrowMut;
+use std::collections::HashSet;
 use std::fmt;
 #[cfg(feature = "mmap")]
-use std::path::Path;
+use std::path::{Path, PathBuf};
 use std::sync::Arc;

 fn load_metas(directory: &dyn Directory, inventory: &SegmentMetaInventory) -> Result<IndexMeta> {
@@ -173,11 +174,11 @@ impl Index {
    }

    /// Helper to access the tokenizer associated to a specific field.
-    pub fn tokenizer_for_field(&self, field: Field) -> Result<Box<dyn BoxedTokenizer>> {
+    pub fn tokenizer_for_field(&self, field: Field) -> Result<BoxedTokenizer> {
        let field_entry = self.schema.get_field_entry(field);
        let field_type = field_entry.field_type();
        let tokenizer_manager: &TokenizerManager = self.tokenizers();
-        let tokenizer_name_opt: Option<Box<dyn BoxedTokenizer>> = match field_type {
+        let tokenizer_name_opt: Option<BoxedTokenizer> = match field_type {
            FieldType::Str(text_options) => text_options
                .get_indexing_options()
                .map(|text_indexing_options| text_indexing_options.tokenizer().to_string())
@@ -216,8 +217,22 @@ impl Index {
        Index::open(mmap_directory)
    }

-    pub(crate) fn inventory(&self) -> &SegmentMetaInventory {
-        &self.inventory
+    /// Returns the list of the segment metas tracked by the index.
+    ///
+    /// Such segments can of course be part of the index,
+    /// but also they could be segments being currently built or in the middle of a merge
+    /// operation.
+    pub fn list_all_segment_metas(&self) -> Vec<SegmentMeta> {
+        self.inventory.all()
+    }
+
+    /// Creates a new segment_meta (Advanced user only).
+    ///
+    /// As long as the `SegmentMeta` lives, the files associated with the
+    /// `SegmentMeta` are guaranteed to not be garbage collected, regardless of
+    /// whether the segment is recorded as part of the index or not.
+    pub fn new_segment_meta(&self, segment_id: SegmentId, max_doc: u32) -> SegmentMeta {
+        self.inventory.new_segment_meta(segment_id, max_doc)
    }

    /// Open the index using the provided directory
@@ -354,6 +369,11 @@ impl Index {
            .map(SegmentMeta::id)
            .collect())
    }
+
+    /// Returns the set of corrupted files
+    pub fn validate_checksum(&self) -> Result<HashSet<PathBuf>> {
+        self.directory.list_damaged().map_err(Into::into)
+    }
 }

 impl fmt::Debug for Index {
@@ -459,13 +479,13 @@ mod tests {

        use super::*;
        use std::path::PathBuf;
-        use tempdir::TempDir;
+        use tempfile::TempDir;

        #[test]
        fn test_index_on_commit_reload_policy_mmap() {
            let schema = throw_away_schema();
            let field = schema.get_field("num_likes").unwrap();
-            let tempdir = TempDir::new("index").unwrap();
+            let tempdir = TempDir::new().unwrap();
            let tempdir_path = PathBuf::from(tempdir.path());
            let index = Index::create_in_dir(&tempdir_path, schema).unwrap();
            let mut writer = index.writer_with_num_threads(1, 3_000_000).unwrap();
@@ -504,7 +524,7 @@ mod tests {
        fn test_index_on_commit_reload_policy_different_directories() {
            let schema = throw_away_schema();
            let field = schema.get_field("num_likes").unwrap();
-            let tempdir = TempDir::new("index").unwrap();
+            let tempdir = TempDir::new().unwrap();
            let tempdir_path = PathBuf::from(tempdir.path());
            let write_index = Index::create_in_dir(&tempdir_path, schema).unwrap();
            let read_index = Index::open_in_dir(&tempdir_path).unwrap();
@@ -581,5 +601,4 @@ mod tests {
        assert_eq!(searcher.num_docs(), 8_000);
        assert!(mem_right_after_merge_finished < mem_right_after_commit);
    }
-
 }
--- a/src/core/index_meta.rs
+++ b/src/core/index_meta.rs
@@ -30,7 +30,6 @@ impl SegmentMetaInventory {
            .collect::<Vec<_>>()
    }

-    #[doc(hidden)]
    pub fn new_segment_meta(&self, segment_id: SegmentId, max_doc: u32) -> SegmentMeta {
        let inner = InnerSegmentMeta {
            segment_id,
--- a/src/core/segment_id.rs
+++ b/src/core/segment_id.rs
@@ -4,6 +4,8 @@ use uuid::Uuid;

 #[cfg(test)]
 use once_cell::sync::Lazy;
+use std::error::Error;
+use std::str::FromStr;
 #[cfg(test)]
 use std::sync::atomic;

@@ -52,15 +54,51 @@ impl SegmentId {
    /// and the rest is random.
    ///
    /// Picking the first 8 chars is ok to identify
-    /// segments in a display message.
+    /// segments in a display message (e.g. a5c4dfcb).
    pub fn short_uuid_string(&self) -> String {
        (&self.0.to_simple_ref().to_string()[..8]).to_string()
    }

    /// Returns a segment uuid string.
+    ///
+    /// It consists in 32 lowercase hexadecimal chars
+    /// (e.g. a5c4dfcbdfe645089129e308e26d5523)
    pub fn uuid_string(&self) -> String {
        self.0.to_simple_ref().to_string()
    }
+
+    /// Build a `SegmentId` string from the full uuid string.
+    ///
+    /// E.g. "a5c4dfcbdfe645089129e308e26d5523"
+    pub fn from_uuid_string(uuid_string: &str) -> Result<SegmentId, SegmentIdParseError> {
+        FromStr::from_str(uuid_string)
+    }
+}
+
+/// Error type used when parsing a `SegmentId` from a string fails.
+pub struct SegmentIdParseError(uuid::parser::ParseError);
+
+impl Error for SegmentIdParseError {}
+
+impl fmt::Debug for SegmentIdParseError {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        self.0.fmt(f)
+    }
+}
+
+impl fmt::Display for SegmentIdParseError {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        self.0.fmt(f)
+    }
+}
+
+impl FromStr for SegmentId {
+    type Err = SegmentIdParseError;
+
+    fn from_str(uuid_string: &str) -> Result<Self, SegmentIdParseError> {
+        let uuid = Uuid::parse_str(uuid_string).map_err(SegmentIdParseError)?;
+        Ok(SegmentId(uuid))
+    }
 }

 impl fmt::Debug for SegmentId {
@@ -80,3 +118,18 @@ impl Ord for SegmentId {
        self.0.as_bytes().cmp(other.0.as_bytes())
    }
 }
+
+#[cfg(test)]
+mod tests {
+    use super::SegmentId;
+
+    #[test]
+    fn test_to_uuid_string() {
+        let full_uuid = "a5c4dfcbdfe645089129e308e26d5523";
+        let segment_id = SegmentId::from_uuid_string(full_uuid).unwrap();
+        assert_eq!(segment_id.uuid_string(), full_uuid);
+        assert_eq!(segment_id.short_uuid_string(), "a5c4dfcb");
+        // one extra char
+        assert!(SegmentId::from_uuid_string("a5c4dfcbdfe645089129e308e26d5523b").is_err());
+    }
+}
--- a/src/directory/directory.rs
+++ b/src/directory/directory.rs
@@ -48,14 +48,14 @@ impl RetryPolicy {
 ///
 /// It is transparently associated to a lock file, that gets deleted
 /// on `Drop.` The lock is released automatically on `Drop`.
-pub struct DirectoryLock(Box<dyn Drop + Send + Sync + 'static>);
+pub struct DirectoryLock(Box<dyn Send + Sync + 'static>);

 struct DirectoryLockGuard {
    directory: Box<dyn Directory>,
    path: PathBuf,
 }

-impl<T: Drop + Send + Sync + 'static> From<Box<T>> for DirectoryLock {
+impl<T: Send + Sync + 'static> From<Box<T>> for DirectoryLock {
    fn from(underlying: Box<T>) -> Self {
        DirectoryLock(underlying)
    }
@@ -118,6 +118,8 @@ pub trait Directory: DirectoryClone + fmt::Debug + Send + Sync + 'static {
    ///
    /// Specifically, subsequent writes or flushes should
    /// have no effect on the returned `ReadOnlySource` object.
+    ///
+    /// You should only use this to read files create with [`open_write`]
    fn open_read(&self, path: &Path) -> result::Result<ReadOnlySource, OpenReadError>;

    /// Removes a file
@@ -157,6 +159,8 @@ pub trait Directory: DirectoryClone + fmt::Debug + Send + Sync + 'static {
    /// atomic_write.
    ///
    /// This should only be used for small files.
+    ///
+    /// You should only use this to read files create with [`atomic_write`]
    fn atomic_read(&self, path: &Path) -> Result<Vec<u8>, OpenReadError>;

    /// Atomically replace the content of a file with data.
--- a/src/directory/footer.rs
+++ b/src/directory/footer.rs
@@ -0,0 +1,213 @@
+use crate::directory::read_only_source::ReadOnlySource;
+use crate::directory::{AntiCallToken, TerminatingWrite};
+use byteorder::{ByteOrder, LittleEndian};
+use crc32fast::Hasher;
+use std::io;
+use std::io::Write;
+
+const COMMON_FOOTER_SIZE: usize = 4 * 5;
+
+#[derive(Debug, Clone, PartialEq)]
+pub struct Footer {
+    pub tantivy_version: (u32, u32, u32),
+    pub meta: String,
+    pub versioned_footer: VersionedFooter,
+}
+
+impl Footer {
+    pub fn new(versioned_footer: VersionedFooter) -> Self {
+        let tantivy_version = (
+            env!("CARGO_PKG_VERSION_MAJOR").parse().unwrap(),
+            env!("CARGO_PKG_VERSION_MINOR").parse().unwrap(),
+            env!("CARGO_PKG_VERSION_PATCH").parse().unwrap(),
+        );
+        Footer {
+            tantivy_version,
+            meta: format!(
+                "tantivy {}.{}.{}, index v{}",
+                tantivy_version.0,
+                tantivy_version.1,
+                tantivy_version.2,
+                versioned_footer.version()
+            ),
+            versioned_footer,
+        }
+    }
+
+    pub fn to_bytes(&self) -> Vec<u8> {
+        let mut res = self.versioned_footer.to_bytes();
+        res.extend_from_slice(self.meta.as_bytes());
+        let len = res.len();
+        res.resize(len + COMMON_FOOTER_SIZE, 0);
+        let mut common_footer = &mut res[len..];
+        LittleEndian::write_u32(&mut common_footer, self.meta.len() as u32);
+        LittleEndian::write_u32(&mut common_footer[4..], self.tantivy_version.0);
+        LittleEndian::write_u32(&mut common_footer[8..], self.tantivy_version.1);
+        LittleEndian::write_u32(&mut common_footer[12..], self.tantivy_version.2);
+        LittleEndian::write_u32(&mut common_footer[16..], (len + COMMON_FOOTER_SIZE) as u32);
+        res
+    }
+
+    pub fn from_bytes(data: &[u8]) -> Result<Self, io::Error> {
+        let len = data.len();
+        if len < COMMON_FOOTER_SIZE + 4 {
+            // 4 bytes for index version, stored in versioned footer
+            return Err(io::Error::new(
+                    io::ErrorKind::UnexpectedEof,
+                    format!("File corrupted. The footer len must be over 24, while the entire file len is {}", len)
+                    )
+            );
+        }
+
+        let size = LittleEndian::read_u32(&data[len - 4..]) as usize;
+        if len < size as usize {
+            return Err(io::Error::new(
+                io::ErrorKind::UnexpectedEof,
+                format!(
+                    "File corrupted. The footer len is {}, while the entire file len is {}",
+                    size, len
+                ),
+            ));
+        }
+        let footer = &data[len - size as usize..];
+        let meta_len = LittleEndian::read_u32(&footer[size - 20..]) as usize;
+        let tantivy_major = LittleEndian::read_u32(&footer[size - 16..]);
+        let tantivy_minor = LittleEndian::read_u32(&footer[size - 12..]);
+        let tantivy_patch = LittleEndian::read_u32(&footer[size - 8..]);
+        Ok(Footer {
+            tantivy_version: (tantivy_major, tantivy_minor, tantivy_patch),
+            meta: String::from_utf8_lossy(&footer[size - meta_len - 20..size - 20]).into_owned(),
+            versioned_footer: VersionedFooter::from_bytes(&footer[..size - meta_len - 20])?,
+        })
+    }
+
+    pub fn extract_footer(source: ReadOnlySource) -> Result<(Footer, ReadOnlySource), io::Error> {
+        let footer = Footer::from_bytes(source.as_slice())?;
+        let reader = source.slice_to(source.as_slice().len() - footer.size());
+        Ok((footer, reader))
+    }
+
+    pub fn size(&self) -> usize {
+        self.versioned_footer.size() as usize + self.meta.len() + 20
+    }
+}
+
+#[derive(Debug, Clone, PartialEq)]
+pub enum VersionedFooter {
+    UnknownVersion { version: u32, size: u32 },
+    V0(u32), // crc
+}
+
+impl VersionedFooter {
+    pub fn to_bytes(&self) -> Vec<u8> {
+        match self {
+            VersionedFooter::V0(crc) => {
+                let mut res = vec![0; 8];
+                LittleEndian::write_u32(&mut res, 0);
+                LittleEndian::write_u32(&mut res[4..], *crc);
+                res
+            }
+            VersionedFooter::UnknownVersion { .. } => {
+                panic!("Unsupported index should never get serialized");
+            }
+        }
+    }
+
+    pub fn from_bytes(footer: &[u8]) -> Result<Self, io::Error> {
+        assert!(footer.len() >= 4);
+        let version = LittleEndian::read_u32(footer);
+        match version {
+            0 => {
+                if footer.len() == 8 {
+                    Ok(VersionedFooter::V0(LittleEndian::read_u32(&footer[4..])))
+                } else {
+                    Err(io::Error::new(
+                        io::ErrorKind::UnexpectedEof,
+                        format!(
+                            "File corrupted. The versioned footer len is {}, while it should be 8",
+                            footer.len()
+                        ),
+                    ))
+                }
+            }
+            version => Ok(VersionedFooter::UnknownVersion {
+                version,
+                size: footer.len() as u32,
+            }),
+        }
+    }
+
+    pub fn size(&self) -> u32 {
+        match self {
+            VersionedFooter::V0(_) => 8,
+            VersionedFooter::UnknownVersion { size, .. } => *size,
+        }
+    }
+
+    pub fn version(&self) -> u32 {
+        match self {
+            VersionedFooter::V0(_) => 0,
+            VersionedFooter::UnknownVersion { version, .. } => *version,
+        }
+    }
+
+    pub fn crc(&self) -> Option<u32> {
+        match self {
+            VersionedFooter::V0(crc) => Some(*crc),
+            VersionedFooter::UnknownVersion { .. } => None,
+        }
+    }
+}
+
+pub(crate) struct FooterProxy<W: TerminatingWrite> {
+    /// always Some except after terminate call
+    hasher: Option<Hasher>,
+    /// always Some except after terminate call
+    writer: Option<W>,
+}
+
+impl<W: TerminatingWrite> FooterProxy<W> {
+    pub fn new(writer: W) -> Self {
+        FooterProxy {
+            hasher: Some(Hasher::new()),
+            writer: Some(writer),
+        }
+    }
+}
+
+impl<W: TerminatingWrite> Write for FooterProxy<W> {
+    fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
+        let count = self.writer.as_mut().unwrap().write(buf)?;
+        self.hasher.as_mut().unwrap().update(&buf[..count]);
+        Ok(count)
+    }
+
+    fn flush(&mut self) -> io::Result<()> {
+        self.writer.as_mut().unwrap().flush()
+    }
+}
+
+impl<W: TerminatingWrite> TerminatingWrite for FooterProxy<W> {
+    fn terminate_ref(&mut self, _: AntiCallToken) -> io::Result<()> {
+        let crc = self.hasher.take().unwrap().finalize();
+
+        let footer = Footer::new(VersionedFooter::V0(crc)).to_bytes();
+        let mut writer = self.writer.take().unwrap();
+        writer.write_all(&footer)?;
+        writer.terminate()
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use crate::directory::footer::{Footer, VersionedFooter};
+
+    #[test]
+    fn test_serialize_deserialize_footer() {
+        let crc = 123456;
+        let footer = Footer::new(VersionedFooter::V0(crc));
+        let footer_bytes = footer.to_bytes();
+
+        assert_eq!(Footer::from_bytes(&footer_bytes).unwrap(), footer);
+    }
+}
--- a/src/directory/managed_directory.rs
+++ b/src/directory/managed_directory.rs
@@ -1,5 +1,6 @@
 use crate::core::MANAGED_FILEPATH;
 use crate::directory::error::{DeleteError, IOError, LockError, OpenReadError, OpenWriteError};
+use crate::directory::footer::{Footer, FooterProxy};
 use crate::directory::DirectoryLock;
 use crate::directory::Lock;
 use crate::directory::META_LOCK;
@@ -8,6 +9,7 @@ use crate::directory::{WatchCallback, WatchHandle};
 use crate::error::DataCorruption;
 use crate::Directory;
 use crate::Result;
+use crc32fast::Hasher;
 use serde_json;
 use std::collections::HashSet;
 use std::io;
@@ -207,17 +209,59 @@ impl ManagedDirectory {
        }
        Ok(())
    }
+
+    /// Verify checksum of a managed file
+    pub fn validate_checksum(&self, path: &Path) -> result::Result<bool, OpenReadError> {
+        let reader = self.directory.open_read(path)?;
+        let (footer, data) = Footer::extract_footer(reader)
+            .map_err(|err| IOError::with_path(path.to_path_buf(), err))?;
+        let mut hasher = Hasher::new();
+        hasher.update(data.as_slice());
+        let crc = hasher.finalize();
+        Ok(footer
+            .versioned_footer
+            .crc()
+            .map(|v| v == crc)
+            .unwrap_or(false))
+    }
+
+    /// List files for which checksum does not match content
+    pub fn list_damaged(&self) -> result::Result<HashSet<PathBuf>, OpenReadError> {
+        let mut hashset = HashSet::new();
+        let managed_paths = self
+            .meta_informations
+            .read()
+            .expect("Managed directory rlock poisoned in list damaged.")
+            .managed_paths
+            .clone();
+
+        for path in managed_paths.into_iter() {
+            if !self.validate_checksum(&path)? {
+                hashset.insert(path);
+            }
+        }
+        Ok(hashset)
+    }
 }

 impl Directory for ManagedDirectory {
    fn open_read(&self, path: &Path) -> result::Result<ReadOnlySource, OpenReadError> {
-        self.directory.open_read(path)
+        let read_only_source = self.directory.open_read(path)?;
+        let (_footer, reader) = Footer::extract_footer(read_only_source)
+            .map_err(|err| IOError::with_path(path.to_path_buf(), err))?;
+        Ok(reader)
    }

    fn open_write(&mut self, path: &Path) -> result::Result<WritePtr, OpenWriteError> {
        self.register_file_as_managed(path)
            .map_err(|e| IOError::with_path(path.to_owned(), e))?;
-        self.directory.open_write(path)
+        Ok(io::BufWriter::new(Box::new(FooterProxy::new(
+            self.directory
+                .open_write(path)?
+                .into_inner()
+                .map_err(|_| ())
+                .expect("buffer should be empty"),
+        ))))
    }

    fn atomic_write(&mut self, path: &Path, data: &[u8]) -> io::Result<()> {
@@ -259,15 +303,16 @@ impl Clone for ManagedDirectory {
 #[cfg(test)]
 mod tests_mmap_specific {

-    use crate::directory::{Directory, ManagedDirectory, MmapDirectory};
+    use crate::directory::{Directory, ManagedDirectory, MmapDirectory, TerminatingWrite};
    use std::collections::HashSet;
+    use std::fs::OpenOptions;
    use std::io::Write;
    use std::path::{Path, PathBuf};
-    use tempdir::TempDir;
+    use tempfile::TempDir;

    #[test]
    fn test_managed_directory() {
-        let tempdir = TempDir::new("tantivy-test").unwrap();
+        let tempdir = TempDir::new().unwrap();
        let tempdir_path = PathBuf::from(tempdir.path());

        let test_path1: &'static Path = Path::new("some_path_for_test");
@@ -275,8 +320,8 @@ mod tests_mmap_specific {
        {
            let mmap_directory = MmapDirectory::open(&tempdir_path).unwrap();
            let mut managed_directory = ManagedDirectory::wrap(mmap_directory).unwrap();
-            let mut write_file = managed_directory.open_write(test_path1).unwrap();
-            write_file.flush().unwrap();
+            let write_file = managed_directory.open_write(test_path1).unwrap();
+            write_file.terminate().unwrap();
            managed_directory
                .atomic_write(test_path2, &[0u8, 1u8])
                .unwrap();
@@ -304,15 +349,15 @@ mod tests_mmap_specific {
    fn test_managed_directory_gc_while_mmapped() {
        let test_path1: &'static Path = Path::new("some_path_for_test");

-        let tempdir = TempDir::new("index").unwrap();
+        let tempdir = TempDir::new().unwrap();
        let tempdir_path = PathBuf::from(tempdir.path());
        let living_files = HashSet::new();

        let mmap_directory = MmapDirectory::open(&tempdir_path).unwrap();
        let mut managed_directory = ManagedDirectory::wrap(mmap_directory).unwrap();
-        managed_directory
-            .atomic_write(test_path1, &vec![0u8, 1u8])
-            .unwrap();
+        let mut write = managed_directory.open_write(test_path1).unwrap();
+        write.write_all(&[0u8, 1u8]).unwrap();
+        write.terminate().unwrap();
        assert!(managed_directory.exists(test_path1));

        let _mmap_read = managed_directory.open_read(test_path1).unwrap();
@@ -331,4 +376,38 @@ mod tests_mmap_specific {
        }
    }

+    #[test]
+    fn test_checksum() {
+        let test_path1: &'static Path = Path::new("some_path_for_test");
+        let test_path2: &'static Path = Path::new("other_test_path");
+
+        let tempdir = TempDir::new().unwrap();
+        let tempdir_path = PathBuf::from(tempdir.path());
+
+        let mmap_directory = MmapDirectory::open(&tempdir_path).unwrap();
+        let mut managed_directory = ManagedDirectory::wrap(mmap_directory).unwrap();
+        let mut write = managed_directory.open_write(test_path1).unwrap();
+        write.write_all(&[0u8, 1u8]).unwrap();
+        write.terminate().unwrap();
+
+        let mut write = managed_directory.open_write(test_path2).unwrap();
+        write.write_all(&[3u8, 4u8, 5u8]).unwrap();
+        write.terminate().unwrap();
+
+        assert!(managed_directory.list_damaged().unwrap().is_empty());
+
+        let mut corrupted_path = tempdir_path.clone();
+        corrupted_path.push(test_path2);
+        let mut file = OpenOptions::new()
+            .write(true)
+            .open(&corrupted_path)
+            .unwrap();
+        file.write_all(&[255u8]).unwrap();
+        file.flush().unwrap();
+        drop(file);
+
+        let damaged = managed_directory.list_damaged().unwrap();
+        assert_eq!(damaged.len(), 1);
+        assert!(damaged.contains(test_path2));
+    }
 }
--- a/src/directory/mmap_directory.rs
+++ b/src/directory/mmap_directory.rs
@@ -11,6 +11,7 @@ use crate::directory::error::{
    DeleteError, IOError, OpenDirectoryError, OpenReadError, OpenWriteError,
 };
 use crate::directory::read_only_source::BoxedData;
+use crate::directory::AntiCallToken;
 use crate::directory::Directory;
 use crate::directory::DirectoryLock;
 use crate::directory::Lock;
@@ -18,7 +19,7 @@ use crate::directory::ReadOnlySource;
 use crate::directory::WatchCallback;
 use crate::directory::WatchCallbackList;
 use crate::directory::WatchHandle;
-use crate::directory::WritePtr;
+use crate::directory::{TerminatingWrite, WritePtr};
 use atomicwrites;
 use memmap::Mmap;
 use std::collections::HashMap;
@@ -36,7 +37,7 @@ use std::sync::Mutex;
 use std::sync::RwLock;
 use std::sync::Weak;
 use std::thread;
-use tempdir::TempDir;
+use tempfile::TempDir;

 /// Create a default io error given a string.
 pub(crate) fn make_io_err(msg: String) -> io::Error {
@@ -141,42 +142,28 @@ impl MmapCache {
    }
 }

-struct InnerWatcherWrapper {
-    _watcher: Mutex<notify::RecommendedWatcher>,
-    watcher_router: WatchCallbackList,
-}
-
-impl InnerWatcherWrapper {
-    pub fn new(path: &Path) -> Result<(Self, Receiver<notify::RawEvent>), notify::Error> {
-        let (tx, watcher_recv): (Sender<RawEvent>, Receiver<RawEvent>) = channel();
-        // We need to initialize the
-        let mut watcher = notify::raw_watcher(tx)?;
-        watcher.watch(path, RecursiveMode::Recursive)?;
-        let inner = InnerWatcherWrapper {
-            _watcher: Mutex::new(watcher),
-            watcher_router: Default::default(),
-        };
-        Ok((inner, watcher_recv))
-    }
-}
-
-#[derive(Clone)]
 struct WatcherWrapper {
-    inner: Arc<InnerWatcherWrapper>,
+    _watcher: Mutex<notify::RecommendedWatcher>,
+    watcher_router: Arc<WatchCallbackList>,
 }

 impl WatcherWrapper {
    pub fn new(path: &Path) -> Result<Self, OpenDirectoryError> {
-        let (inner, watcher_recv) = InnerWatcherWrapper::new(path).map_err(|err| match err {
-            notify::Error::PathNotFound => OpenDirectoryError::DoesNotExist(path.to_owned()),
-            _ => {
-                panic!("Unknown error while starting watching directory {:?}", path);
-            }
-        })?;
-        let watcher_wrapper = WatcherWrapper {
-            inner: Arc::new(inner),
-        };
-        let watcher_wrapper_clone = watcher_wrapper.clone();
+        let (tx, watcher_recv): (Sender<RawEvent>, Receiver<RawEvent>) = channel();
+        // We need to initialize the
+        let watcher = notify::raw_watcher(tx)
+            .and_then(|mut watcher| {
+                watcher.watch(path, RecursiveMode::Recursive)?;
+                Ok(watcher)
+            })
+            .map_err(|err| match err {
+                notify::Error::PathNotFound => OpenDirectoryError::DoesNotExist(path.to_owned()),
+                _ => {
+                    panic!("Unknown error while starting watching directory {:?}", path);
+                }
+            })?;
+        let watcher_router: Arc<WatchCallbackList> = Default::default();
+        let watcher_router_clone = watcher_router.clone();
        thread::Builder::new()
            .name("meta-file-watch-thread".to_string())
            .spawn(move || {
@@ -187,7 +174,7 @@ impl WatcherWrapper {
                            // We might want to be more accurate than this at one point.
                            if let Some(filename) = changed_path.file_name() {
                                if filename == *META_FILEPATH {
-                                    watcher_wrapper_clone.inner.watcher_router.broadcast();
+                                    watcher_router_clone.broadcast();
                                }
                            }
                        }
@@ -200,13 +187,15 @@ impl WatcherWrapper {
                        }
                    }
                }
-            })
-            .expect("Failed to spawn thread to watch meta.json");
-        Ok(watcher_wrapper)
+            })?;
+        Ok(WatcherWrapper {
+            _watcher: Mutex::new(watcher),
+            watcher_router,
+        })
    }

    pub fn watch(&mut self, watch_callback: WatchCallback) -> WatchHandle {
-        self.inner.watcher_router.subscribe(watch_callback)
+        self.watcher_router.subscribe(watch_callback)
    }
 }

@@ -265,7 +254,7 @@ impl MmapDirectoryInner {
            }
        }
        if let Some(watch_wrapper) = self.watcher.write().unwrap().as_mut() {
-            return Ok(watch_wrapper.watch(watch_callback));
+            Ok(watch_wrapper.watch(watch_callback))
        } else {
            unreachable!("At this point, watch wrapper is supposed to be initialized");
        }
@@ -294,7 +283,7 @@ impl MmapDirectory {
    /// This is mostly useful to test the MmapDirectory itself.
    /// For your unit tests, prefer the RAMDirectory.
    pub fn create_from_tempdir() -> Result<MmapDirectory, OpenDirectoryError> {
-        let tempdir = TempDir::new("index").map_err(OpenDirectoryError::IoError)?;
+        let tempdir = TempDir::new().map_err(OpenDirectoryError::IoError)?;
        let tempdir_path = PathBuf::from(tempdir.path());
        MmapDirectory::new(tempdir_path, Some(tempdir))
    }
@@ -412,6 +401,12 @@ impl Seek for SafeFileWriter {
    }
 }

+impl TerminatingWrite for SafeFileWriter {
+    fn terminate_ref(&mut self, _: AntiCallToken) -> io::Result<()> {
+        self.flush()
+    }
+}
+
 impl Directory for MmapDirectory {
    fn open_read(&self, path: &Path) -> result::Result<ReadOnlySource, OpenReadError> {
        debug!("Open Read {:?}", path);
@@ -642,7 +637,7 @@ mod tests {
    fn test_watch_wrapper() {
        let counter: Arc<AtomicUsize> = Default::default();
        let counter_clone = counter.clone();
-        let tmp_dir: TempDir = tempdir::TempDir::new("test_watch_wrapper").unwrap();
+        let tmp_dir = tempfile::TempDir::new().unwrap();
        let tmp_dirpath = tmp_dir.path().to_owned();
        let mut watch_wrapper = WatcherWrapper::new(&tmp_dirpath).unwrap();
        let tmp_file = tmp_dirpath.join("coucou");
--- a/src/directory/mod.rs
+++ b/src/directory/mod.rs
@@ -9,6 +9,7 @@ mod mmap_directory;

 mod directory;
 mod directory_lock;
+mod footer;
 mod managed_directory;
 mod ram_directory;
 mod read_only_source;
@@ -24,18 +25,49 @@ pub use self::ram_directory::RAMDirectory;
 pub use self::read_only_source::ReadOnlySource;
 pub(crate) use self::watch_event_router::WatchCallbackList;
 pub use self::watch_event_router::{WatchCallback, WatchHandle};
-use std::io::{BufWriter, Write};
+use std::io::{self, BufWriter, Write};

 #[cfg(feature = "mmap")]
 pub use self::mmap_directory::MmapDirectory;

 pub use self::managed_directory::ManagedDirectory;

+/// Struct used to prevent from calling [`terminate_ref`](trait.TerminatingWrite#method.terminate_ref) directly
+pub struct AntiCallToken(());
+
+/// Trait used to indicate when no more write need to be done on a writer
+pub trait TerminatingWrite: Write {
+    /// Indicate that the writer will no longer be used. Internally call terminate_ref.
+    fn terminate(mut self) -> io::Result<()>
+    where
+        Self: Sized,
+    {
+        self.terminate_ref(AntiCallToken(()))
+    }
+
+    /// You should implement this function to define custom behavior.
+    /// This function should flush any buffer it may hold.
+    fn terminate_ref(&mut self, _: AntiCallToken) -> io::Result<()>;
+}
+
+impl<W: TerminatingWrite + ?Sized> TerminatingWrite for Box<W> {
+    fn terminate_ref(&mut self, token: AntiCallToken) -> io::Result<()> {
+        self.as_mut().terminate_ref(token)
+    }
+}
+
+impl<W: TerminatingWrite> TerminatingWrite for BufWriter<W> {
+    fn terminate_ref(&mut self, a: AntiCallToken) -> io::Result<()> {
+        self.flush()?;
+        self.get_mut().terminate_ref(a)
+    }
+}
+
 /// Write object for Directory.
 ///
 /// `WritePtr` are required to implement both Write
 /// and Seek.
-pub type WritePtr = BufWriter<Box<dyn Write>>;
+pub type WritePtr = BufWriter<Box<dyn TerminatingWrite>>;

 #[cfg(test)]
 mod tests;
--- a/src/directory/ram_directory.rs
+++ b/src/directory/ram_directory.rs
@@ -1,8 +1,9 @@
 use crate::core::META_FILEPATH;
 use crate::directory::error::{DeleteError, OpenReadError, OpenWriteError};
+use crate::directory::AntiCallToken;
 use crate::directory::WatchCallbackList;
-use crate::directory::WritePtr;
 use crate::directory::{Directory, ReadOnlySource, WatchCallback, WatchHandle};
+use crate::directory::{TerminatingWrite, WritePtr};
 use fail::fail_point;
 use std::collections::HashMap;
 use std::fmt;
@@ -71,6 +72,12 @@ impl Write for VecWriter {
    }
 }

+impl TerminatingWrite for VecWriter {
+    fn terminate_ref(&mut self, _: AntiCallToken) -> io::Result<()> {
+        self.flush()
+    }
+}
+
 #[derive(Default)]
 struct InnerDirectory {
    fs: HashMap<PathBuf, ReadOnlySource>,
@@ -177,7 +184,7 @@ impl Directory for RAMDirectory {
    fn atomic_write(&mut self, path: &Path, data: &[u8]) -> io::Result<()> {
        fail_point!("RAMDirectory::atomic_write", |msg| Err(io::Error::new(
            io::ErrorKind::Other,
-            msg.unwrap_or("Undefined".to_string())
+            msg.unwrap_or_else(|| "Undefined".to_string())
        )));
        let path_buf = PathBuf::from(path);

--- a/src/directory/tests.rs
+++ b/src/directory/tests.rs
@@ -127,7 +127,7 @@ fn test_watch(directory: &mut dyn Directory) {
        assert!(directory
            .atomic_write(Path::new("meta.json"), b"random_test_data_2")
            .is_ok());
-        for _ in 0..100 {
+        for _ in 0..1_000 {
            if counter.load(Ordering::SeqCst) > i {
                break;
            }
--- a/src/directory/watch_event_router.rs
+++ b/src/directory/watch_event_router.rs
@@ -152,5 +152,4 @@ mod tests {
        thread::sleep(Duration::from_millis(WAIT_TIME));
        assert_eq!(2, counter.load(Ordering::SeqCst));
    }
-
 }
--- a/src/fastfield/mod.rs
+++ b/src/fastfield/mod.rs
@@ -429,7 +429,6 @@ mod tests {
            }
        }
    }
-
 }

 #[cfg(all(test, feature = "unstable"))]
@@ -437,9 +436,9 @@ mod bench {
    use super::tests::FIELD;
    use super::tests::{generate_permutation, SCHEMA};
    use super::*;
-    use common::CompositeFile;
-    use directory::{Directory, RAMDirectory, WritePtr};
-    use fastfield::FastFieldReader;
+    use crate::common::CompositeFile;
+    use crate::directory::{Directory, RAMDirectory, WritePtr};
+    use crate::fastfield::FastFieldReader;
    use std::collections::HashMap;
    use std::path::Path;
    use test::{self, Bencher};
@@ -537,5 +536,4 @@ mod bench {
            });
        }
    }
-
 }
--- a/src/fastfield/multivalued/writer.rs
+++ b/src/fastfield/multivalued/writer.rs
@@ -5,8 +5,8 @@ use crate::postings::UnorderedTermId;
 use crate::schema::{Document, Field};
 use crate::termdict::TermOrdinal;
 use crate::DocId;
+use fnv::FnvHashMap;
 use itertools::Itertools;
-use std::collections::HashMap;
 use std::io;

 /// Writer for multi-valued (as in, more than one value per document)
@@ -102,7 +102,7 @@ impl MultiValueIntFastFieldWriter {
    pub fn serialize(
        &self,
        serializer: &mut FastFieldSerializer,
-        mapping_opt: Option<&HashMap<UnorderedTermId, TermOrdinal>>,
+        mapping_opt: Option<&FnvHashMap<UnorderedTermId, TermOrdinal>>,
    ) -> io::Result<()> {
        {
            // writing the offset index
--- a/src/fastfield/readers.rs
+++ b/src/fastfield/readers.rs
@@ -59,8 +59,7 @@ impl FastFieldReaders {
            fast_bytes: Default::default(),
            fast_fields_composite: fast_fields_composite.clone(),
        };
-        for (field_id, field_entry) in schema.fields().iter().enumerate() {
-            let field = Field(field_id as u32);
+        for (field, field_entry) in schema.fields() {
            let field_type = field_entry.field_type();
            if field_type == &FieldType::Bytes {
                let idx_reader = fast_fields_composite
--- a/src/fastfield/writer.rs
+++ b/src/fastfield/writer.rs
@@ -6,6 +6,7 @@ use crate::fastfield::{BytesFastFieldWriter, FastFieldSerializer};
 use crate::postings::UnorderedTermId;
 use crate::schema::{Cardinality, Document, Field, FieldType, Schema};
 use crate::termdict::TermOrdinal;
+use fnv::FnvHashMap;
 use std::collections::HashMap;
 use std::io;

@@ -23,15 +24,16 @@ impl FastFieldsWriter {
        let mut multi_values_writers = Vec::new();
        let mut bytes_value_writers = Vec::new();

-        for (field_id, field_entry) in schema.fields().iter().enumerate() {
-            let field = Field(field_id as u32);
+        for (field, field_entry) in schema.fields() {
            let default_value = match *field_entry.field_type() {
                FieldType::I64(_) => common::i64_to_u64(0i64),
                FieldType::F64(_) => common::f64_to_u64(0.0f64),
                _ => 0u64,
            };
            match *field_entry.field_type() {
-                FieldType::I64(ref int_options) | FieldType::U64(ref int_options) | FieldType::F64(ref int_options) => {
+                FieldType::I64(ref int_options)
+                | FieldType::U64(ref int_options)
+                | FieldType::F64(ref int_options) => {
                    match int_options.get_fastfield_cardinality() {
                        Some(Cardinality::SingleValue) => {
                            let mut fast_field_writer = IntFastFieldWriter::new(field);
@@ -114,7 +116,7 @@ impl FastFieldsWriter {
    pub fn serialize(
        &self,
        serializer: &mut FastFieldSerializer,
-        mapping: &HashMap<Field, HashMap<UnorderedTermId, TermOrdinal>>,
+        mapping: &HashMap<Field, FnvHashMap<UnorderedTermId, TermOrdinal>>,
    ) -> io::Result<()> {
        for field_writer in &self.single_value_writers {
            field_writer.serialize(serializer)?;
--- a/src/fieldnorm/writer.rs
+++ b/src/fieldnorm/writer.rs
@@ -22,11 +22,14 @@ impl FieldNormsWriter {
    pub(crate) fn fields_with_fieldnorm(schema: &Schema) -> Vec<Field> {
        schema
            .fields()
-            .iter()
-            .enumerate()
-            .filter(|&(_, field_entry)| field_entry.is_indexed())
-            .map(|(field, _)| Field(field as u32))
-            .collect::<Vec<Field>>()
+            .filter_map(|(field, field_entry)| {
+                if field_entry.is_indexed() {
+                    Some(field)
+                } else {
+                    None
+                }
+            })
+            .collect::<Vec<_>>()
    }

    /// Initialize with state for tracking the field norm fields
@@ -35,7 +38,7 @@ impl FieldNormsWriter {
        let fields = FieldNormsWriter::fields_with_fieldnorm(schema);
        let max_field = fields
            .iter()
-            .map(|field| field.0)
+            .map(Field::field_id)
            .max()
            .map(|max_field_id| max_field_id as usize + 1)
            .unwrap_or(0);
@@ -50,8 +53,8 @@ impl FieldNormsWriter {
    ///
    /// Will extend with 0-bytes for documents that have not been seen.
    pub fn fill_up_to_max_doc(&mut self, max_doc: DocId) {
-        for &field in self.fields.iter() {
-            self.fieldnorms_buffer[field.0 as usize].resize(max_doc as usize, 0u8);
+        for field in self.fields.iter() {
+            self.fieldnorms_buffer[field.field_id() as usize].resize(max_doc as usize, 0u8);
        }
    }

@@ -64,7 +67,7 @@ impl FieldNormsWriter {
    /// * field     - the field being set
    /// * fieldnorm - the number of terms present in document `doc` in field `field`
    pub fn record(&mut self, doc: DocId, field: Field, fieldnorm: u32) {
-        let fieldnorm_buffer: &mut Vec<u8> = &mut self.fieldnorms_buffer[field.0 as usize];
+        let fieldnorm_buffer: &mut Vec<u8> = &mut self.fieldnorms_buffer[field.field_id() as usize];
        assert!(
            fieldnorm_buffer.len() <= doc as usize,
            "Cannot register a given fieldnorm twice"
@@ -77,7 +80,7 @@ impl FieldNormsWriter {
    /// Serialize the seen fieldnorm values to the serializer for all fields.
    pub fn serialize(&self, fieldnorms_serializer: &mut FieldNormsSerializer) -> io::Result<()> {
        for &field in self.fields.iter() {
-            let fieldnorm_values: &[u8] = &self.fieldnorms_buffer[field.0 as usize][..];
+            let fieldnorm_values: &[u8] = &self.fieldnorms_buffer[field.field_id() as usize][..];
            fieldnorms_serializer.serialize_field(field, fieldnorm_values)?;
        }
        Ok(())
--- a/src/indexer/delete_queue.rs
+++ b/src/indexer/delete_queue.rs
@@ -258,7 +258,7 @@ mod tests {
        let delete_queue = DeleteQueue::new();

        let make_op = |i: usize| {
-            let field = Field(1u32);
+            let field = Field::from_field_id(1u32);
            DeleteOperation {
                opstamp: i as u64,
                term: Term::from_field_u64(field, i as u64),
--- a/src/indexer/index_writer.rs
+++ b/src/indexer/index_writer.rs
@@ -8,6 +8,7 @@ use crate::core::SegmentId;
 use crate::core::SegmentMeta;
 use crate::core::SegmentReader;
 use crate::directory::DirectoryLock;
+use crate::directory::TerminatingWrite;
 use crate::docset::DocSet;
 use crate::error::TantivyError;
 use crate::fastfield::write_delete_bitset;
@@ -168,6 +169,7 @@ pub(crate) fn advance_deletes(
            segment = segment.with_delete_meta(num_deleted_docs as u32, target_opstamp);
            let mut delete_file = segment.open_write(SegmentComponent::DELETE)?;
            write_delete_bitset(&delete_bitset, &mut delete_file)?;
+            delete_file.terminate()?;
        }
    }
    segment_entry.set_meta(segment.meta().clone());
@@ -209,10 +211,7 @@ fn index_documents(
    assert!(num_docs > 0);

    let doc_opstamps: Vec<Opstamp> = segment_writer.finalize()?;
-    let segment_meta = segment
-        .index()
-        .inventory()
-        .new_segment_meta(segment_id, num_docs);
+    let segment_meta = segment.index().new_segment_meta(segment_id, num_docs);

    let last_docstamp: Opstamp = *(doc_opstamps.last().unwrap());

@@ -450,12 +449,10 @@ impl IndexWriter {
    /// by clearing and resubmitting necessary documents
    ///
    /// ```rust
-    /// #[macro_use]
-    /// extern crate tantivy;
-    /// use tantivy::query::QueryParser;
    /// use tantivy::collector::TopDocs;
+    /// use tantivy::query::QueryParser;
    /// use tantivy::schema::*;
-    /// use tantivy::Index;
+    /// use tantivy::{doc, Index};
    ///
    /// fn main() -> tantivy::Result<()> {
    ///     let mut schema_builder = Schema::builder();
@@ -761,7 +758,6 @@ mod tests {
    use crate::Index;
    use crate::ReloadPolicy;
    use crate::Term;
-    use fail;

    #[test]
    fn test_operations_group() {
@@ -1183,5 +1179,4 @@ mod tests {
        assert!(clear_again.is_ok());
        assert!(commit_again.is_ok());
    }
-
 }
--- a/src/indexer/merger.rs
+++ b/src/indexer/merger.rs
@@ -190,8 +190,7 @@ impl IndexMerger {
        fast_field_serializer: &mut FastFieldSerializer,
        mut term_ord_mappings: HashMap<Field, TermOrdinalMapping>,
    ) -> Result<()> {
-        for (field_id, field_entry) in self.schema.fields().iter().enumerate() {
-            let field = Field(field_id as u32);
+        for (field, field_entry) in self.schema.fields() {
            let field_type = field_entry.field_type();
            match *field_type {
                FieldType::HierarchicalFacet => {
@@ -649,15 +648,12 @@ impl IndexMerger {
        serializer: &mut InvertedIndexSerializer,
    ) -> Result<HashMap<Field, TermOrdinalMapping>> {
        let mut term_ordinal_mappings = HashMap::new();
-        for (field_ord, field_entry) in self.schema.fields().iter().enumerate() {
+        for (field, field_entry) in self.schema.fields() {
            if field_entry.is_indexed() {
-                let indexed_field = Field(field_ord as u32);
-                if let Some(term_ordinal_mapping) = self.write_postings_for_field(
-                    indexed_field,
-                    field_entry.field_type(),
-                    serializer,
-                )? {
-                    term_ordinal_mappings.insert(indexed_field, term_ordinal_mapping);
+                if let Some(term_ordinal_mapping) =
+                    self.write_postings_for_field(field, field_entry.field_type(), serializer)?
+                {
+                    term_ordinal_mappings.insert(field, term_ordinal_mapping);
                }
            }
        }
--- a/src/indexer/segment_register.rs
+++ b/src/indexer/segment_register.rs
@@ -134,5 +134,4 @@ mod tests {
        }
        assert_eq!(segment_ids(&segment_register), vec![segment_id_merged]);
    }
-
 }
--- a/src/indexer/segment_updater.rs
+++ b/src/indexer/segment_updater.rs
@@ -126,9 +126,7 @@ fn perform_merge(

    let num_docs = merger.write(segment_serializer)?;

-    let segment_meta = index
-        .inventory()
-        .new_segment_meta(merged_segment.id(), num_docs);
+    let segment_meta = index.new_segment_meta(merged_segment.id(), num_docs);

    let after_merge_segment_entry = SegmentEntry::new(segment_meta.clone(), delete_cursor, None);
    Ok(after_merge_segment_entry)
@@ -282,7 +280,7 @@ impl SegmentUpdater {
    fn list_files(&self) -> HashSet<PathBuf> {
        let mut files = HashSet::new();
        files.insert(META_FILEPATH.to_path_buf());
-        for segment_meta in self.0.index.inventory().all() {
+        for segment_meta in self.0.index.list_all_segment_metas() {
            files.extend(segment_meta.list_files());
        }
        files
--- a/src/indexer/segment_writer.rs
+++ b/src/indexer/segment_writer.rs
@@ -6,11 +6,11 @@ use crate::fieldnorm::FieldNormsWriter;
 use crate::indexer::segment_serializer::SegmentSerializer;
 use crate::postings::compute_table_size;
 use crate::postings::MultiFieldPostingsWriter;
-use crate::schema::FieldEntry;
 use crate::schema::FieldType;
 use crate::schema::Schema;
 use crate::schema::Term;
 use crate::schema::Value;
+use crate::schema::{Field, FieldEntry};
 use crate::tokenizer::BoxedTokenizer;
 use crate::tokenizer::FacetTokenizer;
 use crate::tokenizer::{TokenStream, Tokenizer};
@@ -49,7 +49,7 @@ pub struct SegmentWriter {
    fast_field_writers: FastFieldsWriter,
    fieldnorms_writer: FieldNormsWriter,
    doc_opstamps: Vec<Opstamp>,
-    tokenizers: Vec<Option<Box<dyn BoxedTokenizer>>>,
+    tokenizers: Vec<Option<BoxedTokenizer>>,
 }

 impl SegmentWriter {
@@ -70,12 +70,10 @@ impl SegmentWriter {
        let table_num_bits = initial_table_size(memory_budget)?;
        let segment_serializer = SegmentSerializer::for_segment(&mut segment)?;
        let multifield_postings = MultiFieldPostingsWriter::new(schema, table_num_bits);
-        let tokenizers =
-            schema
-                .fields()
-                .iter()
-                .map(FieldEntry::field_type)
-                .map(|field_type| match *field_type {
+        let tokenizers = schema
+            .fields()
+            .map(
+                |(_, field_entry): (Field, &FieldEntry)| match field_entry.field_type() {
                    FieldType::Str(ref text_options) => text_options
                        .get_indexing_options()
                        .and_then(|text_index_option| {
@@ -83,8 +81,9 @@ impl SegmentWriter {
                            segment.index().tokenizers().get(tokenizer_name)
                        }),
                    _ => None,
-                })
-                .collect();
+                },
+            )
+            .collect();
        Ok(SegmentWriter {
            max_doc: 0,
            multifield_postings,
@@ -160,7 +159,7 @@ impl SegmentWriter {
                }
                FieldType::Str(_) => {
                    let num_tokens = if let Some(ref mut tokenizer) =
-                        self.tokenizers[field.0 as usize]
+                        self.tokenizers[field.field_id() as usize]
                    {
                        let texts: Vec<&str> = field_values
                            .iter()
@@ -296,5 +295,4 @@ mod tests {
        assert_eq!(initial_table_size(10_000_000).unwrap(), 17);
        assert_eq!(initial_table_size(1_000_000_000).unwrap(), 19);
    }
-
 }
--- a/src/lib.rs
+++ b/src/lib.rs
@@ -3,7 +3,6 @@
 #![cfg_attr(feature = "cargo-clippy", allow(clippy::module_inception))]
 #![doc(test(attr(allow(unused_variables), deny(warnings))))]
 #![warn(missing_docs)]
-#![recursion_limit = "80"]

 //! # `tantivy`
 //!
@@ -11,26 +10,17 @@
 //! Think `Lucene`, but in Rust.
 //!
 //! ```rust
-
-//! # extern crate tempdir;
-//! #
-//! #[macro_use]
-//! extern crate tantivy;
-//!
-//! // ...
-//!
 //! # use std::path::Path;
-//! # use tempdir::TempDir;
-//! # use tantivy::Index;
-//! # use tantivy::schema::*;
-//! # use tantivy::{Score, DocAddress};
+//! # use tempfile::TempDir;
 //! # use tantivy::collector::TopDocs;
 //! # use tantivy::query::QueryParser;
+//! # use tantivy::schema::*;
+//! # use tantivy::{doc, DocAddress, Index, Score};
 //! #
 //! # fn main() {
 //! #     // Let's create a temporary directory for the
 //! #     // sake of this example
-//! #     if let Ok(dir) = TempDir::new("tantivy_example_dir") {
+//! #     if let Ok(dir) = TempDir::new() {
 //! #         run_example(dir.path()).unwrap();
 //! #         dir.close().unwrap();
 //! #     }
@@ -222,15 +212,13 @@ pub type Score = f32;
 pub type SegmentLocalId = u32;

 impl DocAddress {
-    /// Return the segment ordinal.
-    /// The segment ordinal is an id identifying the segment
-    /// hosting the document. It is only meaningful, in the context
-    /// of a searcher.
+    /// Return the segment ordinal id that identifies the segment
+    /// hosting the document in the `Searcher` it is called from.
    pub fn segment_ord(self) -> SegmentLocalId {
        self.0
    }

-    /// Return the segment local `DocId`
+    /// Return the segment-local `DocId`
    pub fn doc(self) -> DocId {
        self.1
    }
@@ -239,11 +227,11 @@ impl DocAddress {
 /// `DocAddress` contains all the necessary information
 /// to identify a document given a `Searcher` object.
 ///
-/// It consists in an id identifying its segment, and
-/// its segment-local `DocId`.
+/// It consists of an id identifying its segment, and
+/// a segment-local `DocId`.
 ///
 /// The id used for the segment is actually an ordinal
-/// in the list of segment hold by a `Searcher`.
+/// in the list of `Segment`s held by a `Searcher`.
 #[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
 pub struct DocAddress(pub SegmentLocalId, pub DocId);

--- a/src/macros.rs
+++ b/src/macros.rs
@@ -22,11 +22,9 @@
 ///
 /// # Example
 ///
-/// ```
-/// #[macro_use]
-/// extern crate tantivy;
-///
+/// ```rust
 /// use tantivy::schema::{Schema, TEXT, FAST};
+/// use tantivy::doc;
 ///
 /// //...
 ///
--- a/src/postings/compression/mod.rs
+++ b/src/postings/compression/mod.rs
@@ -274,13 +274,15 @@ pub mod tests {
 mod bench {

    use super::*;
+    use rand::rngs::StdRng;
+    use rand::Rng;
    use rand::SeedableRng;
-    use rand::{Rng, XorShiftRng};
    use test::Bencher;

    fn generate_array_with_seed(n: usize, ratio: f64, seed_val: u8) -> Vec<u32> {
-        let seed: &[u8; 16] = &[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, seed_val];
-        let mut rng: XorShiftRng = XorShiftRng::from_seed(*seed);
+        let mut seed: [u8; 32] = [0; 32];
+        seed[31] = seed_val;
+        let mut rng = StdRng::from_seed(seed);
        (0u32..).filter(|_| rng.gen_bool(ratio)).take(n).collect()
    }

--- a/src/postings/mod.rs
+++ b/src/postings/mod.rs
@@ -356,9 +356,9 @@ pub mod tests {

    #[test]
    fn test_skip_next() {
-        let term_0 = Term::from_field_u64(Field(0), 0);
-        let term_1 = Term::from_field_u64(Field(0), 1);
-        let term_2 = Term::from_field_u64(Field(0), 2);
+        let term_0 = Term::from_field_u64(Field::from_field_id(0), 0);
+        let term_1 = Term::from_field_u64(Field::from_field_id(0), 1);
+        let term_2 = Term::from_field_u64(Field::from_field_id(0), 2);

        let num_docs = 300u32;

@@ -511,19 +511,19 @@ pub mod tests {
    }

    pub static TERM_A: Lazy<Term> = Lazy::new(|| {
-        let field = Field(0);
+        let field = Field::from_field_id(0);
        Term::from_field_text(field, "a")
    });
    pub static TERM_B: Lazy<Term> = Lazy::new(|| {
-        let field = Field(0);
+        let field = Field::from_field_id(0);
        Term::from_field_text(field, "b")
    });
    pub static TERM_C: Lazy<Term> = Lazy::new(|| {
-        let field = Field(0);
+        let field = Field::from_field_id(0);
        Term::from_field_text(field, "c")
    });
    pub static TERM_D: Lazy<Term> = Lazy::new(|| {
-        let field = Field(0);
+        let field = Field::from_field_id(0);
        Term::from_field_text(field, "d")
    });

@@ -622,23 +622,23 @@ pub mod tests {
            assert!(!postings_unopt.advance());
        }
    }
-
 }

 #[cfg(all(test, feature = "unstable"))]
 mod bench {

    use super::tests::*;
-    use docset::SkipResult;
-    use query::Intersection;
-    use schema::IndexRecordOption;
+    use crate::docset::SkipResult;
+    use crate::query::Intersection;
+    use crate::schema::IndexRecordOption;
+    use crate::tests;
+    use crate::DocSet;
    use test::{self, Bencher};
-    use tests;
-    use DocSet;

    #[bench]
    fn bench_segment_postings(b: &mut Bencher) {
-        let searcher = INDEX.searcher();
+        let reader = INDEX.reader().unwrap();
+        let searcher = reader.searcher();
        let segment_reader = searcher.segment_reader(0);

        b.iter(|| {
@@ -652,7 +652,8 @@ mod bench {

    #[bench]
    fn bench_segment_intersection(b: &mut Bencher) {
-        let searcher = INDEX.searcher();
+        let reader = INDEX.reader().unwrap();
+        let searcher = reader.searcher();
        let segment_reader = searcher.segment_reader(0);
        b.iter(|| {
            let segment_postings_a = segment_reader
@@ -682,7 +683,8 @@ mod bench {
    }

    fn bench_skip_next(p: f64, b: &mut Bencher) {
-        let searcher = INDEX.searcher();
+        let reader = INDEX.reader().unwrap();
+        let searcher = reader.searcher();
        let segment_reader = searcher.segment_reader(0);
        let docs = tests::sample(segment_reader.num_docs(), p);

@@ -737,7 +739,8 @@ mod bench {

    #[bench]
    fn bench_iterate_segment_postings(b: &mut Bencher) {
-        let searcher = INDEX.searcher();
+        let reader = INDEX.reader().unwrap();
+        let searcher = reader.searcher();
        let segment_reader = searcher.segment_reader(0);
        b.iter(|| {
            let n: u32 = test::black_box(17);
--- a/src/postings/postings_writer.rs
+++ b/src/postings/postings_writer.rs
@@ -12,6 +12,7 @@ use crate::tokenizer::TokenStream;
 use crate::tokenizer::{Token, MAX_TOKEN_LEN};
 use crate::DocId;
 use crate::Result;
+use fnv::FnvHashMap;
 use std::collections::HashMap;
 use std::io;
 use std::marker::PhantomData;
@@ -60,12 +61,12 @@ fn make_field_partition(
        .iter()
        .map(|(key, _, _)| Term::wrap(key).field())
        .enumerate();
-    let mut prev_field = Field(u32::max_value());
+    let mut prev_field_opt = None;
    let mut fields = vec![];
    let mut offsets = vec![];
    for (offset, field) in term_offsets_it {
-        if field != prev_field {
-            prev_field = field;
+        if Some(field) != prev_field_opt {
+            prev_field_opt = Some(field);
            fields.push(field);
            offsets.push(offset);
        }
@@ -85,8 +86,7 @@ impl MultiFieldPostingsWriter {
        let term_index = TermHashMap::new(table_bits);
        let per_field_postings_writers: Vec<_> = schema
            .fields()
-            .iter()
-            .map(|field_entry| posting_from_field_entry(field_entry))
+            .map(|(_, field_entry)| posting_from_field_entry(field_entry))
            .collect();
        MultiFieldPostingsWriter {
            heap: MemoryArena::new(),
@@ -106,7 +106,8 @@ impl MultiFieldPostingsWriter {
        field: Field,
        token_stream: &mut dyn TokenStream,
    ) -> u32 {
-        let postings_writer = self.per_field_postings_writers[field.0 as usize].deref_mut();
+        let postings_writer =
+            self.per_field_postings_writers[field.field_id() as usize].deref_mut();
        postings_writer.index_text(
            &mut self.term_index,
            doc,
@@ -117,7 +118,8 @@ impl MultiFieldPostingsWriter {
    }

    pub fn subscribe(&mut self, doc: DocId, term: &Term) -> UnorderedTermId {
-        let postings_writer = self.per_field_postings_writers[term.field().0 as usize].deref_mut();
+        let postings_writer =
+            self.per_field_postings_writers[term.field().field_id() as usize].deref_mut();
        postings_writer.subscribe(&mut self.term_index, doc, 0u32, term, &mut self.heap)
    }

@@ -127,12 +129,12 @@ impl MultiFieldPostingsWriter {
    pub fn serialize(
        &self,
        serializer: &mut InvertedIndexSerializer,
-    ) -> Result<HashMap<Field, HashMap<UnorderedTermId, TermOrdinal>>> {
+    ) -> Result<HashMap<Field, FnvHashMap<UnorderedTermId, TermOrdinal>>> {
        let mut term_offsets: Vec<(&[u8], Addr, UnorderedTermId)> =
            self.term_index.iter().collect();
        term_offsets.sort_unstable_by_key(|&(k, _, _)| k);

-        let mut unordered_term_mappings: HashMap<Field, HashMap<UnorderedTermId, TermOrdinal>> =
+        let mut unordered_term_mappings: HashMap<Field, FnvHashMap<UnorderedTermId, TermOrdinal>> =
            HashMap::new();

        let field_offsets = make_field_partition(&term_offsets);
@@ -147,7 +149,7 @@ impl MultiFieldPostingsWriter {
                    let unordered_term_ids = term_offsets[start..stop]
                        .iter()
                        .map(|&(_, _, bucket)| bucket);
-                    let mapping: HashMap<UnorderedTermId, TermOrdinal> = unordered_term_ids
+                    let mapping: FnvHashMap<UnorderedTermId, TermOrdinal> = unordered_term_ids
                        .enumerate()
                        .map(|(term_ord, unord_term_id)| {
                            (unord_term_id as UnorderedTermId, term_ord as TermOrdinal)
@@ -159,7 +161,7 @@ impl MultiFieldPostingsWriter {
                FieldType::Bytes => {}
            }

-            let postings_writer = &self.per_field_postings_writers[field.0 as usize];
+            let postings_writer = &self.per_field_postings_writers[field.field_id() as usize];
            let mut field_serializer =
                serializer.new_field(field, postings_writer.total_num_tokens())?;
            postings_writer.serialize(
--- a/src/postings/serializer.rs
+++ b/src/postings/serializer.rs
@@ -141,10 +141,7 @@ impl<'a> FieldSerializer<'a> {
            FieldType::Str(ref text_options) => {
                if let Some(text_indexing_options) = text_options.get_indexing_options() {
                    let index_option = text_indexing_options.index_option();
-                    (
-                        index_option.is_termfreq_enabled(),
-                        index_option.is_position_enabled(),
-                    )
+                    (index_option.has_freq(), index_option.has_positions())
                } else {
                    (false, false)
                }
--- a/src/postings/stacker/expull.rs
+++ b/src/postings/stacker/expull.rs
@@ -310,6 +310,7 @@ mod bench {
    use super::super::MemoryArena;
    use super::ExpUnrolledLinkedList;
    use byteorder::{NativeEndian, WriteBytesExt};
+    use std::iter;
    use test::Bencher;

    const NUM_STACK: usize = 10_000;
@@ -335,11 +336,10 @@ mod bench {
    fn bench_push_stack(bench: &mut Bencher) {
        bench.iter(|| {
            let mut heap = MemoryArena::new();
-            let mut stacks = Vec::with_capacity(100);
-            for _ in 0..NUM_STACK {
-                let mut stack = ExpUnrolledLinkedList::new();
-                stacks.push(stack);
-            }
+            let mut stacks: Vec<ExpUnrolledLinkedList> =
+                iter::repeat_with(ExpUnrolledLinkedList::new)
+                    .take(NUM_STACK)
+                    .collect();
            for s in 0..NUM_STACK {
                for i in 0u32..STACK_SIZE {
                    let t = s * 392017 % NUM_STACK;
--- a/src/query/all_query.rs
+++ b/src/query/all_query.rs
@@ -130,5 +130,4 @@ mod tests {
            assert!(!scorer.advance());
        }
    }
-
 }
--- a/src/query/automaton_weight.rs
+++ b/src/query/automaton_weight.rs
@@ -8,15 +8,13 @@ use crate::termdict::{TermDictionary, TermStreamer};
 use crate::DocId;
 use crate::TantivyError;
 use crate::{Result, SkipResult};
+use std::sync::Arc;
 use tantivy_fst::Automaton;

 /// A weight struct for Fuzzy Term and Regex Queries
-pub struct AutomatonWeight<A>
-where
-    A: Automaton + Send + Sync + 'static,
-{
+pub struct AutomatonWeight<A> {
    field: Field,
-    automaton: A,
+    automaton: Arc<A>,
 }

 impl<A> AutomatonWeight<A>
@@ -24,12 +22,16 @@ where
    A: Automaton + Send + Sync + 'static,
 {
    /// Create a new AutomationWeight
-    pub fn new(field: Field, automaton: A) -> AutomatonWeight<A> {
-        AutomatonWeight { field, automaton }
+    pub fn new<IntoArcA: Into<Arc<A>>>(field: Field, automaton: IntoArcA) -> AutomatonWeight<A> {
+        AutomatonWeight {
+            field,
+            automaton: automaton.into(),
+        }
    }

    fn automaton_stream<'a>(&'a self, term_dict: &'a TermDictionary) -> TermStreamer<'a, &'a A> {
-        let term_stream_builder = term_dict.search(&self.automaton);
+        let automaton: &A = &*self.automaton;
+        let term_stream_builder = term_dict.search(automaton);
        term_stream_builder.into_stream()
    }
 }
--- a/src/query/bitset/mod.rs
+++ b/src/query/bitset/mod.rs
@@ -216,7 +216,6 @@ mod tests {
            assert!(!docset.advance());
        }
    }
-
 }

 #[cfg(all(test, feature = "unstable"))]
@@ -224,13 +223,12 @@ mod bench {

    use super::BitSet;
    use super::BitSetDocSet;
-    use test;
-    use tests;
-    use DocSet;
+    use crate::test;
+    use crate::tests;
+    use crate::DocSet;

    #[bench]
    fn bench_bitset_1pct_insert(b: &mut test::Bencher) {
-        use tests;
        let els = tests::generate_nonunique_unsorted(1_000_000u32, 10_000);
        b.iter(|| {
            let mut bitset = BitSet::with_max_value(1_000_000);
@@ -242,7 +240,6 @@ mod bench {

    #[bench]
    fn bench_bitset_1pct_clone(b: &mut test::Bencher) {
-        use tests;
        let els = tests::generate_nonunique_unsorted(1_000_000u32, 10_000);
        let mut bitset = BitSet::with_max_value(1_000_000);
        for el in els {
--- a/src/query/bm25.rs
+++ b/src/query/bm25.rs
@@ -137,5 +137,4 @@ mod tests {
    fn test_idf() {
        assert_nearly_equals(idf(1, 2), 0.6931472);
    }
-
 }
--- a/src/query/boolean_query/boolean_query.rs
+++ b/src/query/boolean_query/boolean_query.rs
@@ -9,7 +9,8 @@ use crate::Result;
 use crate::Searcher;
 use std::collections::BTreeSet;

-/// The boolean query combines a set of queries
+/// The boolean query returns a set of documents
+/// that matches the Boolean combination of constituent subqueries.
 ///
 /// The documents matched by the boolean query are
 /// those which
@@ -19,6 +20,113 @@ use std::collections::BTreeSet;
 /// `MustNot` occurence.
 /// * match at least one of the subqueries that is not
 /// a `MustNot` occurence.
+///
+///
+/// You can combine other query types and their `Occur`ances into one `BooleanQuery`
+///
+/// ```rust
+///use tantivy::collector::Count;
+///use tantivy::doc;
+///use tantivy::query::{BooleanQuery, Occur, PhraseQuery, Query, TermQuery};
+///use tantivy::schema::{IndexRecordOption, Schema, TEXT};
+///use tantivy::Term;
+///use tantivy::{Index, Result};
+///
+///fn main() -> Result<()> {
+///    let mut schema_builder = Schema::builder();
+///    let title = schema_builder.add_text_field("title", TEXT);
+///    let body = schema_builder.add_text_field("body", TEXT);
+///    let schema = schema_builder.build();
+///    let index = Index::create_in_ram(schema);
+///    {
+///        let mut index_writer = index.writer(3_000_000)?;
+///        index_writer.add_document(doc!(
+///            title => "The Name of the Wind",
+///        ));
+///        index_writer.add_document(doc!(
+///            title => "The Diary of Muadib",
+///        ));
+///        index_writer.add_document(doc!(
+///            title => "A Dairy Cow",
+///            body => "hidden",
+///        ));
+///        index_writer.add_document(doc!(
+///            title => "A Dairy Cow",
+///            body => "found",
+///        ));
+///        index_writer.add_document(doc!(
+///            title => "The Diary of a Young Girl",
+///        ));
+///        index_writer.commit().unwrap();
+///    }
+///
+///    let reader = index.reader()?;
+///    let searcher = reader.searcher();
+///
+///    // Make TermQuery's for "girl" and "diary" in the title
+///    let girl_term_query: Box<dyn Query> = Box::new(TermQuery::new(
+///        Term::from_field_text(title, "girl"),
+///        IndexRecordOption::Basic,
+///    ));
+///    let diary_term_query: Box<dyn Query> = Box::new(TermQuery::new(
+///        Term::from_field_text(title, "diary"),
+///        IndexRecordOption::Basic,
+///    ));
+///    // A TermQuery with "found" in the body
+///    let body_term_query: Box<dyn Query> = Box::new(TermQuery::new(
+///        Term::from_field_text(body, "found"),
+///        IndexRecordOption::Basic,
+///    ));
+///    // TermQuery "diary" must and "girl" must not be present
+///    let queries_with_occurs1 = vec![
+///        (Occur::Must, diary_term_query.box_clone()),
+///        (Occur::MustNot, girl_term_query),
+///    ];
+///    // Make a BooleanQuery equivalent to
+///    // title:+diary title:-girl
+///    let diary_must_and_girl_mustnot = BooleanQuery::from(queries_with_occurs1);
+///    let count1 = searcher.search(&diary_must_and_girl_mustnot, &Count)?;
+///    assert_eq!(count1, 1);
+///
+///    // TermQuery for "cow" in the title
+///    let cow_term_query: Box<dyn Query> = Box::new(TermQuery::new(
+///        Term::from_field_text(title, "cow"),
+///        IndexRecordOption::Basic,
+///    ));
+///    // "title:diary OR title:cow"
+///    let title_diary_or_cow = BooleanQuery::from(vec![
+///        (Occur::Should, diary_term_query.box_clone()),
+///        (Occur::Should, cow_term_query),
+///    ]);
+///    let count2 = searcher.search(&title_diary_or_cow, &Count)?;
+///    assert_eq!(count2, 4);
+///
+///    // Make a `PhraseQuery` from a vector of `Term`s
+///    let phrase_query: Box<dyn Query> = Box::new(PhraseQuery::new(vec![
+///        Term::from_field_text(title, "dairy"),
+///        Term::from_field_text(title, "cow"),
+///    ]));
+///    // You can combine subqueries of different types into 1 BooleanQuery:
+///    // `TermQuery` and `PhraseQuery`
+///    // "title:diary OR "dairy cow"
+///    let term_of_phrase_query = BooleanQuery::from(vec![
+///        (Occur::Should, diary_term_query.box_clone()),
+///        (Occur::Should, phrase_query.box_clone()),
+///    ]);
+///    let count3 = searcher.search(&term_of_phrase_query, &Count)?;
+///    assert_eq!(count3, 4);
+///
+///    // You can nest one BooleanQuery inside another
+///    // body:found AND ("title:diary OR "dairy cow")
+///    let nested_query = BooleanQuery::from(vec![
+///        (Occur::Must, body_term_query),
+///        (Occur::Must, Box::new(term_of_phrase_query))
+///    ]);
+///    let count4 = searcher.search(&nested_query, &Count)?;
+///    assert_eq!(count4, 1);
+///    Ok(())
+///}
+/// ```
 #[derive(Debug)]
 pub struct BooleanQuery {
    subqueries: Vec<(Occur, Box<dyn Query>)>,
--- a/src/query/boolean_query/mod.rs
+++ b/src/query/boolean_query/mod.rs
@@ -247,9 +247,7 @@ mod tests {
        let reader = index.reader().unwrap();
        let searcher = reader.searcher();
        let query_parser = QueryParser::for_index(&index, vec![title, text]);
-        let query = query_parser
-            .parse_query("Оксана Лифенко")
-            .unwrap();
+        let query = query_parser.parse_query("Оксана Лифенко").unwrap();
        let weight = query.weight(&searcher, true).unwrap();
        let mut scorer = weight.scorer(searcher.segment_reader(0u32)).unwrap();
        scorer.advance();
--- a/src/query/exclude.rs
+++ b/src/query/exclude.rs
@@ -175,5 +175,4 @@ mod tests {
            sample_skip,
        );
    }
-
 }
--- a/src/query/fuzzy_query.rs
+++ b/src/query/fuzzy_query.rs
@@ -1,3 +1,4 @@
+use crate::error::TantivyError::InvalidArgument;
 use crate::query::{AutomatonWeight, Query, Weight};
 use crate::schema::Term;
 use crate::Result;
@@ -5,11 +6,16 @@ use crate::Searcher;
 use levenshtein_automata::{LevenshteinAutomatonBuilder, DFA};
 use once_cell::sync::Lazy;
 use std::collections::HashMap;
+use std::ops::Range;
+
+/// A range of Levenshtein distances that we will build DFAs for our terms
+/// The computation is exponential, so best keep it to low single digits
+const VALID_LEVENSHTEIN_DISTANCE_RANGE: Range<u8> = (0..3);

 static LEV_BUILDER: Lazy<HashMap<(u8, bool), LevenshteinAutomatonBuilder>> = Lazy::new(|| {
    let mut lev_builder_cache = HashMap::new();
    // TODO make population lazy on a `(distance, val)` basis
-    for distance in 0..3 {
+    for distance in VALID_LEVENSHTEIN_DISTANCE_RANGE {
        for &transposition in &[false, true] {
            let lev_automaton_builder = LevenshteinAutomatonBuilder::new(distance, transposition);
            lev_builder_cache.insert((distance, transposition), lev_automaton_builder);
@@ -22,12 +28,10 @@ static LEV_BUILDER: Lazy<HashMap<(u8, bool), LevenshteinAutomatonBuilder>> = Laz
 /// containing a specific term that is within
 /// Levenshtein distance
 /// ```rust
-/// #[macro_use]
-/// extern crate tantivy;
-/// use tantivy::schema::{Schema, TEXT};
-/// use tantivy::{Index, Result, Term};
 /// use tantivy::collector::{Count, TopDocs};
 /// use tantivy::query::FuzzyTermQuery;
+/// use tantivy::schema::{Schema, TEXT};
+/// use tantivy::{doc, Index, Result, Term};
 ///
 /// # fn main() { example().unwrap(); }
 /// fn example() -> Result<()> {
@@ -100,10 +104,18 @@ impl FuzzyTermQuery {
    }

    fn specialized_weight(&self) -> Result<AutomatonWeight<DFA>> {
-        let automaton = LEV_BUILDER.get(&(self.distance, false))
-            .unwrap() // TODO return an error
-            .build_dfa(self.term.text());
-        Ok(AutomatonWeight::new(self.term.field(), automaton))
+        // LEV_BUILDER is a HashMap, whose `get` method returns an Option
+        match LEV_BUILDER.get(&(self.distance, false)) {
+            // Unwrap the option and build the Ok(AutomatonWeight)
+            Some(automaton_builder) => {
+                let automaton = automaton_builder.build_dfa(self.term.text());
+                Ok(AutomatonWeight::new(self.term.field(), automaton))
+            }
+            None => Err(InvalidArgument(format!(
+                "Levenshtein distance of {} is not allowed. Choose a value in the {:?} range",
+                self.distance, VALID_LEVENSHTEIN_DISTANCE_RANGE
+            ))),
+        }
    }
 }

--- a/src/query/intersection.rs
+++ b/src/query/intersection.rs
@@ -45,7 +45,7 @@ pub fn intersect_scorers(mut scorers: Vec<Box<dyn Scorer>>) -> Box<dyn Scorer> {
    })
 }

-/// Creates a `DocSet` that iterator through the intersection of two `DocSet`s.
+/// Creates a `DocSet` that iterate through the intersection of two or more `DocSet`s.
 pub struct Intersection<TDocSet: DocSet, TOtherDocSet: DocSet = Box<dyn Scorer>> {
    left: TDocSet,
    right: TDocSet,
--- a/src/query/intersection_two.rs
+++ b/src/query/intersection_two.rs
@@ -5,7 +5,7 @@ use Score;
 use SkipResult;


-/// Creates a `DocSet` that iterator through the intersection of two `DocSet`s.
+/// Creates a `DocSet` that iterate through the intersection of two `DocSet`s.
 pub struct IntersectionTwoTerms<TDocSet> {
    left: TDocSet,
    right: TDocSet
--- a/src/query/mod.rs
+++ b/src/query/mod.rs
@@ -12,7 +12,6 @@ mod exclude;
 mod explanation;
 mod fuzzy_query;
 mod intersection;
-mod occur;
 mod phrase_query;
 mod query;
 mod query_parser;
@@ -43,7 +42,6 @@ pub use self::exclude::Exclude;
 pub use self::explanation::Explanation;
 pub use self::fuzzy_query::FuzzyTermQuery;
 pub use self::intersection::intersect_scorers;
-pub use self::occur::Occur;
 pub use self::phrase_query::PhraseQuery;
 pub use self::query::Query;
 pub use self::query_parser::QueryParser;
@@ -55,6 +53,7 @@ pub use self::scorer::ConstScorer;
 pub use self::scorer::Scorer;
 pub use self::term_query::TermQuery;
 pub use self::weight::Weight;
+pub use tantivy_query_grammar::Occur;

 #[cfg(test)]
 mod tests {
--- a/src/query/phrase_query/phrase_query.rs
+++ b/src/query/phrase_query/phrase_query.rs
@@ -40,7 +40,7 @@ impl PhraseQuery {
        PhraseQuery::new_with_offset(terms_with_offset)
    }

-    /// Creates a new `PhraseQuery` given a list of terms and there offsets.
+    /// Creates a new `PhraseQuery` given a list of terms and their offsets.
    ///
    /// Can be used to provide custom offset for each term.
    pub fn new_with_offset(mut terms: Vec<(usize, Term)>) -> PhraseQuery {
@@ -73,7 +73,7 @@ impl PhraseQuery {
            .collect::<Vec<Term>>()
    }

-    /// Returns the `PhraseWeight` for the given phrase query given a specific `searcher`.  
+    /// Returns the `PhraseWeight` for the given phrase query given a specific `searcher`.
    ///
    /// This function is the same as `.weight(...)` except it returns
    /// a specialized type `PhraseWeight` instead of a Boxed trait.
--- a/src/query/query_parser/logical_ast.rs
+++ b/src/query/query_parser/logical_ast.rs
@@ -18,7 +18,6 @@ pub enum LogicalLiteral {
    All,
 }

-#[derive(Clone)]
 pub enum LogicalAST {
    Clause(Vec<(Occur, LogicalAST)>),
    Leaf(Box<LogicalLiteral>),
--- a/src/query/query_parser/mod.rs
+++ b/src/query/query_parser/mod.rs
@@ -1,6 +1,4 @@
-mod query_grammar;
 mod query_parser;
-mod user_input_ast;

 pub mod logical_ast;
 pub use self::query_parser::QueryParser;
--- a/src/query/query_parser/query_grammar.rs
+++ b/src/query/query_parser/query_grammar.rs
@@ -1,284 +0,0 @@
-use super::query_grammar;
-use super::user_input_ast::*;
-use crate::query::occur::Occur;
-use crate::query::query_parser::user_input_ast::UserInputBound;
-use combine::char::*;
-use combine::error::StreamError;
-use combine::stream::StreamErrorFor;
-use combine::*;
-
-parser! {
-    fn field[I]()(I) -> String
-    where [I: Stream<Item = char>] {
-        (
-            letter(),
-            many(satisfy(|c: char| c.is_alphanumeric() || c == '_')),
-        ).map(|(s1, s2): (char, String)| format!("{}{}", s1, s2))
-    }
-}
-
-parser! {
-    fn word[I]()(I) -> String
-    where [I: Stream<Item = char>] {
-        many1(satisfy(|c: char| c.is_alphanumeric() || c=='.'))
-               .and_then(|s: String| {
-                   match s.as_str() {
-                     "OR" => Err(StreamErrorFor::<I>::unexpected_static_message("OR")),
-                     "AND" => Err(StreamErrorFor::<I>::unexpected_static_message("AND")),
-                     "NOT" => Err(StreamErrorFor::<I>::unexpected_static_message("NOT")),
-                     _ => Ok(s)
-                   }
-               })
-    }
-}
-
-parser! {
-    fn literal[I]()(I) -> UserInputLeaf
-    where [I: Stream<Item = char>]
-    {
-        let term_val = || {
-            let phrase = (char('"'), many1(satisfy(|c| c != '"')), char('"')).map(|(_, s, _)| s);
-            phrase.or(word())
-        };
-        let term_val_with_field = negative_number().or(term_val());
-        let term_query =
-            (field(), char(':'), term_val_with_field).map(|(field_name, _, phrase)| UserInputLiteral {
-                field_name: Some(field_name),
-                phrase,
-            });
-        let term_default_field = term_val().map(|phrase| UserInputLiteral {
-            field_name: None,
-            phrase,
-        });
-        attempt(term_query)
-            .or(term_default_field)
-            .map(UserInputLeaf::from)
-    }
-}
-
-parser! {
-    fn negative_number[I]()(I) -> String
-    where [I: Stream<Item = char>]
-    {
-            (char('-'), many1(satisfy(char::is_numeric)))
-                .map(|(s1, s2): (char, String)| format!("{}{}", s1, s2))
-    }
-}
-
-parser! {
-    fn spaces1[I]()(I) -> ()
-    where [I: Stream<Item = char>] {
-        skip_many1(space())
-    }
-}
-
-parser! {
-    fn range[I]()(I) -> UserInputLeaf
-    where [I: Stream<Item = char>] {
-        let term_val = || {
-            word().or(negative_number()).or(char('*').map(|_| "*".to_string()))
-        };
-        let lower_bound = {
-            let excl = (char('{'), term_val()).map(|(_, w)| UserInputBound::Exclusive(w));
-            let incl = (char('['), term_val()).map(|(_, w)| UserInputBound::Inclusive(w));
-            attempt(excl).or(incl)
-        };
-        let upper_bound = {
-            let excl = (term_val(), char('}')).map(|(w, _)| UserInputBound::Exclusive(w));
-            let incl = (term_val(), char(']')).map(|(w, _)| UserInputBound::Inclusive(w));
-            attempt(excl).or(incl)
-        };
-        (
-            optional((field(), char(':')).map(|x| x.0)),
-            lower_bound,
-            spaces(),
-            string("TO"),
-            spaces(),
-            upper_bound,
-        ).map(|(field, lower, _, _, _, upper)| UserInputLeaf::Range {
-                field,
-                lower,
-                upper
-        })
-    }
-}
-
-parser! {
-    fn leaf[I]()(I) -> UserInputAST
-    where [I: Stream<Item = char>] {
-         (char('-'), leaf()).map(|(_, expr)| expr.unary(Occur::MustNot) )
-        .or((char('+'), leaf()).map(|(_, expr)| expr.unary(Occur::Must) ))
-        .or((char('('), parse_to_ast(), char(')')).map(|(_, expr, _)| expr))
-        .or(char('*').map(|_| UserInputAST::from(UserInputLeaf::All) ))
-        .or(attempt(
-            (string("NOT"), spaces1(), leaf()).map(|(_, _, expr)| expr.unary(Occur::MustNot))
-            )
-         )
-        .or(attempt(
-            range().map(UserInputAST::from)
-            )
-        )
-        .or(literal().map(|leaf| UserInputAST::Leaf(Box::new(leaf))))
-    }
-}
-
-enum BinaryOperand {
-    Or,
-    And,
-}
-
-parser! {
-    fn binary_operand[I]()(I) -> BinaryOperand
-    where [I: Stream<Item = char>] {
-        (spaces1(),
-         (
-            string("AND").map(|_| BinaryOperand::And)
-           .or(string("OR").map(|_| BinaryOperand::Or))
-         ),
-         spaces1()).map(|(_, op,_)| op)
-    }
-}
-
-enum Element {
-    SingleEl(UserInputAST),
-    NormalDisjunctive(Vec<Vec<UserInputAST>>),
-}
-
-impl Element {
-    pub fn into_dnf(self) -> Vec<Vec<UserInputAST>> {
-        match self {
-            Element::NormalDisjunctive(conjunctions) => conjunctions,
-            Element::SingleEl(el) => vec![vec![el]],
-        }
-    }
-}
-
-parser! {
-    pub fn parse_to_ast[I]()(I) -> UserInputAST
-    where [I: Stream<Item = char>]
-    {
-        (
-            attempt(
-                chainl1(
-                    leaf().map(Element::SingleEl),
-                    binary_operand().map(|op: BinaryOperand|
-                        move |left: Element, right: Element| {
-                            let mut dnf = left.into_dnf();
-                            if let Element::SingleEl(el) = right {
-                                match op {
-                                    BinaryOperand::And => {
-                                        if let Some(last) = dnf.last_mut() {
-                                            last.push(el);
-                                        }
-                                    }
-                                    BinaryOperand::Or => {
-                                        dnf.push(vec!(el));
-                                    }
-                                }
-                            } else {
-                                unreachable!("Please report.")
-                            }
-                            Element::NormalDisjunctive(dnf)
-                        }
-                    )
-                )
-                .map(query_grammar::Element::into_dnf)
-                .map(|fnd| {
-                    if fnd.len() == 1 {
-                        UserInputAST::and(fnd.into_iter().next().unwrap()) //< safe
-                    } else {
-                        let conjunctions = fnd
-                        .into_iter()
-                        .map(UserInputAST::and)
-                        .collect();
-                        UserInputAST::or(conjunctions)
-                    }
-                })
-            )
-            .or(
-                sep_by(leaf(), spaces())
-                .map(|subqueries: Vec<UserInputAST>| {
-                    if subqueries.len() == 1 {
-                        subqueries.into_iter().next().unwrap()
-                    } else {
-                        UserInputAST::Clause(subqueries.into_iter().collect())
-                    }
-                })
-            )
-        )
-
-    }
-}
-
-#[cfg(test)]
-mod test {
-
-    use super::*;
-
-    fn test_parse_query_to_ast_helper(query: &str, expected: &str) {
-        let query = parse_to_ast().parse(query).unwrap().0;
-        let query_str = format!("{:?}", query);
-        assert_eq!(query_str, expected);
-    }
-
-    fn test_is_parse_err(query: &str) {
-        assert!(parse_to_ast().parse(query).is_err());
-    }
-
-    #[test]
-    fn test_parse_query_to_ast_not_op() {
-        assert_eq!(
-            format!("{:?}", parse_to_ast().parse("NOT")),
-            "Err(UnexpectedParse)"
-        );
-        test_parse_query_to_ast_helper("NOTa", "\"NOTa\"");
-        test_parse_query_to_ast_helper("NOT a", "-(\"a\")");
-    }
-
-    #[test]
-    fn test_parse_query_to_ast_binary_op() {
-        test_parse_query_to_ast_helper("a AND b", "(+(\"a\") +(\"b\"))");
-        test_parse_query_to_ast_helper("a OR b", "(?(\"a\") ?(\"b\"))");
-        test_parse_query_to_ast_helper("a OR b AND c", "(?(\"a\") ?((+(\"b\") +(\"c\"))))");
-        test_parse_query_to_ast_helper("a AND b         AND c", "(+(\"a\") +(\"b\") +(\"c\"))");
-        assert_eq!(
-            format!("{:?}", parse_to_ast().parse("a OR b aaa")),
-            "Err(UnexpectedParse)"
-        );
-        assert_eq!(
-            format!("{:?}", parse_to_ast().parse("a AND b aaa")),
-            "Err(UnexpectedParse)"
-        );
-        assert_eq!(
-            format!("{:?}", parse_to_ast().parse("aaa a OR b ")),
-            "Err(UnexpectedParse)"
-        );
-        assert_eq!(
-            format!("{:?}", parse_to_ast().parse("aaa ccc a OR b ")),
-            "Err(UnexpectedParse)"
-        );
-    }
-
-    #[test]
-    fn test_parse_query_to_ast() {
-        test_parse_query_to_ast_helper("+(a b) +d", "(+((\"a\" \"b\")) +(\"d\"))");
-        test_parse_query_to_ast_helper("(+a +b) d", "((+(\"a\") +(\"b\")) \"d\")");
-        test_parse_query_to_ast_helper("(+a)", "+(\"a\")");
-        test_parse_query_to_ast_helper("(+a +b)", "(+(\"a\") +(\"b\"))");
-        test_parse_query_to_ast_helper("abc:toto", "abc:\"toto\"");
-        test_parse_query_to_ast_helper("abc:1.1", "abc:\"1.1\"");
-        test_parse_query_to_ast_helper("+abc:toto", "+(abc:\"toto\")");
-        test_parse_query_to_ast_helper("(+abc:toto -titi)", "(+(abc:\"toto\") -(\"titi\"))");
-        test_parse_query_to_ast_helper("-abc:toto", "-(abc:\"toto\")");
-        test_parse_query_to_ast_helper("abc:a b", "(abc:\"a\" \"b\")");
-        test_parse_query_to_ast_helper("abc:\"a b\"", "abc:\"a b\"");
-        test_parse_query_to_ast_helper("foo:[1 TO 5]", "foo:[\"1\" TO \"5\"]");
-        test_parse_query_to_ast_helper("[1 TO 5]", "[\"1\" TO \"5\"]");
-        test_parse_query_to_ast_helper("foo:{a TO z}", "foo:{\"a\" TO \"z\"}");
-        test_parse_query_to_ast_helper("foo:[1 TO toto}", "foo:[\"1\" TO \"toto\"}");
-        test_parse_query_to_ast_helper("foo:[* TO toto}", "foo:[\"*\" TO \"toto\"}");
-        test_parse_query_to_ast_helper("foo:[1 TO *}", "foo:[\"1\" TO \"*\"}");
-        test_parse_query_to_ast_helper("foo:[1.1 TO *}", "foo:[\"1.1\" TO \"*\"}");
-        test_is_parse_err("abc +    ");
-    }
-}
--- a/src/query/query_parser/query_parser.rs
+++ b/src/query/query_parser/query_parser.rs
@@ -1,9 +1,5 @@
 use super::logical_ast::*;
-use super::query_grammar::parse_to_ast;
-use super::user_input_ast::*;
 use crate::core::Index;
-use crate::query::occur::compose_occur;
-use crate::query::query_parser::logical_ast::LogicalAST;
 use crate::query::AllQuery;
 use crate::query::BooleanQuery;
 use crate::query::EmptyQuery;
@@ -16,11 +12,11 @@ use crate::schema::IndexRecordOption;
 use crate::schema::{Field, Schema};
 use crate::schema::{FieldType, Term};
 use crate::tokenizer::TokenizerManager;
-use combine::Parser;
 use std::borrow::Cow;
 use std::num::{ParseFloatError, ParseIntError};
 use std::ops::Bound;
 use std::str::FromStr;
+use tantivy_query_grammar::{UserInputAST, UserInputBound, UserInputLeaf};

 /// Possible error that may happen when parsing a query.
 #[derive(Debug, PartialEq, Eq, Fail)]
@@ -222,9 +218,8 @@ impl QueryParser {

    /// Parse the user query into an AST.
    fn parse_query_to_logical_ast(&self, query: &str) -> Result<LogicalAST, QueryParserError> {
-        let (user_input_ast, _remaining) = parse_to_ast()
-            .parse(query)
-            .map_err(|_| QueryParserError::SyntaxError)?;
+        let user_input_ast =
+            tantivy_query_grammar::parse_query(query).map_err(|_| QueryParserError::SyntaxError)?;
        self.compute_logical_ast(user_input_ast)
    }

@@ -369,6 +364,7 @@ impl QueryParser {
        match *bound {
            UserInputBound::Inclusive(_) => Ok(Bound::Included(term)),
            UserInputBound::Exclusive(_) => Ok(Bound::Excluded(term)),
+            UserInputBound::Unbounded => Ok(Bound::Unbounded),
        }
    }

@@ -398,7 +394,7 @@ impl QueryParser {
                let mut logical_sub_queries: Vec<(Occur, LogicalAST)> = Vec::new();
                for sub_query in sub_queries {
                    let (occur, sub_ast) = self.compute_logical_ast_with_occur(sub_query)?;
-                    let new_occur = compose_occur(default_occur, occur);
+                    let new_occur = Occur::compose(default_occur, occur);
                    logical_sub_queries.push((new_occur, sub_ast));
                }
                Ok((Occur::Should, LogicalAST::Clause(logical_sub_queries)))
@@ -406,7 +402,7 @@ impl QueryParser {
            UserInputAST::Unary(left_occur, subquery) => {
                let (right_occur, logical_sub_queries) =
                    self.compute_logical_ast_with_occur(*subquery)?;
-                Ok((compose_occur(left_occur, right_occur), logical_sub_queries))
+                Ok((Occur::compose(left_occur, right_occur), logical_sub_queries))
            }
            UserInputAST::Leaf(leaf) => {
                let result_ast = self.compute_logical_ast_from_leaf(*leaf)?;
@@ -628,7 +624,7 @@ mod test {
    pub fn test_parse_query_untokenized() {
        test_parse_query_to_logical_ast_helper(
            "nottokenized:\"wordone wordtwo\"",
-            "Term([0, 0, 0, 7, 119, 111, 114, 100, 111, 110, \
+            "Term(field=7,bytes=[119, 111, 114, 100, 111, 110, \
             101, 32, 119, 111, 114, 100, 116, 119, 111])",
            false,
        );
@@ -672,40 +668,46 @@ mod test {
            .is_ok());
        test_parse_query_to_logical_ast_helper(
            "unsigned:2324",
-            "Term([0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 9, 20])",
+            "Term(field=3,bytes=[0, 0, 0, 0, 0, 0, 9, 20])",
            false,
        );

        test_parse_query_to_logical_ast_helper(
            "signed:-2324",
-            &format!("{:?}", Term::from_field_i64(Field(2u32), -2324)),
+            &format!(
+                "{:?}",
+                Term::from_field_i64(Field::from_field_id(2u32), -2324)
+            ),
            false,
        );

        test_parse_query_to_logical_ast_helper(
            "float:2.5",
-            &format!("{:?}", Term::from_field_f64(Field(10u32), 2.5)),
+            &format!(
+                "{:?}",
+                Term::from_field_f64(Field::from_field_id(10u32), 2.5)
+            ),
            false,
        );
    }

    #[test]
-    pub fn test_parse_query_to_ast_disjunction() {
+    pub fn test_parse_query_to_ast_single_term() {
        test_parse_query_to_logical_ast_helper(
            "title:toto",
-            "Term([0, 0, 0, 0, 116, 111, 116, 111])",
+            "Term(field=0,bytes=[116, 111, 116, 111])",
            false,
        );
        test_parse_query_to_logical_ast_helper(
            "+title:toto",
-            "Term([0, 0, 0, 0, 116, 111, 116, 111])",
+            "Term(field=0,bytes=[116, 111, 116, 111])",
            false,
        );
        test_parse_query_to_logical_ast_helper(
            "+title:toto -titi",
-            "(+Term([0, 0, 0, 0, 116, 111, 116, 111]) \
-             -(Term([0, 0, 0, 0, 116, 105, 116, 105]) \
-             Term([0, 0, 0, 1, 116, 105, 116, 105])))",
+            "(+Term(field=0,bytes=[116, 111, 116, 111]) \
+             -(Term(field=0,bytes=[116, 105, 116, 105]) \
+             Term(field=1,bytes=[116, 105, 116, 105])))",
            false,
        );
        assert_eq!(
@@ -714,49 +716,67 @@ mod test {
                .unwrap(),
            QueryParserError::AllButQueryForbidden
        );
+    }
+
+    #[test]
+    pub fn test_parse_query_to_ast_two_terms() {
        test_parse_query_to_logical_ast_helper(
            "title:a b",
-            "(Term([0, 0, 0, 0, 97]) (Term([0, 0, 0, 0, 98]) \
-             Term([0, 0, 0, 1, 98])))",
+            "(Term(field=0,bytes=[97]) (Term(field=0,bytes=[98]) Term(field=1,bytes=[98])))",
            false,
        );
        test_parse_query_to_logical_ast_helper(
            "title:\"a b\"",
-            "\"[(0, Term([0, 0, 0, 0, 97])), \
-             (1, Term([0, 0, 0, 0, 98]))]\"",
+            "\"[(0, Term(field=0,bytes=[97])), \
+             (1, Term(field=0,bytes=[98]))]\"",
            false,
        );
+    }
+
+    #[test]
+    pub fn test_parse_query_to_ast_ranges() {
        test_parse_query_to_logical_ast_helper(
            "title:[a TO b]",
-            "(Included(Term([0, 0, 0, 0, 97])) TO \
-             Included(Term([0, 0, 0, 0, 98])))",
+            "(Included(Term(field=0,bytes=[97])) TO Included(Term(field=0,bytes=[98])))",
            false,
        );
        test_parse_query_to_logical_ast_helper(
            "[a TO b]",
-            "((Included(Term([0, 0, 0, 0, 97])) TO \
-             Included(Term([0, 0, 0, 0, 98]))) \
-             (Included(Term([0, 0, 0, 1, 97])) TO \
-             Included(Term([0, 0, 0, 1, 98]))))",
+            "((Included(Term(field=0,bytes=[97])) TO \
+             Included(Term(field=0,bytes=[98]))) \
+             (Included(Term(field=1,bytes=[97])) TO \
+             Included(Term(field=1,bytes=[98]))))",
            false,
        );
        test_parse_query_to_logical_ast_helper(
            "title:{titi TO toto}",
-            "(Excluded(Term([0, 0, 0, 0, 116, 105, 116, 105])) TO \
-             Excluded(Term([0, 0, 0, 0, 116, 111, 116, 111])))",
+            "(Excluded(Term(field=0,bytes=[116, 105, 116, 105])) TO \
+             Excluded(Term(field=0,bytes=[116, 111, 116, 111])))",
            false,
        );
        test_parse_query_to_logical_ast_helper(
            "title:{* TO toto}",
-            "(Unbounded TO \
-             Excluded(Term([0, 0, 0, 0, 116, 111, 116, 111])))",
+            "(Unbounded TO Excluded(Term(field=0,bytes=[116, 111, 116, 111])))",
            false,
        );
        test_parse_query_to_logical_ast_helper(
            "title:{titi TO *}",
-            "(Excluded(Term([0, 0, 0, 0, 116, 105, 116, 105])) TO Unbounded)",
+            "(Excluded(Term(field=0,bytes=[116, 105, 116, 105])) TO Unbounded)",
            false,
        );
+        test_parse_query_to_logical_ast_helper(
+            "signed:{-5 TO 3}",
+            "(Excluded(Term(field=2,bytes=[127, 255, 255, 255, 255, 255, 255, 251])) TO \
+             Excluded(Term(field=2,bytes=[128, 0, 0, 0, 0, 0, 0, 3])))",
+            false,
+        );
+        test_parse_query_to_logical_ast_helper(
+            "float:{-1.5 TO 1.5}",
+            "(Excluded(Term(field=10,bytes=[64, 7, 255, 255, 255, 255, 255, 255])) TO \
+             Excluded(Term(field=10,bytes=[191, 248, 0, 0, 0, 0, 0, 0])))",
+            false,
+        );
+
        test_parse_query_to_logical_ast_helper("*", "*", false);
    }

@@ -858,19 +878,19 @@ mod test {
    pub fn test_parse_query_to_ast_conjunction() {
        test_parse_query_to_logical_ast_helper(
            "title:toto",
-            "Term([0, 0, 0, 0, 116, 111, 116, 111])",
+            "Term(field=0,bytes=[116, 111, 116, 111])",
            true,
        );
        test_parse_query_to_logical_ast_helper(
            "+title:toto",
-            "Term([0, 0, 0, 0, 116, 111, 116, 111])",
+            "Term(field=0,bytes=[116, 111, 116, 111])",
            true,
        );
        test_parse_query_to_logical_ast_helper(
            "+title:toto -titi",
-            "(+Term([0, 0, 0, 0, 116, 111, 116, 111]) \
-             -(Term([0, 0, 0, 0, 116, 105, 116, 105]) \
-             Term([0, 0, 0, 1, 116, 105, 116, 105])))",
+            "(+Term(field=0,bytes=[116, 111, 116, 111]) \
+             -(Term(field=0,bytes=[116, 105, 116, 105]) \
+             Term(field=1,bytes=[116, 105, 116, 105])))",
            true,
        );
        assert_eq!(
@@ -881,16 +901,25 @@ mod test {
        );
        test_parse_query_to_logical_ast_helper(
            "title:a b",
-            "(+Term([0, 0, 0, 0, 97]) \
-             +(Term([0, 0, 0, 0, 98]) \
-             Term([0, 0, 0, 1, 98])))",
+            "(+Term(field=0,bytes=[97]) \
+             +(Term(field=0,bytes=[98]) \
+             Term(field=1,bytes=[98])))",
            true,
        );
        test_parse_query_to_logical_ast_helper(
            "title:\"a b\"",
-            "\"[(0, Term([0, 0, 0, 0, 97])), \
-             (1, Term([0, 0, 0, 0, 98]))]\"",
+            "\"[(0, Term(field=0,bytes=[97])), \
+             (1, Term(field=0,bytes=[98]))]\"",
            true,
        );
    }
+
+    #[test]
+    pub fn test_query_parser_hyphen() {
+        test_parse_query_to_logical_ast_helper(
+            "title:www-form-encoded",
+            "\"[(0, Term(field=0,bytes=[119, 119, 119])), (1, Term(field=0,bytes=[102, 111, 114, 109])), (2, Term(field=0,bytes=[101, 110, 99, 111, 100, 101, 100]))]\"",
+            false
+        );
+    }
 }
--- a/src/query/query_parser/stemmer.rs
+++ b/src/query/query_parser/stemmer.rs
@@ -1,44 +0,0 @@
-use std::sync::Arc;
-use stemmer;
-
-
-pub struct StemmerTokenStream<TailTokenStream> 
-    where TailTokenStream: TokenStream {
-    tail: TailTokenStream,
-    stemmer: Arc<stemmer::Stemmer>,
-}
-
-impl<TailTokenStream> TokenStream for StemmerTokenStream<TailTokenStream>
-    where TailTokenStream: TokenStream {
-
-    fn token(&self) -> &Token {
-        self.tail.token()
-    }
-    
-    fn token_mut(&mut self) -> &mut Token {
-        self.tail.token_mut()
-    }
-
-    fn advance(&mut self) -> bool {
-        if self.tail.advance() {
-            // self.tail.token_mut().term.make_ascii_lowercase();
-            let new_str = self.stemmer.stem_str(&self.token().term);
-            true
-        }
-        else {
-            false
-        }
-    }
-
-}
-
-impl<TailTokenStream> StemmerTokenStream<TailTokenStream>
-    where TailTokenStream: TokenStream {
-    
-    fn wrap(stemmer: Arc<stemmer::Stemmer>, tail: TailTokenStream) -> StemmerTokenStream<TailTokenStream> {
-        StemmerTokenStream {
-            tail,
-            stemmer,
-        }
-    } 
-}
--- a/src/query/range_query.rs
+++ b/src/query/range_query.rs
@@ -38,14 +38,10 @@ fn map_bound<TFrom, TTo, Transform: Fn(&TFrom) -> TTo>(
 /// # Example
 ///
 /// ```rust
-///
-/// # #[macro_use]
-/// # extern crate tantivy;
-/// # use tantivy::Index;
-/// # use tantivy::schema::{Schema, INDEXED};
 /// # use tantivy::collector::Count;
-/// # use tantivy::Result;
 /// # use tantivy::query::RangeQuery;
+/// # use tantivy::schema::{Schema, INDEXED};
+/// # use tantivy::{doc, Index, Result};
 /// #
 /// # fn run() -> Result<()> {
 /// #     let mut schema_builder = Schema::builder();
@@ -338,39 +334,33 @@ mod tests {
    use crate::collector::Count;
    use crate::schema::{Document, Field, Schema, INDEXED};
    use crate::Index;
-    use crate::Result;
    use std::collections::Bound;

    #[test]
    fn test_range_query_simple() {
-        fn run() -> Result<()> {
-            let mut schema_builder = Schema::builder();
-            let year_field = schema_builder.add_u64_field("year", INDEXED);
-            let schema = schema_builder.build();
+        let mut schema_builder = Schema::builder();
+        let year_field = schema_builder.add_u64_field("year", INDEXED);
+        let schema = schema_builder.build();

-            let index = Index::create_in_ram(schema);
-            {
-                let mut index_writer = index.writer_with_num_threads(1, 6_000_000).unwrap();
-                for year in 1950u64..2017u64 {
-                    let num_docs_within_year = 10 + (year - 1950) * (year - 1950);
-                    for _ in 0..num_docs_within_year {
-                        index_writer.add_document(doc!(year_field => year));
-                    }
+        let index = Index::create_in_ram(schema);
+        {
+            let mut index_writer = index.writer_with_num_threads(1, 6_000_000).unwrap();
+            for year in 1950u64..2017u64 {
+                let num_docs_within_year = 10 + (year - 1950) * (year - 1950);
+                for _ in 0..num_docs_within_year {
+                    index_writer.add_document(doc!(year_field => year));
                }
-                index_writer.commit().unwrap();
            }
-            let reader = index.reader().unwrap();
-            let searcher = reader.searcher();
-
-            let docs_in_the_sixties = RangeQuery::new_u64(year_field, 1960u64..1970u64);
-
-            // ... or `1960..=1969` if inclusive range is enabled.
-            let count = searcher.search(&docs_in_the_sixties, &Count)?;
-            assert_eq!(count, 2285);
-            Ok(())
+            index_writer.commit().unwrap();
        }
+        let reader = index.reader().unwrap();
+        let searcher = reader.searcher();

-        run().unwrap();
+        let docs_in_the_sixties = RangeQuery::new_u64(year_field, 1960u64..1970u64);
+
+        // ... or `1960..=1969` if inclusive range is enabled.
+        let count = searcher.search(&docs_in_the_sixties, &Count).unwrap();
+        assert_eq!(count, 2285);
    }

    #[test]
@@ -460,7 +450,10 @@ mod tests {
        let count_multiples =
            |range_query: RangeQuery| searcher.search(&range_query, &Count).unwrap();

-        assert_eq!(count_multiples(RangeQuery::new_f64(float_field, 10.0..11.0)), 9);
+        assert_eq!(
+            count_multiples(RangeQuery::new_f64(float_field, 10.0..11.0)),
+            9
+        );
        assert_eq!(
            count_multiples(RangeQuery::new_f64_bounds(
                float_field,
@@ -486,5 +479,4 @@ mod tests {
            91
        );
    }
-
 }
--- a/src/query/regex_query.rs
+++ b/src/query/regex_query.rs
@@ -4,22 +4,18 @@ use crate::schema::Field;
 use crate::Result;
 use crate::Searcher;
 use std::clone::Clone;
+use std::sync::Arc;
 use tantivy_fst::Regex;

-// A Regex Query matches all of the documents
+/// A Regex Query matches all of the documents
 /// containing a specific term that matches
-/// a regex pattern
-/// A Fuzzy Query matches all of the documents
-/// containing a specific term that is within
-/// Levenshtein distance
+/// a regex pattern.
 ///
 /// ```rust
-/// #[macro_use]
-/// extern crate tantivy;
-/// use tantivy::schema::{Schema, TEXT};
-/// use tantivy::{Index, Result, Term};
 /// use tantivy::collector::Count;
 /// use tantivy::query::RegexQuery;
+/// use tantivy::schema::{Schema, TEXT};
+/// use tantivy::{doc, Index, Result, Term};
 ///
 /// # fn main() { example().unwrap(); }
 /// fn example() -> Result<()> {
@@ -48,7 +44,7 @@ use tantivy_fst::Regex;
 ///     let searcher = reader.searcher();
 ///
 ///     let term = Term::from_field_text(title, "Diary");
-///     let query = RegexQuery::new("d[ai]{2}ry".to_string(), title);
+///     let query = RegexQuery::from_pattern("d[ai]{2}ry", title)?;
 ///     let count = searcher.search(&query, &Count)?;
 ///     assert_eq!(count, 3);
 ///     Ok(())
@@ -56,30 +52,34 @@ use tantivy_fst::Regex;
 /// ```
 #[derive(Debug, Clone)]
 pub struct RegexQuery {
-    regex_pattern: String,
+    regex: Arc<Regex>,
    field: Field,
 }

 impl RegexQuery {
-    /// Creates a new Fuzzy Query
-    pub fn new(regex_pattern: String, field: Field) -> RegexQuery {
+    /// Creates a new RegexQuery from a given pattern
+    pub fn from_pattern(regex_pattern: &str, field: Field) -> Result<Self> {
+        let regex = Regex::new(&regex_pattern)
+            .map_err(|_| TantivyError::InvalidArgument(regex_pattern.to_string()))?;
+        Ok(RegexQuery::from_regex(regex, field))
+    }
+
+    /// Creates a new RegexQuery from a fully built Regex
+    pub fn from_regex<T: Into<Arc<Regex>>>(regex: T, field: Field) -> Self {
        RegexQuery {
-            regex_pattern,
+            regex: regex.into(),
            field,
        }
    }

-    fn specialized_weight(&self) -> Result<AutomatonWeight<Regex>> {
-        let automaton = Regex::new(&self.regex_pattern)
-            .map_err(|_| TantivyError::InvalidArgument(self.regex_pattern.clone()))?;
-
-        Ok(AutomatonWeight::new(self.field, automaton))
+    fn specialized_weight(&self) -> AutomatonWeight<Regex> {
+        AutomatonWeight::new(self.field, self.regex.clone())
    }
 }

 impl Query for RegexQuery {
    fn weight(&self, _searcher: &Searcher, _scoring_enabled: bool) -> Result<Box<dyn Weight>> {
-        Ok(Box::new(self.specialized_weight()?))
+        Ok(Box::new(self.specialized_weight()))
    }
 }

@@ -87,13 +87,14 @@ impl Query for RegexQuery {
 mod test {
    use super::RegexQuery;
    use crate::collector::TopDocs;
-    use crate::schema::Schema;
    use crate::schema::TEXT;
+    use crate::schema::{Field, Schema};
    use crate::tests::assert_nearly_equals;
-    use crate::Index;
+    use crate::{Index, IndexReader};
+    use std::sync::Arc;
+    use tantivy_fst::Regex;

-    #[test]
-    pub fn test_regex_query() {
+    fn build_test_index() -> (IndexReader, Field) {
        let mut schema_builder = Schema::builder();
        let country_field = schema_builder.add_text_field("country", TEXT);
        let schema = schema_builder.build();
@@ -109,20 +110,65 @@ mod test {
            index_writer.commit().unwrap();
        }
        let reader = index.reader().unwrap();
+
+        (reader, country_field)
+    }
+
+    fn verify_regex_query(
+        query_matching_one: RegexQuery,
+        query_matching_zero: RegexQuery,
+        reader: IndexReader,
+    ) {
        let searcher = reader.searcher();
        {
-            let regex_query = RegexQuery::new("jap[ao]n".to_string(), country_field);
            let scored_docs = searcher
-                .search(&regex_query, &TopDocs::with_limit(2))
+                .search(&query_matching_one, &TopDocs::with_limit(2))
                .unwrap();
            assert_eq!(scored_docs.len(), 1, "Expected only 1 document");
            let (score, _) = scored_docs[0];
            assert_nearly_equals(1f32, score);
        }
-        let regex_query = RegexQuery::new("jap[A-Z]n".to_string(), country_field);
        let top_docs = searcher
-            .search(&regex_query, &TopDocs::with_limit(2))
+            .search(&query_matching_zero, &TopDocs::with_limit(2))
            .unwrap();
        assert!(top_docs.is_empty(), "Expected ZERO document");
    }
+
+    #[test]
+    pub fn test_regex_query() {
+        let (reader, field) = build_test_index();
+
+        let matching_one = RegexQuery::from_pattern("jap[ao]n", field).unwrap();
+        let matching_zero = RegexQuery::from_pattern("jap[A-Z]n", field).unwrap();
+
+        verify_regex_query(matching_one, matching_zero, reader);
+    }
+
+    #[test]
+    pub fn test_construct_from_regex() {
+        let (reader, field) = build_test_index();
+
+        let matching_one = RegexQuery::from_regex(Regex::new("jap[ao]n").unwrap(), field);
+        let matching_zero = RegexQuery::from_regex(Regex::new("jap[A-Z]n").unwrap(), field);
+
+        verify_regex_query(matching_one, matching_zero, reader);
+    }
+
+    #[test]
+    pub fn test_construct_from_reused_regex() {
+        let r1 = Arc::new(Regex::new("jap[ao]n").unwrap());
+        let r2 = Arc::new(Regex::new("jap[A-Z]n").unwrap());
+
+        let (reader, field) = build_test_index();
+
+        let matching_one = RegexQuery::from_regex(r1.clone(), field);
+        let matching_zero = RegexQuery::from_regex(r2.clone(), field);
+
+        verify_regex_query(matching_one, matching_zero, reader.clone());
+
+        let matching_one = RegexQuery::from_regex(r1.clone(), field);
+        let matching_zero = RegexQuery::from_regex(r2.clone(), field);
+
+        verify_regex_query(matching_one, matching_zero, reader.clone());
+    }
 }
--- a/src/query/reqopt_scorer.rs
+++ b/src/query/reqopt_scorer.rs
@@ -190,5 +190,4 @@ mod tests {
            skip_docs,
        );
    }
-
 }
--- a/src/query/term_query/mod.rs
+++ b/src/query/term_query/mod.rs
@@ -12,7 +12,7 @@ mod tests {
    use crate::collector::TopDocs;
    use crate::docset::DocSet;
    use crate::query::{Query, QueryParser, Scorer, TermQuery};
-    use crate::schema::{IndexRecordOption, Schema, STRING, TEXT};
+    use crate::schema::{Field, IndexRecordOption, Schema, STRING, TEXT};
    use crate::tests::assert_nearly_equals;
    use crate::Index;
    use crate::Term;
@@ -114,4 +114,16 @@ mod tests {
        let reader = index.reader().unwrap();
        assert_eq!(term_query.count(&*reader.searcher()).unwrap(), 1);
    }
+
+    #[test]
+    fn test_term_query_debug() {
+        let term_query = TermQuery::new(
+            Term::from_field_text(Field::from_field_id(1), "hello"),
+            IndexRecordOption::WithFreqs,
+        );
+        assert_eq!(
+            format!("{:?}", term_query),
+            "TermQuery(Term(field=1,bytes=[104, 101, 108, 108, 111]))"
+        );
+    }
 }
--- a/src/query/term_query/term_query.rs
+++ b/src/query/term_query/term_query.rs
@@ -7,6 +7,7 @@ use crate::Result;
 use crate::Searcher;
 use crate::Term;
 use std::collections::BTreeSet;
+use std::fmt;

 /// A Term query matches all of the documents
 /// containing a specific term.
@@ -19,12 +20,10 @@ use std::collections::BTreeSet;
 /// * `field norm` - number of tokens in the field.
 ///
 /// ```rust
-/// #[macro_use]
-/// extern crate tantivy;
-/// use tantivy::schema::{Schema, TEXT, IndexRecordOption};
-/// use tantivy::{Index, Result, Term};
 /// use tantivy::collector::{Count, TopDocs};
 /// use tantivy::query::TermQuery;
+/// use tantivy::schema::{Schema, TEXT, IndexRecordOption};
+/// use tantivy::{doc, Index, Result, Term};
 ///
 /// # fn main() { example().unwrap(); }
 /// fn example() -> Result<()> {
@@ -61,12 +60,18 @@ use std::collections::BTreeSet;
 ///     Ok(())
 /// }
 /// ```
-#[derive(Clone, Debug)]
+#[derive(Clone)]
 pub struct TermQuery {
    term: Term,
    index_record_option: IndexRecordOption,
 }

+impl fmt::Debug for TermQuery {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+        write!(f, "TermQuery({:?})", self.term)
+    }
+}
+
 impl TermQuery {
    /// Creates a new term query.
    pub fn new(term: Term, segment_postings_options: IndexRecordOption) -> TermQuery {
--- a/src/query/union.rs
+++ b/src/query/union.rs
@@ -28,7 +28,7 @@ where
    }
 }

-/// Creates a `DocSet` that iterator through the intersection of two `DocSet`s.
+/// Creates a `DocSet` that iterate through the union of two or more `DocSet`s.
 pub struct Union<TScorer, TScoreCombiner = DoNothingCombiner> {
    docsets: Vec<TScorer>,
    bitsets: Box<[TinySet; HORIZON_NUM_TINYBITSETS]>,
@@ -409,20 +409,17 @@ mod tests {
            vec![1, 2, 3, 7, 8, 9, 99, 100, 101, 500, 20000],
        );
    }
-
 }

 #[cfg(all(test, feature = "unstable"))]
 mod bench {

-    use query::score_combiner::DoNothingCombiner;
-    use query::ConstScorer;
-    use query::Union;
-    use query::VecDocSet;
+    use crate::query::score_combiner::DoNothingCombiner;
+    use crate::query::{ConstScorer, Union, VecDocSet};
+    use crate::tests;
+    use crate::DocId;
+    use crate::DocSet;
    use test::Bencher;
-    use tests;
-    use DocId;
-    use DocSet;

    #[bench]
    fn bench_union_3_high(bench: &mut Bencher) {
--- a/src/query/vec_docset.rs
+++ b/src/query/vec_docset.rs
@@ -82,5 +82,4 @@ pub mod tests {
        }
        assert_eq!(postings.fill_buffer(&mut buffer[..]), 9);
    }
-
 }
--- a/src/schema/document.rs
+++ b/src/schema/document.rs
@@ -178,5 +178,4 @@ mod tests {
        doc.add_text(text_field, "My title");
        assert_eq!(doc.field_values().len(), 1);
    }
-
 }
--- a/src/schema/facet.rs
+++ b/src/schema/facet.rs
@@ -120,9 +120,7 @@ impl Facet {

    /// Extract path from the `Facet`.
    pub fn to_path(&self) -> Vec<&str> {
-        self.encoded_str()
-            .split(|c| c == FACET_SEP_CHAR)
-            .collect()
+        self.encoded_str().split(|c| c == FACET_SEP_CHAR).collect()
    }
 }

--- a/src/schema/field.rs
+++ b/src/schema/field.rs
@@ -3,14 +3,22 @@ use std::io;
 use std::io::Read;
 use std::io::Write;

-/// `Field` is actually a `u8` identifying a `Field`
-/// The schema is in charge of holding mapping between field names
-/// to `Field` objects.
-///
-/// Because the field id is a `u8`, tantivy can only have at most `255` fields.
-/// Value 255 is reserved.
+/// `Field` is represented by an unsigned 32-bit integer type
+/// The schema holds the mapping between field names and `Field` objects.
 #[derive(Copy, Clone, Debug, PartialEq, PartialOrd, Eq, Ord, Hash, Serialize, Deserialize)]
-pub struct Field(pub u32);
+pub struct Field(u32);
+
+impl Field {
+    /// Create a new field object for the given FieldId.
+    pub fn from_field_id(field_id: u32) -> Field {
+        Field(field_id)
+    }
+
+    /// Returns a u32 identifying uniquely a field within a schema.
+    pub fn field_id(&self) -> u32 {
+        self.0
+    }
+}

 impl BinarySerializable for Field {
    fn serialize<W: Write>(&self, writer: &mut W) -> io::Result<()> {
--- a/src/schema/field_entry.rs
+++ b/src/schema/field_entry.rs
@@ -108,7 +108,9 @@ impl FieldEntry {
    /// Returns true iff the field is a int (signed or unsigned) fast field
    pub fn is_int_fast(&self) -> bool {
        match self.field_type {
-            FieldType::U64(ref options) | FieldType::I64(ref options) | FieldType::F64(ref options) => options.is_fast(),
+            FieldType::U64(ref options)
+            | FieldType::I64(ref options)
+            | FieldType::F64(ref options) => options.is_fast(),
            _ => false,
        }
    }
--- a/src/schema/field_type.rs
+++ b/src/schema/field_type.rs
@@ -10,7 +10,7 @@ use serde_json::Value as JsonValue;

 /// Possible error that may occur while parsing a field value
 /// At this point the JSON is known to be valid.
-#[derive(Debug)]
+#[derive(Debug, PartialEq)]
 pub enum ValueParsingError {
    /// Encountered a numerical value that overflows or underflow its integer type.
    OverflowError(String),
@@ -83,9 +83,9 @@ impl FieldType {
    pub fn is_indexed(&self) -> bool {
        match *self {
            FieldType::Str(ref text_options) => text_options.get_indexing_options().is_some(),
-            FieldType::U64(ref int_options) | FieldType::I64(ref int_options) | FieldType::F64(ref int_options) => {
-                int_options.is_indexed()
-            }
+            FieldType::U64(ref int_options)
+            | FieldType::I64(ref int_options)
+            | FieldType::F64(ref int_options) => int_options.is_indexed(),
            FieldType::Date(ref date_options) => date_options.is_indexed(),
            FieldType::HierarchicalFacet => true,
            FieldType::Bytes => false,
@@ -125,9 +125,12 @@ impl FieldType {
        match *json {
            JsonValue::String(ref field_text) => match *self {
                FieldType::Str(_) => Ok(Value::Str(field_text.clone())),
-                FieldType::U64(_) | FieldType::I64(_) | FieldType::F64(_) | FieldType::Date(_) => Err(
-                    ValueParsingError::TypeError(format!("Expected an integer, got {:?}", json)),
-                ),
+                FieldType::U64(_) | FieldType::I64(_) | FieldType::F64(_) | FieldType::Date(_) => {
+                    Err(ValueParsingError::TypeError(format!(
+                        "Expected an integer, got {:?}",
+                        json
+                    )))
+                }
                FieldType::HierarchicalFacet => Ok(Value::Facet(Facet::from(field_text))),
                FieldType::Bytes => decode(field_text).map(Value::Bytes).map_err(|_| {
                    ValueParsingError::InvalidBase64(format!(
@@ -152,7 +155,7 @@ impl FieldType {
                        let msg = format!("Expected a u64 int, got {:?}", json);
                        Err(ValueParsingError::OverflowError(msg))
                    }
-                },
+                }
                FieldType::F64(_) => {
                    if let Some(field_val_f64) = field_val_num.as_f64() {
                        Ok(Value::F64(field_val_f64))
--- a/src/schema/index_record_option.rs
+++ b/src/schema/index_record_option.rs
@@ -29,22 +29,6 @@ pub enum IndexRecordOption {
 }

 impl IndexRecordOption {
-    /// Returns true iff the term frequency will be encoded.
-    pub fn is_termfreq_enabled(self) -> bool {
-        match self {
-            IndexRecordOption::WithFreqsAndPositions | IndexRecordOption::WithFreqs => true,
-            _ => false,
-        }
-    }
-
-    /// Returns true iff the term positions within the document are stored as well.
-    pub fn is_position_enabled(self) -> bool {
-        match self {
-            IndexRecordOption::WithFreqsAndPositions => true,
-            _ => false,
-        }
-    }
-
    /// Returns true iff this option includes encoding
    /// term frequencies.
    pub fn has_freq(self) -> bool {
--- a/src/schema/mod.rs
+++ b/src/schema/mod.rs
@@ -174,5 +174,4 @@ mod tests {
        assert!(!is_valid_field_name("シャボン玉"));
        assert!(is_valid_field_name("my_text_field"));
    }
-
 }
--- a/src/schema/schema.rs
+++ b/src/schema/schema.rs
@@ -167,7 +167,7 @@ impl SchemaBuilder {

    /// Adds a field entry to the schema in build.
    fn add_field(&mut self, field_entry: FieldEntry) -> Field {
-        let field = Field(self.fields.len() as u32);
+        let field = Field::from_field_id(self.fields.len() as u32);
        let field_name = field_entry.name().to_string();
        self.fields.push(field_entry);
        self.fields_map.insert(field_name, field);
@@ -223,7 +223,7 @@ pub struct Schema(Arc<InnerSchema>);
 impl Schema {
    /// Return the `FieldEntry` associated to a `Field`.
    pub fn get_field_entry(&self, field: Field) -> &FieldEntry {
-        &self.0.fields[field.0 as usize]
+        &self.0.fields[field.field_id() as usize]
    }

    /// Return the field name for a given `Field`.
@@ -232,8 +232,12 @@ impl Schema {
    }

    /// Return the list of all the `Field`s.
-    pub fn fields(&self) -> &[FieldEntry] {
-        &self.0.fields
+    pub fn fields(&self) -> impl Iterator<Item = (Field, &FieldEntry)> {
+        self.0
+            .fields
+            .iter()
+            .enumerate()
+            .map(|(field_id, field_entry)| (Field::from_field_id(field_id as u32), field_entry))
    }

    /// Creates a new builder.
@@ -246,6 +250,25 @@ impl Schema {
        self.0.fields_map.get(field_name).cloned()
    }

+    /// Create a named document off the doc.
+    pub fn convert_named_doc(
+        &self,
+        named_doc: NamedFieldDocument,
+    ) -> Result<Document, DocParsingError> {
+        let mut document = Document::new();
+        for (field_name, values) in named_doc.0 {
+            if let Some(field) = self.get_field(&field_name) {
+                for value in values {
+                    let field_value = FieldValue::new(field, value);
+                    document.add(field_value);
+                }
+            } else {
+                return Err(DocParsingError::NoSuchFieldInSchema(field_name));
+            }
+        }
+        Ok(document)
+    }
+
    /// Create a named document off the doc.
    pub fn to_named_doc(&self, doc: &Document) -> NamedFieldDocument {
        let mut field_map = BTreeMap::new();
@@ -261,24 +284,6 @@ impl Schema {
        NamedFieldDocument(field_map)
    }

-    /// Converts a named doc into a document.
-    pub fn from_named_doc(
-        &self,
-        named_doc: NamedFieldDocument,
-    ) -> Result<Document, DocParsingError> {
-        let mut doc = Document::default();
-        for (field_name, field_values) in named_doc.0 {
-            if let Some(field) = self.get_field(&field_name) {
-                for field_value in field_values {
-                    doc.add(FieldValue::new(field, field_value));
-                }
-            } else {
-                return Err(DocParsingError::NoSuchFieldInSchema(field_name.clone()));
-            }
-        }
-        Ok(doc)
-    }
-
    /// Encode the schema in JSON.
    ///
    /// Encoding a document cannot fail.
@@ -297,30 +302,29 @@ impl Schema {
                };
                DocParsingError::NotJSON(doc_json_sample)
            })?;
+
        let mut doc = Document::default();
        for (field_name, json_value) in json_obj.iter() {
-            match self.get_field(field_name) {
-                Some(field) => {
-                    let field_entry = self.get_field_entry(field);
-                    let field_type = field_entry.field_type();
-                    match *json_value {
-                        JsonValue::Array(ref json_items) => {
-                            for json_item in json_items {
-                                let value = field_type.value_from_json(json_item).map_err(|e| {
-                                    DocParsingError::ValueError(field_name.clone(), e)
-                                })?;
-                                doc.add(FieldValue::new(field, value));
-                            }
-                        }
-                        _ => {
-                            let value = field_type
-                                .value_from_json(json_value)
-                                .map_err(|e| DocParsingError::ValueError(field_name.clone(), e))?;
-                            doc.add(FieldValue::new(field, value));
-                        }
+            let field = self
+                .get_field(field_name)
+                .ok_or_else(|| DocParsingError::NoSuchFieldInSchema(field_name.clone()))?;
+            let field_entry = self.get_field_entry(field);
+            let field_type = field_entry.field_type();
+            match *json_value {
+                JsonValue::Array(ref json_items) => {
+                    for json_item in json_items {
+                        let value = field_type
+                            .value_from_json(json_item)
+                            .map_err(|e| DocParsingError::ValueError(field_name.clone(), e))?;
+                        doc.add(FieldValue::new(field, value));
                    }
                }
-                None => return Err(DocParsingError::NoSuchFieldInSchema(field_name.clone())),
+                _ => {
+                    let value = field_type
+                        .value_from_json(json_value)
+                        .map_err(|e| DocParsingError::ValueError(field_name.clone(), e))?;
+                    doc.add(FieldValue::new(field, value));
+                }
            }
        }
        Ok(doc)
@@ -377,7 +381,7 @@ impl<'de> Deserialize<'de> for Schema {

 /// Error that may happen when deserializing
 /// a document from JSON.
-#[derive(Debug, Fail)]
+#[derive(Debug, Fail, PartialEq)]
 pub enum DocParsingError {
    /// The payload given is not valid JSON.
    #[fail(display = "The provided string is not valid JSON")]
@@ -386,7 +390,10 @@ pub enum DocParsingError {
    #[fail(display = "The field '{:?}' could not be parsed: {:?}", _0, _1)]
    ValueError(String, ValueParsingError),
    /// The json-document contains a field that is not declared in the schema.
-    #[fail(display = "The json-document contains an unknown field: {:?}", _0)]
+    #[fail(
+        display = "The document contains a field that is not declared in the schema: {:?}",
+        _0
+    )]
    NoSuchFieldInSchema(String),
 }

@@ -398,6 +405,7 @@ mod tests {
    use crate::schema::*;
    use matches::{assert_matches, matches};
    use serde_json;
+    use std::collections::BTreeMap;

    #[test]
    pub fn is_indexed_test() {
@@ -481,13 +489,32 @@ mod tests {

        let schema: Schema = serde_json::from_str(expected).unwrap();

-        let mut fields = schema.fields().iter();
-
-        assert_eq!("title", fields.next().unwrap().name());
-        assert_eq!("author", fields.next().unwrap().name());
-        assert_eq!("count", fields.next().unwrap().name());
-        assert_eq!("popularity", fields.next().unwrap().name());
-        assert_eq!("score", fields.next().unwrap().name());
+        let mut fields = schema.fields();
+        {
+            let (field, field_entry) = fields.next().unwrap();
+            assert_eq!("title", field_entry.name());
+            assert_eq!(0, field.field_id());
+        }
+        {
+            let (field, field_entry) = fields.next().unwrap();
+            assert_eq!("author", field_entry.name());
+            assert_eq!(1, field.field_id());
+        }
+        {
+            let (field, field_entry) = fields.next().unwrap();
+            assert_eq!("count", field_entry.name());
+            assert_eq!(2, field.field_id());
+        }
+        {
+            let (field, field_entry) = fields.next().unwrap();
+            assert_eq!("popularity", field_entry.name());
+            assert_eq!(3, field.field_id());
+        }
+        {
+            let (field, field_entry) = fields.next().unwrap();
+            assert_eq!("score", field_entry.name());
+            assert_eq!(4, field.field_id());
+        }
        assert!(fields.next().is_none());
    }

@@ -512,6 +539,54 @@ mod tests {
        assert_eq!(doc, doc_serdeser);
    }

+    #[test]
+    pub fn test_document_from_nameddoc() {
+        let mut schema_builder = Schema::builder();
+        let title = schema_builder.add_text_field("title", TEXT);
+        let val = schema_builder.add_i64_field("val", INDEXED);
+        let schema = schema_builder.build();
+        let mut named_doc_map = BTreeMap::default();
+        named_doc_map.insert(
+            "title".to_string(),
+            vec![Value::from("title1"), Value::from("title2")],
+        );
+        named_doc_map.insert(
+            "val".to_string(),
+            vec![Value::from(14u64), Value::from(-1i64)],
+        );
+        let doc = schema
+            .convert_named_doc(NamedFieldDocument(named_doc_map))
+            .unwrap();
+        assert_eq!(
+            doc.get_all(title),
+            vec![
+                &Value::from("title1".to_string()),
+                &Value::from("title2".to_string())
+            ]
+        );
+        assert_eq!(
+            doc.get_all(val),
+            vec![&Value::from(14u64), &Value::from(-1i64)]
+        );
+    }
+
+    #[test]
+    pub fn test_document_from_nameddoc_error() {
+        let schema = Schema::builder().build();
+        let mut named_doc_map = BTreeMap::default();
+        named_doc_map.insert(
+            "title".to_string(),
+            vec![Value::from("title1"), Value::from("title2")],
+        );
+        let err = schema
+            .convert_named_doc(NamedFieldDocument(named_doc_map))
+            .unwrap_err();
+        assert_eq!(
+            err,
+            DocParsingError::NoSuchFieldInSchema("title".to_string())
+        );
+    }
+
    #[test]
    pub fn test_parse_document() {
        let mut schema_builder = Schema::builder();
--- a/src/schema/term.rs
+++ b/src/schema/term.rs
@@ -22,10 +22,10 @@ impl Term {
    /// Builds a term given a field, and a i64-value
    ///
    /// Assuming the term has a field id of 1, and a i64 value of 3234,
-    /// the Term will have 8 bytes.
+    /// the Term will have 12 bytes.
    ///
    /// The first four byte are dedicated to storing the field id as a u64.
-    /// The 4 following bytes are encoding the u64 value.
+    /// The 8 following bytes are encoding the u64 value.
    pub fn from_field_i64(field: Field, val: i64) -> Term {
        let val_u64: u64 = common::i64_to_u64(val);
        Term::from_field_u64(field, val_u64)
@@ -33,11 +33,11 @@ impl Term {

    /// Builds a term given a field, and a f64-value
    ///
-    /// Assuming the term has a field id of 1, and a u64 value of 3234,
-    /// the Term will have 8 bytes. <= this is wrong
+    /// Assuming the term has a field id of 1, and a f64 value of 1.5,
+    /// the Term will have 12 bytes.
    ///
    /// The first four byte are dedicated to storing the field id as a u64.
-    /// The 4 following bytes are encoding the u64 value.
+    /// The 8 following bytes are encoding the f64 as a u64 value.
    pub fn from_field_f64(field: Field, val: f64) -> Term {
        let val_u64: u64 = common::f64_to_u64(val);
        Term::from_field_u64(field, val_u64)
@@ -46,10 +46,10 @@ impl Term {
    /// Builds a term given a field, and a DateTime value
    ///
    /// Assuming the term has a field id of 1, and a timestamp i64 value of 3234,
-    /// the Term will have 8 bytes.
+    /// the Term will have 12 bytes.
    ///
    /// The first four byte are dedicated to storing the field id as a u64.
-    /// The 4 following bytes are encoding the DateTime as i64 timestamp value.
+    /// The 8 following bytes are encoding the DateTime as i64 timestamp value.
    pub fn from_field_date(field: Field, val: &DateTime) -> Term {
        let val_timestamp = val.timestamp();
        Term::from_field_i64(field, val_timestamp)
@@ -82,10 +82,10 @@ impl Term {
    /// Builds a term given a field, and a u64-value
    ///
    /// Assuming the term has a field id of 1, and a u64 value of 3234,
-    /// the Term will have 8 bytes.
+    /// the Term will have 12 bytes.
    ///
    /// The first four byte are dedicated to storing the field id as a u64.
-    /// The 4 following bytes are encoding the u64 value.
+    /// The 8 following bytes are encoding the u64 value.
    pub fn from_field_u64(field: Field, val: u64) -> Term {
        let mut term = Term(vec![0u8; INT_TERM_LEN]);
        term.set_field(field);
@@ -105,7 +105,7 @@ impl Term {
        if self.0.len() < 4 {
            self.0.resize(4, 0u8);
        }
-        BigEndian::write_u32(&mut self.0[0..4], field.0);
+        BigEndian::write_u32(&mut self.0[0..4], field.field_id());
    }

    /// Sets a u64 value in the term.
@@ -157,7 +157,7 @@ where

    /// Returns the field.
    pub fn field(&self) -> Field {
-        Field(BigEndian::read_u32(&self.0.as_ref()[..4]))
+        Field::from_field_id(BigEndian::read_u32(&self.0.as_ref()[..4]))
    }

    /// Returns the `u64` value stored in a term.
@@ -182,7 +182,7 @@ where
    ///
    /// # Panics
    /// ... or returns an invalid value
-    /// if the term is not a `i64` field.
+    /// if the term is not a `f64` field.
    pub fn get_f64(&self) -> f64 {
        common::u64_to_f64(BigEndian::read_u64(&self.0.as_ref()[4..]))
    }
@@ -224,7 +224,12 @@ where

 impl fmt::Debug for Term {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
-        write!(f, "Term({:?})", &self.0[..])
+        write!(
+            f,
+            "Term(field={},bytes={:?})",
+            self.field().field_id(),
+            self.value_bytes()
+        )
    }
 }

--- a/src/schema/value.rs
+++ b/src/schema/value.rs
@@ -2,7 +2,7 @@ use crate::schema::Facet;
 use crate::DateTime;
 use serde::de::Visitor;
 use serde::{Deserialize, Deserializer, Serialize, Serializer};
-use std::{fmt, cmp::Ordering};
+use std::{cmp::Ordering, fmt};

 /// Value represents the value of a any field.
 /// It is an enum over all over all of the possible field type.
@@ -27,7 +27,7 @@ pub enum Value {
 impl Eq for Value {}
 impl Ord for Value {
    fn cmp(&self, other: &Self) -> Ordering {
-        match (self,other) {
+        match (self, other) {
            (Value::Str(l), Value::Str(r)) => l.cmp(r),
            (Value::U64(l), Value::U64(r)) => l.cmp(r),
            (Value::I64(l), Value::I64(r)) => l.cmp(r),
@@ -35,7 +35,7 @@ impl Ord for Value {
            (Value::Facet(l), Value::Facet(r)) => l.cmp(r),
            (Value::Bytes(l), Value::Bytes(r)) => l.cmp(r),
            (Value::F64(l), Value::F64(r)) => {
-                match (l.is_nan(),r.is_nan()) {
+                match (l.is_nan(), r.is_nan()) {
                    (false, false) => l.partial_cmp(r).unwrap(), // only fail on NaN
                    (true, true) => Ordering::Equal,
                    (true, false) => Ordering::Less, // we define NaN as less than -∞
@@ -155,7 +155,7 @@ impl Value {
            Value::F64(ref value) => *value,
            _ => panic!("This is not a f64 field."),
        }
-    }    
+    }

    /// Returns the Date-value, provided the value is of the `Date` type.
    ///
@@ -219,7 +219,7 @@ impl From<Vec<u8>> for Value {

 mod binary_serialize {
    use super::Value;
-    use crate::common::{BinarySerializable, f64_to_u64, u64_to_f64};
+    use crate::common::{f64_to_u64, u64_to_f64, BinarySerializable};
    use crate::schema::Facet;
    use chrono::{TimeZone, Utc};
    use std::io::{self, Read, Write};
--- a/src/snippet/mod.rs
+++ b/src/snippet/mod.rs
@@ -63,7 +63,7 @@ impl FragmentCandidate {
    fn try_add_token(&mut self, token: &Token, terms: &BTreeMap<String, f32>) {
        self.stop_offset = token.offset_to;

-        if let Some(score) = terms.get(&token.text.to_lowercase()) {
+        if let Some(&score) = terms.get(&token.text.to_lowercase()) {
            self.score += score;
            self.highlighted
                .push(HighlightSection::new(token.offset_from, token.offset_to));
@@ -142,7 +142,7 @@ impl Snippet {
 /// Fragments must be valid in the sense that `&text[fragment.start..fragment.stop]`\
 /// has to be a valid string.
 fn search_fragments<'a>(
-    tokenizer: &dyn BoxedTokenizer,
+    tokenizer: &BoxedTokenizer,
    text: &'a str,
    terms: &BTreeMap<String, f32>,
    max_num_chars: usize,
@@ -150,7 +150,6 @@ fn search_fragments<'a>(
    let mut token_stream = tokenizer.token_stream(text);
    let mut fragment = FragmentCandidate::new(0);
    let mut fragments: Vec<FragmentCandidate> = vec![];
-
    while let Some(next) = token_stream.next() {
        if (next.offset_to - fragment.start_offset) > max_num_chars {
            if fragment.score > 0.0 {
@@ -214,11 +213,9 @@ fn select_best_fragment_combination(fragments: &[FragmentCandidate], text: &str)
 /// # Example
 ///
 /// ```rust
-/// # #[macro_use]
-/// # extern crate tantivy;
-/// # use tantivy::Index;
-/// # use tantivy::schema::{Schema, TEXT};
 /// # use tantivy::query::QueryParser;
+/// # use tantivy::schema::{Schema, TEXT};
+/// # use tantivy::{doc, Index};
 /// use tantivy::SnippetGenerator;
 ///
 /// # fn main() -> tantivy::Result<()> {
@@ -254,7 +251,7 @@ fn select_best_fragment_combination(fragments: &[FragmentCandidate], text: &str)
 /// ```
 pub struct SnippetGenerator {
    terms_text: BTreeMap<String, f32>,
-    tokenizer: Box<dyn BoxedTokenizer>,
+    tokenizer: BoxedTokenizer,
    field: Field,
    max_num_chars: usize,
 }
@@ -316,12 +313,8 @@ impl SnippetGenerator {

    /// Generates a snippet for the given text.
    pub fn snippet(&self, text: &str) -> Snippet {
-        let fragment_candidates = search_fragments(
-            &*self.tokenizer,
-            &text,
-            &self.terms_text,
-            self.max_num_chars,
-        );
+        let fragment_candidates =
+            search_fragments(&self.tokenizer, &text, &self.terms_text, self.max_num_chars);
        select_best_fragment_combination(&fragment_candidates[..], &text)
    }
 }
@@ -331,7 +324,7 @@ mod tests {
    use super::{search_fragments, select_best_fragment_combination};
    use crate::query::QueryParser;
    use crate::schema::{IndexRecordOption, Schema, TextFieldIndexing, TextOptions, TEXT};
-    use crate::tokenizer::{box_tokenizer, SimpleTokenizer};
+    use crate::tokenizer::SimpleTokenizer;
    use crate::Index;
    use crate::SnippetGenerator;
    use maplit::btreemap;
@@ -355,12 +348,12 @@ Survey in 2016, 2017, and 2018."#;

    #[test]
    fn test_snippet() {
-        let boxed_tokenizer = box_tokenizer(SimpleTokenizer);
+        let boxed_tokenizer = SimpleTokenizer.into();
        let terms = btreemap! {
            String::from("rust") => 1.0,
            String::from("language") => 0.9
        };
-        let fragments = search_fragments(&*boxed_tokenizer, TEST_TEXT, &terms, 100);
+        let fragments = search_fragments(&boxed_tokenizer, TEST_TEXT, &terms, 100);
        assert_eq!(fragments.len(), 7);
        {
            let first = &fragments[0];
@@ -382,13 +375,13 @@ Survey in 2016, 2017, and 2018."#;

    #[test]
    fn test_snippet_scored_fragment() {
-        let boxed_tokenizer = box_tokenizer(SimpleTokenizer);
+        let boxed_tokenizer = SimpleTokenizer.into();
        {
            let terms = btreemap! {
                String::from("rust") =>1.0f32,
                String::from("language") => 0.9f32
            };
-            let fragments = search_fragments(&*boxed_tokenizer, TEST_TEXT, &terms, 20);
+            let fragments = search_fragments(&boxed_tokenizer, TEST_TEXT, &terms, 20);
            {
                let first = &fragments[0];
                assert_eq!(first.score, 1.0);
@@ -397,13 +390,13 @@ Survey in 2016, 2017, and 2018."#;
            let snippet = select_best_fragment_combination(&fragments[..], &TEST_TEXT);
            assert_eq!(snippet.to_html(), "<b>Rust</b> is a systems")
        }
-        let boxed_tokenizer = box_tokenizer(SimpleTokenizer);
+        let boxed_tokenizer = SimpleTokenizer.into();
        {
            let terms = btreemap! {
                String::from("rust") =>0.9f32,
                String::from("language") => 1.0f32
            };
-            let fragments = search_fragments(&*boxed_tokenizer, TEST_TEXT, &terms, 20);
+            let fragments = search_fragments(&boxed_tokenizer, TEST_TEXT, &terms, 20);
            //assert_eq!(fragments.len(), 7);
            {
                let first = &fragments[0];
@@ -417,14 +410,14 @@ Survey in 2016, 2017, and 2018."#;

    #[test]
    fn test_snippet_in_second_fragment() {
-        let boxed_tokenizer = box_tokenizer(SimpleTokenizer);
+        let boxed_tokenizer = SimpleTokenizer.into();

        let text = "a b c d e f g";

        let mut terms = BTreeMap::new();
        terms.insert(String::from("c"), 1.0);

-        let fragments = search_fragments(&*boxed_tokenizer, &text, &terms, 3);
+        let fragments = search_fragments(&boxed_tokenizer, &text, &terms, 3);

        assert_eq!(fragments.len(), 1);
        {
@@ -441,14 +434,14 @@ Survey in 2016, 2017, and 2018."#;

    #[test]
    fn test_snippet_with_term_at_the_end_of_fragment() {
-        let boxed_tokenizer = box_tokenizer(SimpleTokenizer);
+        let boxed_tokenizer = SimpleTokenizer.into();

        let text = "a b c d e f f g";

        let mut terms = BTreeMap::new();
        terms.insert(String::from("f"), 1.0);

-        let fragments = search_fragments(&*boxed_tokenizer, &text, &terms, 3);
+        let fragments = search_fragments(&boxed_tokenizer, &text, &terms, 3);

        assert_eq!(fragments.len(), 2);
        {
@@ -465,7 +458,7 @@ Survey in 2016, 2017, and 2018."#;

    #[test]
    fn test_snippet_with_second_fragment_has_the_highest_score() {
-        let boxed_tokenizer = box_tokenizer(SimpleTokenizer);
+        let boxed_tokenizer = SimpleTokenizer.into();

        let text = "a b c d e f g";

@@ -473,7 +466,7 @@ Survey in 2016, 2017, and 2018."#;
        terms.insert(String::from("f"), 1.0);
        terms.insert(String::from("a"), 0.9);

-        let fragments = search_fragments(&*boxed_tokenizer, &text, &terms, 7);
+        let fragments = search_fragments(&boxed_tokenizer, &text, &terms, 7);

        assert_eq!(fragments.len(), 2);
        {
@@ -490,14 +483,14 @@ Survey in 2016, 2017, and 2018."#;

    #[test]
    fn test_snippet_with_term_not_in_text() {
-        let boxed_tokenizer = box_tokenizer(SimpleTokenizer);
+        let boxed_tokenizer = SimpleTokenizer.into();

        let text = "a b c d";

        let mut terms = BTreeMap::new();
        terms.insert(String::from("z"), 1.0);

-        let fragments = search_fragments(&*boxed_tokenizer, &text, &terms, 3);
+        let fragments = search_fragments(&boxed_tokenizer, &text, &terms, 3);

        assert_eq!(fragments.len(), 0);

@@ -508,12 +501,12 @@ Survey in 2016, 2017, and 2018."#;

    #[test]
    fn test_snippet_with_no_terms() {
-        let boxed_tokenizer = box_tokenizer(SimpleTokenizer);
+        let boxed_tokenizer = SimpleTokenizer.into();

        let text = "a b c d";

        let terms = BTreeMap::new();
-        let fragments = search_fragments(&*boxed_tokenizer, &text, &terms, 3);
+        let fragments = search_fragments(&boxed_tokenizer, &text, &terms, 3);
        assert_eq!(fragments.len(), 0);

        let snippet = select_best_fragment_combination(&fragments[..], &text);
--- a/src/store/mod.rs
+++ b/src/store/mod.rs
@@ -120,17 +120,16 @@ pub mod tests {
            );
        }
    }
-
 }

 #[cfg(all(test, feature = "unstable"))]
 mod bench {

    use super::tests::write_lorem_ipsum_store;
-    use directory::Directory;
-    use directory::RAMDirectory;
+    use crate::directory::Directory;
+    use crate::directory::RAMDirectory;
+    use crate::store::StoreReader;
    use std::path::Path;
-    use store::StoreReader;
    use test::Bencher;

    #[bench]
--- a/src/store/skiplist/mod.rs
+++ b/src/store/skiplist/mod.rs
@@ -165,5 +165,4 @@ mod tests {
        assert_eq!(output.len(), 65);
        assert_eq!(output[0], 128u8 + 3u8);
    }
-
 }
--- a/src/store/writer.rs
+++ b/src/store/writer.rs
@@ -3,6 +3,7 @@ use super::skiplist::SkipListBuilder;
 use super::StoreReader;
 use crate::common::CountingWriter;
 use crate::common::{BinarySerializable, VInt};
+use crate::directory::TerminatingWrite;
 use crate::directory::WritePtr;
 use crate::schema::Document;
 use crate::DocId;
@@ -109,6 +110,6 @@ impl StoreWriter {
        self.offset_index_writer.write(&mut self.writer)?;
        header_offset.serialize(&mut self.writer)?;
        self.doc.serialize(&mut self.writer)?;
-        self.writer.flush()
+        self.writer.terminate()
    }
 }
--- a/src/termdict/term_info_store.rs
+++ b/src/termdict/term_info_store.rs
@@ -328,5 +328,4 @@ mod tests {
            assert_eq!(term_info_store.get(i as u64), term_infos[i]);
        }
    }
-
 }
--- a/src/tokenizer/alphanum_only.rs
+++ b/src/tokenizer/alphanum_only.rs
@@ -1,6 +1,5 @@
 //! # Example
-//! ```
-//! extern crate tantivy;
+//! ```rust
 //! use tantivy::tokenizer::*;
 //!
 //! # fn main() {
--- a/src/tokenizer/lower_caser.rs
+++ b/src/tokenizer/lower_caser.rs
@@ -98,10 +98,6 @@ mod tests {
    #[test]
    fn test_lowercaser() {
        assert_eq!(lowercase_helper("Tree"), vec!["tree".to_string()]);
-        assert_eq!(
-            lowercase_helper("Русский"),
-            vec!["русский".to_string()]
-        );
+        assert_eq!(lowercase_helper("Русский"), vec!["русский".to_string()]);
    }
-
 }
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Paul Masurel	45d219b6e5	Update field.rs	2019-10-25 09:05:56 +09:00
Paul Masurel	20aaaad04f	Update delete_queue.rs	2019-10-25 09:02:57 +09:00
Paul Masurel	ba0b89da36	Refactoring around Field Removing the contract about the order of the field, and the field id allocation.	2019-10-24 10:03:59 +09:00
petr-tik	1187a02a3e	Fixed #664 (#667 ) Removed references to u8 and old documentation	2019-10-22 09:34:10 +09:00
Andrew Banchich	f6c525b19e	Fix grammar / punctuation (#668 )	2019-10-21 10:50:53 +09:00
petr-tik	4a8f7712f3	Add a doctest to BooleanQuery (#630 ) * Add a doctest to BooleanQuery Closes #446 Mark a function that is only used in tests to be compiled for tests only Fix doc-comments in a couple of related files * Minor corrections remove whitespace, fix typos, add explicit dyn marker * WIP: BooleanQuery doc test Trying to nest several BooleanQueries together * Addressed old review rust 2018 edition + make function available to everyone * Box the previous query to resolve the type error * Rework wording in DocAdress document strings * Reworded and restructured the docstring	2019-10-07 10:05:12 +09:00
Paul Masurel	2f867aad17	Fix bench (#663 ) * fmt * Fixing bench compilation	2019-10-04 17:07:49 +09:00
Paul Masurel	5c6580eb15	fmt (#661 )	2019-10-04 12:10:01 +09:00
Paul Masurel	4c3941750b	Waiting potentially longer on watch	2019-10-01 09:50:46 +09:00
Paul Masurel	2ea8e618f2	Merge branch 'hotfix-656'	2019-10-01 09:44:56 +09:00
Paul Masurel	94f27f990b	Address #656 Broke the reference loop to make sure that the watch_router can be dropped, and the thread exits.	2019-10-01 09:34:22 +09:00
Paul Masurel	349e8aa348	Removed enum variants on type alias	2019-09-26 18:43:29 +09:00
Paul Masurel	cde9b78b8d	Fixing the issue associated with the Regex performance change	2019-09-18 18:29:27 +09:00
fdb-hiroshima	d8894f0bd2	add checksum check in ManagedDirectory (#605 ) * add checksum check in ManagedDirectory fix #400 * flush after writing checksum * don't checksum atomic file access and clone managed_paths * implement a footer storing metadata about a file this is more of a poc, it require some refactoring into multiple files `terminate(self)` is implemented, but not used anywhere yet * address comments and simplify things with new contract use BitOrder for integer to raw byte conversion consider atomic write imply atomic read, which might not actually be true use some indirection to have a boxable terminating writer * implement TerminatingWrite and make terminate() be called where it should add dependancy to drop_bomb to help find where terminate() should be called implement TerminatingWrite for wrapper writers make tests pass /!\ some tests seems to pass where they shouldn't * remove usage of drop_bomb * fmt * add test for checksum * address some review comments * update changelog * fmt	2019-09-18 18:26:25 +09:00
fdb-hiroshima	7e08e0047b	fix Term documentation (#655 ) u64-based fields are actually 4+8=12 bytes long	2019-09-11 18:49:35 +09:00
fdb-hiroshima	1a817f117f	fix documentation error (#654 ) Union missdocumented as doing an intersection Union and Intersection can hold more than 2 DocSets	2019-09-11 17:12:08 +09:00
petr-tik	2ec19b21ae	Remove unnecessary duplicate methods (#650 ) Closes #649 Spotted by @imor	2019-09-09 06:36:04 +09:00
Raminder Singh	141f5a93f7	Using FnvHashMap for mapping UnorderedTermId to TermOrdinal. Fixes #507 (#647 ) * Using FnvHashMap for mapping UnorderedTermId to TermOrdinal. Fixes #507 * Fixed cargo fmt errors	2019-09-07 19:40:21 +09:00
Paul Masurel	df47d55cd2	Occur debug interface (#648 )	2019-09-07 15:08:45 +09:00
Raminder Singh	5e579fd6b7	Fixed clippy warning: unneeded return statement (#646 )	2019-09-07 10:14:37 +09:00
Paul Masurel	4b9c1dce69	Moving queyr grammar to a different crate. (#645 )	2019-09-05 09:37:28 +09:00
Paul Masurel	d74f71bbef	Lighter regex dependency. (#644 ) Detail on https://github.com/rust-lang/regex/pull/613	2019-09-04 13:10:12 +09:00
Paul Masurel	5196ca41d8	Small code clean up	2019-09-03 09:22:32 +09:00
dependabot-preview[bot]	4959e06151	Update once_cell requirement from 0.2 to 1.0 (#643 ) Updates the requirements on [once_cell](https://github.com/matklad/once_cell) to permit the latest version. - [Release notes](https://github.com/matklad/once_cell/releases) - [Changelog](https://github.com/matklad/once_cell/blob/master/CHANGELOG.md) - [Commits](https://github.com/matklad/once_cell/compare/v0.2.0...v1.0.2) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>	2019-09-03 07:00:45 +09:00
Paul Masurel	c1635c13f6	RegexQuery performance: make it possible to cache Regexes - remastered by fulmicoton (Closes #639 ) (#641 ) * small docs cleanup * only compile a regex once per RegexQuery Building a `Regex` is an expensive operation. Users of `RegexQuery` need to cache and reuse regexes when searching across multiple fields. This is the first step towards allowing that: we can store the `Regex` directly in the `RegexQuery`, instead of the string pattern. * RegexQuery: account for possible failure in the constructor When building a regex from a str pattern, we have to account for the possibility that the pattern is invalid. Before the previous commit, the failure would happen in the `specialized_weight` method. Now that we store a compiled `Regex` in `RegexQuery`, `specialized_weight` doesn't fail anymore, and we can fail early while constructing `RegexQuery` if the pattern is invalid. This is a breaking change for users of `RegexQuery::new`. * add RegexQuery::from_regex method This builds a `RegexQuery` from an already compiled `Regex`. The use of `Into<Arc<Regex>>` is to allow the caller to either simply pass a `Regex`, or an `Arc<Regex>`, in case it needs to be cached and shared on the caller's side. * Using an Arc in AutomatonWeight Closes #639	2019-08-22 16:14:01 +09:00
Paul Masurel	135e0ea2e9	Expose new segment meta from Index (#637 )	2019-08-19 10:39:15 +09:00
Paul Masurel	f283bfd7ab	Added segmentid_from_string (#636 )	2019-08-19 10:37:30 +09:00
Joshua Dutton	9f74786db2	Update import statements in examples, doctests (#633 ) Update import statements to edition 2018, including removing `extern crate` and `#[macro_use]`. Alphabetize the statements.	2019-08-19 07:26:35 +09:00
Joshua Dutton	32e5d7a0c7	Fix trait object in doctest (#635 )	2019-08-19 07:25:00 +09:00
Joshua Dutton	84c615cff1	Fixing typos (#634 )	2019-08-19 07:24:05 +09:00
Paul Masurel	039c0a0863	Introducing a wrapper struct instead of Boxed<BoxableTokenizer> (#631 ) Closes #629	2019-08-15 16:37:04 +09:00
Paul Masurel	b3b0138b82	Change for tantivy-py Schema.convert_named_doc Better Debug string for Terms and TermQueries	2019-08-14 17:44:25 +09:00
petr-tik	ea56160cdc	Added cargo-fmt to CI runs (#627 ) * Added cargo-fmt to CI runs Closes #625 * Remove fmt from appveyor builds Windows seems to have issues with install components through rustup. Formatting should be equally informative regardless of the OS, so best to keep it in Linux on Travis	2019-08-12 08:25:47 +09:00
petr-tik	028b0a749c	Elastic unbounded range query (#624 ) * Tidy up fmt remove unneccessary -> Result<()> followed by run.unwrap() in a test * Adding support for elasticsearch-style unbounded queries Extend the UserInputBound to include Unbounded, so we can reuse formatting and internal query format * Still working on elastic-style range queries Fixes #498 Merge the elastic_range into range Reformat to make code easier to follow, use optional() macro to return Some * Fixed bugs Made the range parser insensitive to whitespace between the ":" and the range. Removed optional parsing of field. Added a unit test for the range parser. Derived PartialEq to compare the results of parsing as structs, instead of strings. Found a bug with that unit test - "}" was parsed as an UserInputBound::Exclusive, instead of UserInputBound::Unbounded. Added an early detection-and-return for in the original range parser * Correct failing test Assume that we will use "{" for Unbounded ranges Add a note in the changelog cargo-fmt * Moved parenthesis to a newline to make nested if-else more visible	2019-08-12 08:24:47 +09:00
Paul Masurel	941f06eb9f	Added Schema.from_named_doc	2019-08-11 16:50:32 +09:00
Paul Masurel	04832a86eb	WTF is this file doing here (#622 )	2019-08-08 21:54:10 +09:00
fdb-hiroshima	beb8e990cd	fix parsing neg float in range query (#621 ) fix #620	2019-08-08 20:41:04 +09:00
Paul Masurel	001af3876f	cargo fmt	2019-08-08 18:07:19 +09:00
Paul Masurel	f428f344da	Various bugfix in the query parser (#619 )	2019-08-08 17:48:21 +09:00
Paul Masurel	143f78eced	Trying to fix #609 (#616 )	2019-08-06 20:33:30 +09:00
Kornel	754b55eee5	Bump deps (#613 ) * Bump crossbeam * Warnings-- * Remove outdated tempdir	2019-08-05 22:21:22 +09:00
Paul Masurel	280ea1209c	Changes required for python binding (#610 )	2019-08-01 17:26:21 +09:00
petr-tik	0154dbe477	Replace unwrap with match and proper Error handling (#606 ) * Replace unwrap with match and proper Error handling * Replaced 'magic' values with a documented variable Didn't like the unexplained 0..3 range, thought it was best as a variable Calculating Levenshtein distance is expensive, so best explain why we should keep it low	2019-07-31 08:16:02 +09:00