postings writer enum

Removes the Write generics argument in PostingsSerializer.
This removes useless generic.
Prepares the path for codecs.
Removes one useless CountingWrite layer.
etc.

.claude/skills/rationalize-deps/SKILL.md

@@ -1,125 +0,0 @@
----
-name: rationalize-deps
-description: Analyze Cargo.toml dependencies and attempt to remove unused features to reduce compile times and binary size
----
-
-# Rationalize Dependencies
-
-This skill analyzes Cargo.toml dependencies to identify and remove unused features.
-
-## Overview
-
-Many crates enable features by default that may not be needed. This skill:
-1. Identifies dependencies with default features enabled
-2. Tests if `default-features = false` works
-3. Identifies which specific features are actually needed
-4. Verifies compilation after changes
-
-## Step 1: Identify the target
-
-Ask the user which crate(s) to analyze:
-- A specific crate name (e.g., "tokio", "serde")
-- A specific workspace member (e.g., "quickwit-search")
-- "all" to scan the entire workspace
-
-## Step 2: Analyze current dependencies
-
-For the workspace Cargo.toml (`quickwit/Cargo.toml`), list dependencies that:
-- Do NOT have `default-features = false`
-- Have default features that might be unnecessary
-
-Run: `cargo tree -p <crate> -f "{p} {f}" --edges features` to see what features are actually used.
-
-## Step 3: For each candidate dependency
-
-### 3a: Check the crate's default features
-
-Look up the crate on crates.io or check its Cargo.toml to understand:
-- What features are enabled by default
-- What each feature provides
-
-Use: `cargo metadata --format-version=1 | jq '.packages[] | select(.name == "<crate>") | .features'`
-
-### 3b: Try disabling default features
-
-Modify the dependency in `quickwit/Cargo.toml`:
-
-From:
-```toml
-some-crate = { version = "1.0" }
-```
-
-To:
-```toml
-some-crate = { version = "1.0", default-features = false }
-```
-
-### 3c: Run cargo check
-
-Run: `cargo check --workspace` (or target specific packages for faster feedback)
-
-If compilation fails:
-1. Read the error messages to identify which features are needed
-2. Add only the required features explicitly:
-   ```toml
-   some-crate = { version = "1.0", default-features = false, features = ["needed-feature"] }
-   ```
-3. Re-run cargo check
-
-### 3d: Binary search for minimal features
-
-If there are many default features, use binary search:
-1. Start with no features
-2. If it fails, add half the default features
-3. Continue until you find the minimal set
-
-## Step 4: Document findings
-
-For each dependency analyzed, report:
-- Original configuration
-- New configuration (if changed)
-- Features that were removed
-- Any features that are required
-
-## Step 5: Verify full build
-
-After all changes, run:
-```bash
-cargo check --workspace --all-targets
-cargo test --workspace --no-run
-```
-
-## Common Patterns
-
-### Serde
-Often only needs `derive`:
-```toml
-serde = { version = "1.0", default-features = false, features = ["derive", "std"] }
-```
-
-### Tokio
-Identify which runtime features are actually used:
-```toml
-tokio = { version = "1.0", default-features = false, features = ["rt-multi-thread", "macros", "sync"] }
-```
-
-### Reqwest
-Often doesn't need all TLS backends:
-```toml
-reqwest = { version = "0.11", default-features = false, features = ["rustls-tls", "json"] }
-```
-
-## Rollback
-
-If changes cause issues:
-```bash
-git checkout quickwit/Cargo.toml
-cargo check --workspace
-```
-
-## Tips
-
-- Start with large crates that have many default features (tokio, reqwest, hyper)
-- Use `cargo bloat --crates` to identify large dependencies
-- Check `cargo tree -d` for duplicate dependencies that might indicate feature conflicts
-- Some features are needed only for tests - consider using `[dev-dependencies]` features

.claude/skills/simple-pr/SKILL.md

@@ -1,60 +0,0 @@
----
-name: simple-pr
-description: Create a simple PR from staged changes with an auto-generated commit message
-disable-model-invocation: true
----
-
-# Simple PR
-
-Follow these steps to create a simple PR from staged changes:
-
-## Step 1: Check workspace state
-
-Run: `git status`
-
-Verify that all changes have been staged (no unstaged changes). If there are unstaged changes, abort and ask the user to stage their changes first with `git add`.
-
-Also verify that we are on the `main` branch. If not, abort and ask the user to switch to main first.
-
-## Step 2: Ensure main is up to date
-
-Run: `git pull origin main`
-
-This ensures we're working from the latest code.
-
-## Step 3: Review staged changes
-
-Run: `git diff --cached`
-
-Review the staged changes to understand what the PR will contain.
-
-## Step 4: Generate commit message
-
-Based on the staged changes, generate a concise commit message (1-2 sentences) that describes the "why" rather than the "what".
-
-Display the proposed commit message to the user and ask for confirmation before proceeding.
-
-## Step 5: Create a new branch
-
-Get the git username: `git config user.name | tr ' ' '-' | tr '[:upper:]' '[:lower:]'`
-
-Create a short, descriptive branch name based on the changes (e.g., `fix-typo-in-readme`, `add-retry-logic`, `update-deps`).
-
-Create and checkout the branch: `git checkout -b {username}/{short-descriptive-name}`
-
-## Step 6: Commit changes
-
-Commit with the message from step 3:
-```
-git commit -m "{commit-message}"
-```
-
-## Step 7: Push and open a PR
-
-Push the branch and open a PR:
-```
-git push -u origin {branch-name}
-gh pr create --title "{commit-message-title}" --body "{longer-description-if-needed}"
-```
-
-Report the PR URL to the user when complete.

Cargo.toml

@@ -15,7 +15,7 @@ rust-version = "1.85"
 exclude = ["benches/*.json", "benches/*.txt"]
 
 [dependencies]
-oneshot = "0.1.13"
+oneshot = "0.1.7"
 base64 = "0.22.0"
 byteorder = "1.4.3"
 crc32fast = "1.3.2"
@@ -27,7 +27,7 @@ regex = { version = "1.5.5", default-features = false, features = [
 aho-corasick = "1.0"
 tantivy-fst = "0.5"
 memmap2 = { version = "0.9.0", optional = true }
-lz4_flex = { version = "0.12", default-features = false, optional = true }
+lz4_flex = { version = "0.11", default-features = false, optional = true }
 zstd = { version = "0.13", optional = true, default-features = false }
 tempfile = { version = "3.12.0", optional = true }
 log = "0.4.16"
@@ -50,7 +50,7 @@ fail = { version = "0.5.0", optional = true }
 time = { version = "0.3.35", features = ["serde-well-known"] }
 smallvec = "1.8.0"
 rayon = "1.5.2"
-lru = "0.16.3"
+lru = "0.12.0"
 fastdivide = "0.4.0"
 itertools = "0.14.0"
 measure_time = "0.9.0"
@@ -76,7 +76,7 @@ winapi = "0.3.9"
 
 [dev-dependencies]
 binggan = "0.14.2"
-rand = "0.9"
+rand = "0.8.5"
 maplit = "1.0.2"
 matches = "0.1.9"
 pretty_assertions = "1.2.1"
@@ -85,7 +85,7 @@ test-log = "0.2.10"
 futures = "0.3.21"
 paste = "1.0.11"
 more-asserts = "0.3.1"
-rand_distr = "0.5"
+rand_distr = "0.4.3"
 time = { version = "0.3.10", features = ["serde-well-known", "macros"] }
 postcard = { version = "1.0.4", features = [
     "use-std",
@@ -189,16 +189,3 @@ harness = false
 [[bench]]
 name = "bool_queries_with_range"
 harness = false
-
-[[bench]]
-name = "str_search_and_get"
-harness = false
-
-[[bench]]
-name = "merge_segments"
-harness = false
-
-[[bench]]
-name = "regex_all_terms"
-harness = false
-

benches/agg_bench.rs

@@ -1,8 +1,8 @@
 use binggan::plugins::PeakMemAllocPlugin;
 use binggan::{black_box, InputGroup, PeakMemAlloc, INSTRUMENTED_SYSTEM};
-use rand::distr::weighted::WeightedIndex;
+use rand::distributions::WeightedIndex;
+use rand::prelude::SliceRandom;
 use rand::rngs::StdRng;
-use rand::seq::IndexedRandom;
 use rand::{Rng, SeedableRng};
 use rand_distr::Distribution;
 use serde_json::json;
@@ -532,7 +532,7 @@ fn get_test_index_bench(cardinality: Cardinality) -> tantivy::Result<Index> {
     // Prepare 1000 unique terms sampled using a Zipf distribution.
     // Exponent ~1.1 approximates top-20 terms covering around ~20%.
     let terms_1000: Vec<String> = (1..=1000).map(|i| format!("term_{i}")).collect();
-    let zipf_1000 = rand_distr::Zipf::new(1000.0, 1.1f64).unwrap();
+    let zipf_1000 = rand_distr::Zipf::new(1000, 1.1f64).unwrap();
 
     {
         let mut rng = StdRng::from_seed([1u8; 32]);
@@ -576,8 +576,8 @@ fn get_test_index_bench(cardinality: Cardinality) -> tantivy::Result<Index> {
         }
         let _val_max = 1_000_000.0;
         for _ in 0..doc_with_value {
-            let val: f64 = rng.random_range(0.0..1_000_000.0);
-            let json = if rng.random_bool(0.1) {
+            let val: f64 = rng.gen_range(0.0..1_000_000.0);
+            let json = if rng.gen_bool(0.1) {
                 // 10% are numeric values
                 json!({ "mixed_type": val })
             } else {
@@ -586,7 +586,7 @@ fn get_test_index_bench(cardinality: Cardinality) -> tantivy::Result<Index> {
             index_writer.add_document(doc!(
                 text_field => "cool",
                 json_field => json,
-                text_field_all_unique_terms => format!("unique_term_{}", rng.random::<u64>()),
+                text_field_all_unique_terms => format!("unique_term_{}", rng.gen::<u64>()),
                 text_field_many_terms => many_terms_data.choose(&mut rng).unwrap().to_string(),
                 text_field_few_terms_status => status_field_data[log_level_distribution.sample(&mut rng)].0,
                 text_field_1000_terms_zipf => terms_1000[zipf_1000.sample(&mut rng) as usize - 1].as_str(),

benches/and_or_queries.rs

@@ -55,29 +55,29 @@ fn build_shared_indices(num_docs: usize, p_a: f32, p_b: f32, p_c: f32) -> (Bench
     {
         let mut writer = index.writer_with_num_threads(1, 500_000_000).unwrap();
         for _ in 0..num_docs {
-            let has_a = rng.random_bool(p_a as f64);
-            let has_b = rng.random_bool(p_b as f64);
-            let has_c = rng.random_bool(p_c as f64);
-            let score = rng.random_range(0u64..100u64);
-            let score2 = rng.random_range(0u64..100_000u64);
+            let has_a = rng.gen_bool(p_a as f64);
+            let has_b = rng.gen_bool(p_b as f64);
+            let has_c = rng.gen_bool(p_c as f64);
+            let score = rng.gen_range(0u64..100u64);
+            let score2 = rng.gen_range(0u64..100_000u64);
             let mut title_tokens: Vec<&str> = Vec::new();
             let mut body_tokens: Vec<&str> = Vec::new();
             if has_a {
-                if rng.random_bool(0.1) {
+                if rng.gen_bool(0.1) {
                     title_tokens.push("a");
                 } else {
                     body_tokens.push("a");
                 }
             }
             if has_b {
-                if rng.random_bool(0.1) {
+                if rng.gen_bool(0.1) {
                     title_tokens.push("b");
                 } else {
                     body_tokens.push("b");
                 }
             }
             if has_c {
-                if rng.random_bool(0.1) {
+                if rng.gen_bool(0.1) {
                     title_tokens.push("c");
                 } else {
                     body_tokens.push("c");

benches/bool_queries_with_range.rs

@@ -36,13 +36,13 @@ fn build_shared_indices(num_docs: usize, p_title_a: f32, distribution: &str) ->
             "dense" => {
                 for doc_id in 0..num_docs {
                     // Always add title to avoid empty documents
-                    let title_token = if rng.random_bool(p_title_a as f64) {
+                    let title_token = if rng.gen_bool(p_title_a as f64) {
                         "a"
                     } else {
                         "b"
                     };
 
-                    let num_rand = rng.random_range(0u64..1000u64);
+                    let num_rand = rng.gen_range(0u64..1000u64);
 
                     let num_asc = (doc_id / 10000) as u64;
 
@@ -60,13 +60,13 @@ fn build_shared_indices(num_docs: usize, p_title_a: f32, distribution: &str) ->
             "sparse" => {
                 for doc_id in 0..num_docs {
                     // Always add title to avoid empty documents
-                    let title_token = if rng.random_bool(p_title_a as f64) {
+                    let title_token = if rng.gen_bool(p_title_a as f64) {
                         "a"
                     } else {
                         "b"
                     };
 
-                    let num_rand = rng.random_range(0u64..10000000u64);
+                    let num_rand = rng.gen_range(0u64..10000000u64);
 
                     let num_asc = doc_id as u64;
 

benches/merge_segments.rs

@@ -1,224 +0,0 @@
-// Benchmarks segment merging
-//
-// Notes:
-// - Input segments are kept intact (no deletes / no IndexWriter merge).
-// - Output is written to a `NullDirectory` that discards all files except
-//  fieldnorms (needed for merging).
-
-use std::collections::HashMap;
-use std::io::{self, Write};
-use std::path::{Path, PathBuf};
-use std::sync::{Arc, RwLock};
-
-use binggan::{black_box, BenchRunner};
-use rand::prelude::*;
-use rand::rngs::StdRng;
-use rand::SeedableRng;
-use tantivy::directory::error::{DeleteError, OpenReadError, OpenWriteError};
-use tantivy::directory::{
-    AntiCallToken, Directory, FileHandle, OwnedBytes, TerminatingWrite, WatchCallback, WatchHandle,
-    WritePtr,
-};
-use tantivy::indexer::{merge_filtered_segments, NoMergePolicy};
-use tantivy::schema::{Schema, TEXT};
-use tantivy::{doc, HasLen, Index, IndexSettings, Segment};
-
-#[derive(Clone, Default, Debug)]
-struct NullDirectory {
-    blobs: Arc<RwLock<HashMap<PathBuf, OwnedBytes>>>,
-}
-
-struct NullWriter;
-
-impl Write for NullWriter {
-    fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
-        Ok(buf.len())
-    }
-
-    fn flush(&mut self) -> io::Result<()> {
-        Ok(())
-    }
-}
-
-impl TerminatingWrite for NullWriter {
-    fn terminate_ref(&mut self, _token: AntiCallToken) -> io::Result<()> {
-        Ok(())
-    }
-}
-
-struct InMemoryWriter {
-    path: PathBuf,
-    buffer: Vec<u8>,
-    blobs: Arc<RwLock<HashMap<PathBuf, OwnedBytes>>>,
-}
-
-impl Write for InMemoryWriter {
-    fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
-        self.buffer.extend_from_slice(buf);
-        Ok(buf.len())
-    }
-
-    fn flush(&mut self) -> io::Result<()> {
-        Ok(())
-    }
-}
-
-impl TerminatingWrite for InMemoryWriter {
-    fn terminate_ref(&mut self, _token: AntiCallToken) -> io::Result<()> {
-        let bytes = OwnedBytes::new(std::mem::take(&mut self.buffer));
-        self.blobs.write().unwrap().insert(self.path.clone(), bytes);
-        Ok(())
-    }
-}
-
-#[derive(Debug, Default)]
-struct NullFileHandle;
-impl HasLen for NullFileHandle {
-    fn len(&self) -> usize {
-        0
-    }
-}
-impl FileHandle for NullFileHandle {
-    fn read_bytes(&self, _range: std::ops::Range<usize>) -> io::Result<OwnedBytes> {
-        unimplemented!()
-    }
-}
-
-impl Directory for NullDirectory {
-    fn get_file_handle(&self, path: &Path) -> Result<Arc<dyn FileHandle>, OpenReadError> {
-        if let Some(bytes) = self.blobs.read().unwrap().get(path) {
-            return Ok(Arc::new(bytes.clone()));
-        }
-        Ok(Arc::new(NullFileHandle))
-    }
-
-    fn delete(&self, _path: &Path) -> Result<(), DeleteError> {
-        Ok(())
-    }
-
-    fn exists(&self, _path: &Path) -> Result<bool, OpenReadError> {
-        Ok(true)
-    }
-
-    fn open_write(&self, path: &Path) -> Result<WritePtr, OpenWriteError> {
-        let path_buf = path.to_path_buf();
-        if path.to_string_lossy().ends_with(".fieldnorm") {
-            let writer = InMemoryWriter {
-                path: path_buf,
-                buffer: Vec::new(),
-                blobs: Arc::clone(&self.blobs),
-            };
-            Ok(io::BufWriter::new(Box::new(writer)))
-        } else {
-            Ok(io::BufWriter::new(Box::new(NullWriter)))
-        }
-    }
-
-    fn atomic_read(&self, path: &Path) -> Result<Vec<u8>, OpenReadError> {
-        if let Some(bytes) = self.blobs.read().unwrap().get(path) {
-            return Ok(bytes.as_slice().to_vec());
-        }
-        Err(OpenReadError::FileDoesNotExist(path.to_path_buf()))
-    }
-
-    fn atomic_write(&self, _path: &Path, _data: &[u8]) -> io::Result<()> {
-        Ok(())
-    }
-
-    fn sync_directory(&self) -> io::Result<()> {
-        Ok(())
-    }
-
-    fn watch(&self, _watch_callback: WatchCallback) -> tantivy::Result<WatchHandle> {
-        Ok(WatchHandle::empty())
-    }
-}
-
-struct MergeScenario {
-    #[allow(dead_code)]
-    index: Index,
-    segments: Vec<Segment>,
-    settings: IndexSettings,
-    label: String,
-}
-
-fn build_index(
-    num_segments: usize,
-    docs_per_segment: usize,
-    tokens_per_doc: usize,
-    vocab_size: usize,
-) -> MergeScenario {
-    let mut schema_builder = Schema::builder();
-    let body = schema_builder.add_text_field("body", TEXT);
-    let schema = schema_builder.build();
-    let index = Index::create_in_ram(schema.clone());
-
-    assert!(vocab_size > 0);
-    let total_tokens = num_segments * docs_per_segment * tokens_per_doc;
-    let use_unique_terms = vocab_size >= total_tokens;
-    let mut rng = StdRng::from_seed([7u8; 32]);
-    let mut next_token_id: u64 = 0;
-
-    {
-        let mut writer = index.writer_with_num_threads(1, 256_000_000).unwrap();
-        writer.set_merge_policy(Box::new(NoMergePolicy));
-        for _ in 0..num_segments {
-            for _ in 0..docs_per_segment {
-                let mut tokens = Vec::with_capacity(tokens_per_doc);
-                for _ in 0..tokens_per_doc {
-                    let token_id = if use_unique_terms {
-                        let id = next_token_id;
-                        next_token_id += 1;
-                        id
-                    } else {
-                        rng.random_range(0..vocab_size as u64)
-                    };
-                    tokens.push(format!("term_{token_id}"));
-                }
-                writer.add_document(doc!(body => tokens.join(" "))).unwrap();
-            }
-            writer.commit().unwrap();
-        }
-    }
-
-    let segments = index.searchable_segments().unwrap();
-    let settings = index.settings().clone();
-    let label = format!(
-        "segments={}, docs/seg={}, tokens/doc={}, vocab={}",
-        num_segments, docs_per_segment, tokens_per_doc, vocab_size
-    );
-
-    MergeScenario {
-        index,
-        segments,
-        settings,
-        label,
-    }
-}
-
-fn main() {
-    let scenarios = vec![
-        build_index(8, 50_000, 12, 8),
-        build_index(16, 50_000, 12, 8),
-        build_index(16, 100_000, 12, 8),
-        build_index(8, 50_000, 8, 8 * 50_000 * 8),
-    ];
-
-    let mut runner = BenchRunner::new();
-    for scenario in scenarios {
-        let mut group = runner.new_group();
-        group.set_name(format!("merge_segments inv_index — {}", scenario.label));
-        let segments = scenario.segments.clone();
-        let settings = scenario.settings.clone();
-        group.register("merge", move |_| {
-            let output_dir = NullDirectory::default();
-            let filter_doc_ids = vec![None; segments.len()];
-            let merged_index =
-                merge_filtered_segments(&segments, settings.clone(), filter_doc_ids, output_dir)
-                    .unwrap();
-            black_box(merged_index);
-        });
-
-        group.run();
-    }
-}

benches/range_queries.rs

@@ -33,7 +33,7 @@ fn build_shared_indices(num_docs: usize, distribution: &str) -> BenchIndex {
         match distribution {
             "dense" => {
                 for doc_id in 0..num_docs {
-                    let num_rand = rng.random_range(0u64..1000u64);
+                    let num_rand = rng.gen_range(0u64..1000u64);
                     let num_asc = (doc_id / 10000) as u64;
 
                     writer
@@ -46,7 +46,7 @@ fn build_shared_indices(num_docs: usize, distribution: &str) -> BenchIndex {
             }
             "sparse" => {
                 for doc_id in 0..num_docs {
-                    let num_rand = rng.random_range(0u64..10000000u64);
+                    let num_rand = rng.gen_range(0u64..10000000u64);
                     let num_asc = doc_id as u64;
 
                     writer

benches/range_query.rs

@@ -97,20 +97,20 @@ fn get_index_0_to_100() -> Index {
     let num_vals = 100_000;
     let docs: Vec<_> = (0..num_vals)
         .map(|_i| {
-            let id_name = if rng.random_bool(0.01) {
+            let id_name = if rng.gen_bool(0.01) {
                 "veryfew".to_string() // 1%
-            } else if rng.random_bool(0.1) {
+            } else if rng.gen_bool(0.1) {
                 "few".to_string() // 9%
             } else {
                 "most".to_string() // 90%
             };
             Doc {
                 id_name,
-                id: rng.random_range(0..100),
+                id: rng.gen_range(0..100),
                 // Multiply by 1000, so that we create most buckets in the compact space
                 // The benches depend on this range to select n-percent of elements with the
                 // methods below.
-                ip: Ipv6Addr::from_u128(rng.random_range(0..100) * 1000),
+                ip: Ipv6Addr::from_u128(rng.gen_range(0..100) * 1000),
             }
         })
         .collect();

benches/regex_all_terms.rs

@@ -1,113 +0,0 @@
-// Benchmarks regex query that matches all terms in a synthetic index.
-//
-// Corpus model:
-// - N unique terms: t000000, t000001, ...
-// - M docs
-// - K tokens per doc: doc i gets terms derived from (i, token_index)
-//
-// Query:
-// - Regex "t.*" to match all terms
-//
-// Run with:
-// - cargo bench --bench regex_all_terms
-//
-
-use std::fmt::Write;
-
-use binggan::{black_box, BenchRunner};
-use tantivy::collector::Count;
-use tantivy::query::RegexQuery;
-use tantivy::schema::{Schema, TEXT};
-use tantivy::{doc, Index, ReloadPolicy};
-
-const HEAP_SIZE_BYTES: usize = 200_000_000;
-
-#[derive(Clone, Copy)]
-struct BenchConfig {
-    num_terms: usize,
-    num_docs: usize,
-    tokens_per_doc: usize,
-}
-
-fn main() {
-    let configs = default_configs();
-
-    let mut runner = BenchRunner::new();
-    for config in configs {
-        let (index, text_field) = build_index(config, HEAP_SIZE_BYTES);
-        let reader = index
-            .reader_builder()
-            .reload_policy(ReloadPolicy::Manual)
-            .try_into()
-            .expect("reader");
-        let searcher = reader.searcher();
-        let query = RegexQuery::from_pattern("t.*", text_field).expect("regex query");
-
-        let mut group = runner.new_group();
-        group.set_name(format!(
-            "regex_all_terms_t{}_d{}_k{}",
-            config.num_terms, config.num_docs, config.tokens_per_doc
-        ));
-        group.register("regex_count", move |_| {
-            let count = searcher.search(&query, &Count).expect("search");
-            black_box(count);
-        });
-        group.run();
-    }
-}
-
-fn default_configs() -> Vec<BenchConfig> {
-    vec![
-        BenchConfig {
-            num_terms: 10_000,
-            num_docs: 100_000,
-            tokens_per_doc: 1,
-        },
-        BenchConfig {
-            num_terms: 10_000,
-            num_docs: 100_000,
-            tokens_per_doc: 8,
-        },
-        BenchConfig {
-            num_terms: 100_000,
-            num_docs: 100_000,
-            tokens_per_doc: 1,
-        },
-        BenchConfig {
-            num_terms: 100_000,
-            num_docs: 100_000,
-            tokens_per_doc: 8,
-        },
-    ]
-}
-
-fn build_index(config: BenchConfig, heap_size_bytes: usize) -> (Index, tantivy::schema::Field) {
-    let mut schema_builder = Schema::builder();
-    let text_field = schema_builder.add_text_field("text", TEXT);
-    let schema = schema_builder.build();
-    let index = Index::create_in_ram(schema);
-
-    let term_width = config.num_terms.to_string().len();
-    {
-        let mut writer = index
-            .writer_with_num_threads(1, heap_size_bytes)
-            .expect("writer");
-        let mut buffer = String::new();
-        for doc_id in 0..config.num_docs {
-            buffer.clear();
-            for token_idx in 0..config.tokens_per_doc {
-                if token_idx > 0 {
-                    buffer.push(' ');
-                }
-                let term_id = (doc_id * config.tokens_per_doc + token_idx) % config.num_terms;
-                write!(&mut buffer, "t{term_id:0term_width$}").expect("write token");
-            }
-            writer
-                .add_document(doc!(text_field => buffer.as_str()))
-                .expect("add_document");
-        }
-        writer.commit().expect("commit");
-    }
-
-    (index, text_field)
-}

benches/str_search_and_get.rs

@@ -1,421 +0,0 @@
-// This benchmark compares different approaches for retrieving string values:
-//
-// 1. Fast Field Approach: retrieves string values via term_ords() and ord_to_str()
-//
-// 2. Doc Store Approach: retrieves string values via searcher.doc() and field extraction
-//
-// The benchmark includes various data distributions:
-// - Dense Sequential: Sequential document IDs with dense data
-// - Dense Random: Random document IDs with dense data
-// - Sparse Sequential: Sequential document IDs with sparse data
-// - Sparse Random: Random document IDs with sparse data
-use std::ops::Bound;
-
-use binggan::{black_box, BenchGroup, BenchRunner};
-use rand::prelude::*;
-use rand::rngs::StdRng;
-use rand::SeedableRng;
-use tantivy::collector::{Count, DocSetCollector};
-use tantivy::query::RangeQuery;
-use tantivy::schema::document::TantivyDocument;
-use tantivy::schema::{Schema, Value, FAST, STORED, STRING};
-use tantivy::{doc, Index, ReloadPolicy, Searcher, Term};
-
-#[derive(Clone)]
-struct BenchIndex {
-    #[allow(dead_code)]
-    index: Index,
-    searcher: Searcher,
-}
-
-fn build_shared_indices(num_docs: usize, distribution: &str) -> BenchIndex {
-    // Schema with string fast field and stored field for doc access
-    let mut schema_builder = Schema::builder();
-    let f_str_fast = schema_builder.add_text_field("str_fast", STRING | STORED | FAST);
-    let f_str_stored = schema_builder.add_text_field("str_stored", STRING | STORED);
-    let schema = schema_builder.build();
-    let index = Index::create_in_ram(schema.clone());
-
-    // Populate index with stable RNG for reproducibility.
-    let mut rng = StdRng::from_seed([7u8; 32]);
-
-    {
-        let mut writer = index.writer_with_num_threads(1, 4_000_000_000).unwrap();
-
-        match distribution {
-            "dense_random" => {
-                for _doc_id in 0..num_docs {
-                    let suffix = rng.gen_range(0u64..1000u64);
-                    let str_val = format!("str_{:03}", suffix);
-
-                    writer
-                        .add_document(doc!(
-                            f_str_fast=>str_val.clone(),
-                            f_str_stored=>str_val,
-                        ))
-                        .unwrap();
-                }
-            }
-            "dense_sequential" => {
-                for doc_id in 0..num_docs {
-                    let suffix = doc_id as u64 % 1000;
-                    let str_val = format!("str_{:03}", suffix);
-
-                    writer
-                        .add_document(doc!(
-                            f_str_fast=>str_val.clone(),
-                            f_str_stored=>str_val,
-                        ))
-                        .unwrap();
-                }
-            }
-            "sparse_random" => {
-                for _doc_id in 0..num_docs {
-                    let suffix = rng.gen_range(0u64..1000000u64);
-                    let str_val = format!("str_{:07}", suffix);
-
-                    writer
-                        .add_document(doc!(
-                            f_str_fast=>str_val.clone(),
-                            f_str_stored=>str_val,
-                        ))
-                        .unwrap();
-                }
-            }
-            "sparse_sequential" => {
-                for doc_id in 0..num_docs {
-                    let suffix = doc_id as u64;
-                    let str_val = format!("str_{:07}", suffix);
-
-                    writer
-                        .add_document(doc!(
-                            f_str_fast=>str_val.clone(),
-                            f_str_stored=>str_val,
-                        ))
-                        .unwrap();
-                }
-            }
-            _ => {
-                panic!("Unsupported distribution type");
-            }
-        }
-        writer.commit().unwrap();
-    }
-
-    // Prepare reader/searcher once.
-    let reader = index
-        .reader_builder()
-        .reload_policy(ReloadPolicy::Manual)
-        .try_into()
-        .unwrap();
-    let searcher = reader.searcher();
-
-    BenchIndex { index, searcher }
-}
-
-fn main() {
-    // Prepare corpora with varying scenarios
-    let scenarios = vec![
-        (
-            "dense_random_search_low_range".to_string(),
-            1_000_000,
-            "dense_random",
-            0,
-            9,
-        ),
-        (
-            "dense_random_search_high_range".to_string(),
-            1_000_000,
-            "dense_random",
-            990,
-            999,
-        ),
-        (
-            "dense_sequential_search_low_range".to_string(),
-            1_000_000,
-            "dense_sequential",
-            0,
-            9,
-        ),
-        (
-            "dense_sequential_search_high_range".to_string(),
-            1_000_000,
-            "dense_sequential",
-            990,
-            999,
-        ),
-        (
-            "sparse_random_search_low_range".to_string(),
-            1_000_000,
-            "sparse_random",
-            0,
-            9999,
-        ),
-        (
-            "sparse_random_search_high_range".to_string(),
-            1_000_000,
-            "sparse_random",
-            990_000,
-            999_999,
-        ),
-        (
-            "sparse_sequential_search_low_range".to_string(),
-            1_000_000,
-            "sparse_sequential",
-            0,
-            9999,
-        ),
-        (
-            "sparse_sequential_search_high_range".to_string(),
-            1_000_000,
-            "sparse_sequential",
-            990_000,
-            999_999,
-        ),
-    ];
-
-    let mut runner = BenchRunner::new();
-    for (scenario_id, n, distribution, range_low, range_high) in scenarios {
-        let bench_index = build_shared_indices(n, distribution);
-        let mut group = runner.new_group();
-        group.set_name(scenario_id);
-
-        let field = bench_index.searcher.schema().get_field("str_fast").unwrap();
-
-        let (lower_str, upper_str) =
-            if distribution == "dense_sequential" || distribution == "dense_random" {
-                (
-                    format!("str_{:03}", range_low),
-                    format!("str_{:03}", range_high),
-                )
-            } else {
-                (
-                    format!("str_{:07}", range_low),
-                    format!("str_{:07}", range_high),
-                )
-            };
-
-        let lower_term = Term::from_field_text(field, &lower_str);
-        let upper_term = Term::from_field_text(field, &upper_str);
-
-        let query = RangeQuery::new(Bound::Included(lower_term), Bound::Included(upper_term));
-
-        run_benchmark_tasks(&mut group, &bench_index, query, range_low, range_high);
-
-        group.run();
-    }
-}
-
-/// Run all benchmark tasks for a given range query
-fn run_benchmark_tasks(
-    bench_group: &mut BenchGroup,
-    bench_index: &BenchIndex,
-    query: RangeQuery,
-    range_low: u64,
-    range_high: u64,
-) {
-    // Test count of matching documents
-    add_bench_task_count(
-        bench_group,
-        bench_index,
-        query.clone(),
-        range_low,
-        range_high,
-    );
-
-    // Test fetching all DocIds of matching documents
-    add_bench_task_docset(
-        bench_group,
-        bench_index,
-        query.clone(),
-        range_low,
-        range_high,
-    );
-
-    // Test fetching all string fast field values of matching documents
-    add_bench_task_fetch_all_strings(
-        bench_group,
-        bench_index,
-        query.clone(),
-        range_low,
-        range_high,
-    );
-
-    // Test fetching all string values of matching documents through doc() method
-    add_bench_task_fetch_all_strings_from_doc(
-        bench_group,
-        bench_index,
-        query,
-        range_low,
-        range_high,
-    );
-}
-
-fn add_bench_task_count(
-    bench_group: &mut BenchGroup,
-    bench_index: &BenchIndex,
-    query: RangeQuery,
-    range_low: u64,
-    range_high: u64,
-) {
-    let task_name = format!("string_search_count_[{}-{}]", range_low, range_high);
-
-    let search_task = CountSearchTask {
-        searcher: bench_index.searcher.clone(),
-        query,
-    };
-    bench_group.register(task_name, move |_| black_box(search_task.run()));
-}
-
-fn add_bench_task_docset(
-    bench_group: &mut BenchGroup,
-    bench_index: &BenchIndex,
-    query: RangeQuery,
-    range_low: u64,
-    range_high: u64,
-) {
-    let task_name = format!("string_fetch_all_docset_[{}-{}]", range_low, range_high);
-
-    let search_task = DocSetSearchTask {
-        searcher: bench_index.searcher.clone(),
-        query,
-    };
-    bench_group.register(task_name, move |_| black_box(search_task.run()));
-}
-
-fn add_bench_task_fetch_all_strings(
-    bench_group: &mut BenchGroup,
-    bench_index: &BenchIndex,
-    query: RangeQuery,
-    range_low: u64,
-    range_high: u64,
-) {
-    let task_name = format!(
-        "string_fastfield_fetch_all_strings_[{}-{}]",
-        range_low, range_high
-    );
-
-    let search_task = FetchAllStringsSearchTask {
-        searcher: bench_index.searcher.clone(),
-        query,
-    };
-
-    bench_group.register(task_name, move |_| {
-        let result = black_box(search_task.run());
-        result.len()
-    });
-}
-
-fn add_bench_task_fetch_all_strings_from_doc(
-    bench_group: &mut BenchGroup,
-    bench_index: &BenchIndex,
-    query: RangeQuery,
-    range_low: u64,
-    range_high: u64,
-) {
-    let task_name = format!(
-        "string_doc_fetch_all_strings_[{}-{}]",
-        range_low, range_high
-    );
-
-    let search_task = FetchAllStringsFromDocTask {
-        searcher: bench_index.searcher.clone(),
-        query,
-    };
-
-    bench_group.register(task_name, move |_| {
-        let result = black_box(search_task.run());
-        result.len()
-    });
-}
-
-struct CountSearchTask {
-    searcher: Searcher,
-    query: RangeQuery,
-}
-
-impl CountSearchTask {
-    #[inline(never)]
-    pub fn run(&self) -> usize {
-        self.searcher.search(&self.query, &Count).unwrap()
-    }
-}
-
-struct DocSetSearchTask {
-    searcher: Searcher,
-    query: RangeQuery,
-}
-
-impl DocSetSearchTask {
-    #[inline(never)]
-    pub fn run(&self) -> usize {
-        let result = self.searcher.search(&self.query, &DocSetCollector).unwrap();
-        result.len()
-    }
-}
-
-struct FetchAllStringsSearchTask {
-    searcher: Searcher,
-    query: RangeQuery,
-}
-
-impl FetchAllStringsSearchTask {
-    #[inline(never)]
-    pub fn run(&self) -> Vec<String> {
-        let doc_addresses = self.searcher.search(&self.query, &DocSetCollector).unwrap();
-        let mut docs = doc_addresses.into_iter().collect::<Vec<_>>();
-        docs.sort();
-        let mut strings = Vec::with_capacity(docs.len());
-
-        for doc_address in docs {
-            let segment_reader = &self.searcher.segment_readers()[doc_address.segment_ord as usize];
-            let str_column_opt = segment_reader.fast_fields().str("str_fast");
-
-            if let Ok(Some(str_column)) = str_column_opt {
-                let doc_id = doc_address.doc_id;
-                let term_ord = str_column.term_ords(doc_id).next().unwrap();
-                let mut str_buffer = String::new();
-                if str_column.ord_to_str(term_ord, &mut str_buffer).is_ok() {
-                    strings.push(str_buffer);
-                }
-            }
-        }
-
-        strings
-    }
-}
-
-struct FetchAllStringsFromDocTask {
-    searcher: Searcher,
-    query: RangeQuery,
-}
-
-impl FetchAllStringsFromDocTask {
-    #[inline(never)]
-    pub fn run(&self) -> Vec<String> {
-        let doc_addresses = self.searcher.search(&self.query, &DocSetCollector).unwrap();
-        let mut docs = doc_addresses.into_iter().collect::<Vec<_>>();
-        docs.sort();
-        let mut strings = Vec::with_capacity(docs.len());
-
-        let str_stored_field = self
-            .searcher
-            .schema()
-            .get_field("str_stored")
-            .expect("str_stored field should exist");
-
-        for doc_address in docs {
-            // Get the document from the doc store (row store access)
-            if let Ok(doc) = self.searcher.doc::<TantivyDocument>(doc_address) {
-                // Extract string values from the stored field
-                if let Some(field_value) = doc.get_first(str_stored_field) {
-                    if let Some(text) = field_value.as_value().as_str() {
-                        strings.push(text.to_string());
-                    }
-                }
-            }
-        }
-
-        strings
-    }
-}

bitpacker/Cargo.toml

@@ -18,5 +18,5 @@ homepage = "https://github.com/quickwit-oss/tantivy"
 bitpacking = { version = "0.9.2", default-features = false, features = ["bitpacker1x"] }
 
 [dev-dependencies]
-rand = "0.9"
+rand = "0.8"
 proptest = "1"

bitpacker/benches/bench.rs

@@ -4,8 +4,8 @@ extern crate test;
 
 #[cfg(test)]
 mod tests {
-    use rand::rng;
     use rand::seq::IteratorRandom;
+    use rand::thread_rng;
     use tantivy_bitpacker::{BitPacker, BitUnpacker, BlockedBitpacker};
     use test::Bencher;
 
@@ -27,7 +27,7 @@ mod tests {
         let num_els = 1_000_000u32;
         let bit_unpacker = BitUnpacker::new(bit_width);
         let data = create_bitpacked_data(bit_width, num_els);
-        let idxs: Vec<u32> = (0..num_els).choose_multiple(&mut rng(), 100_000);
+        let idxs: Vec<u32> = (0..num_els).choose_multiple(&mut thread_rng(), 100_000);
         b.iter(|| {
             let mut out = 0u64;
             for &idx in &idxs {

columnar/Cargo.toml

@@ -22,7 +22,7 @@ downcast-rs = "2.0.1"
 [dev-dependencies]
 proptest = "1"
 more-asserts = "0.3.1"
-rand = "0.9"
+rand = "0.8"
 binggan = "0.14.0"
 
 [[bench]]

columnar/benches/bench_column_values_get.rs

@@ -9,7 +9,7 @@ use tantivy_columnar::column_values::{CodecType, serialize_and_load_u64_based_co
 fn get_data() -> Vec<u64> {
     let mut rng = StdRng::seed_from_u64(2u64);
     let mut data: Vec<_> = (100..55_000_u64)
-        .map(|num| num + rng.random::<u8>() as u64)
+        .map(|num| num + rng.r#gen::<u8>() as u64)
         .collect();
     data.push(99_000);
     data.insert(1000, 2000);

columnar/benches/bench_create_column_values.rs

@@ -6,7 +6,7 @@ use tantivy_columnar::column_values::{CodecType, serialize_u64_based_column_valu
 fn get_data() -> Vec<u64> {
     let mut rng = StdRng::seed_from_u64(2u64);
     let mut data: Vec<_> = (100..55_000_u64)
-        .map(|num| num + rng.random::<u8>() as u64)
+        .map(|num| num + rng.r#gen::<u8>() as u64)
         .collect();
     data.push(99_000);
     data.insert(1000, 2000);

columnar/benches/bench_optional_index.rs

@@ -8,7 +8,7 @@ const TOTAL_NUM_VALUES: u32 = 1_000_000;
 fn gen_optional_index(fill_ratio: f64) -> OptionalIndex {
     let mut rng: StdRng = StdRng::from_seed([1u8; 32]);
     let vals: Vec<u32> = (0..TOTAL_NUM_VALUES)
-        .map(|_| rng.random_bool(fill_ratio))
+        .map(|_| rng.gen_bool(fill_ratio))
         .enumerate()
         .filter(|(_pos, val)| *val)
         .map(|(pos, _)| pos as u32)
@@ -25,7 +25,7 @@ fn random_range_iterator(
     let mut rng: StdRng = StdRng::from_seed([1u8; 32]);
     let mut current = start;
     std::iter::from_fn(move || {
-        current += rng.random_range(avg_step_size - avg_deviation..=avg_step_size + avg_deviation);
+        current += rng.gen_range(avg_step_size - avg_deviation..=avg_step_size + avg_deviation);
         if current >= end { None } else { Some(current) }
     })
 }

columnar/benches/bench_values_u128.rs

@@ -39,7 +39,7 @@ fn get_data_50percent_item() -> Vec<u128> {
 
     let mut data = vec![];
     for _ in 0..300_000 {
-        let val = rng.random_range(1..=100);
+        let val = rng.gen_range(1..=100);
         data.push(val);
     }
     data.push(SINGLE_ITEM);

columnar/benches/bench_values_u64.rs

@@ -34,7 +34,7 @@ fn get_data_50percent_item() -> Vec<u128> {
 
     let mut data = vec![];
     for _ in 0..300_000 {
-        let val = rng.random_range(1..=100);
+        let val = rng.gen_range(1..=100);
         data.push(val);
     }
     data.push(SINGLE_ITEM);

columnar/src/column_values/u64_based/linear.rs

@@ -268,7 +268,7 @@ mod tests {
 
     #[test]
     fn linear_interpol_fast_field_rand() {
-        let mut rng = rand::rng();
+        let mut rng = rand::thread_rng();
         for _ in 0..50 {
             let mut data = (0..10_000).map(|_| rng.next_u64()).collect::<Vec<_>>();
             create_and_validate::<LinearCodec>(&data, "random");

columnar/src/column_values/u64_based/tests.rs

@@ -122,7 +122,7 @@ pub(crate) fn create_and_validate<TColumnCodec: ColumnCodec>(
     assert_eq!(vals, buffer);
 
     if !vals.is_empty() {
-        let test_rand_idx = rand::rng().random_range(0..=vals.len() - 1);
+        let test_rand_idx = rand::thread_rng().gen_range(0..=vals.len() - 1);
         let expected_positions: Vec<u32> = vals
             .iter()
             .enumerate()

common/Cargo.toml

@@ -21,5 +21,5 @@ serde = { version = "1.0.136", features = ["derive"] }
 [dev-dependencies]
 binggan = "0.14.0"
 proptest = "1.0.0"
-rand = "0.9"
+rand = "0.8.4"
 

common/benches/bench.rs

@@ -1,6 +1,6 @@
 use binggan::{BenchRunner, black_box};
-use rand::rng;
 use rand::seq::IteratorRandom;
+use rand::thread_rng;
 use tantivy_common::{BitSet, TinySet, serialize_vint_u32};
 
 fn bench_vint() {
@@ -17,7 +17,7 @@ fn bench_vint() {
         black_box(out);
     });
 
-    let vals: Vec<u32> = (0..20_000).choose_multiple(&mut rng(), 100_000);
+    let vals: Vec<u32> = (0..20_000).choose_multiple(&mut thread_rng(), 100_000);
     runner.bench_function("bench_vint_rand", move |_| {
         let mut out = 0u64;
         for val in vals.iter().cloned() {

common/src/bitset.rs

@@ -416,7 +416,7 @@ mod tests {
     use std::collections::HashSet;
 
     use ownedbytes::OwnedBytes;
-    use rand::distr::Bernoulli;
+    use rand::distributions::Bernoulli;
     use rand::rngs::StdRng;
     use rand::{Rng, SeedableRng};
 

doc/src/json.md

@@ -60,7 +60,7 @@ At indexing, tantivy will try to interpret number and strings as different type
 priority order.
 
 Numbers will be interpreted as u64, i64 and f64 in that order.
-Strings will be interpreted as rfc3339 dates or simple strings.
+Strings will be interpreted as rfc3999 dates or simple strings.
 
 The first working type is picked and is the only term that is emitted for indexing.
 Note this interpretation happens on a per-document basis, and there is no effort to try to sniff
@@ -81,7 +81,7 @@ Will be interpreted as
 (my_path.my_segment, String, 233) or (my_path.my_segment, u64, 233)
 ```
 
-Likewise, we need to emit two tokens if the query contains an rfc3339 date.
+Likewise, we need to emit two tokens if the query contains an rfc3999 date.
 Indeed the date could have been actually a single token inside the text of a document at ingestion time. Generally speaking, we will always at least emit a string token in query parsing, and sometimes more.
 
 If one more json field is defined, things get even more complicated.

query-grammar/src/query_grammar.rs

@@ -560,7 +560,7 @@ fn range_infallible(inp: &str) -> JResult<&str, UserInputLeaf> {
             (
                 (
                     value((), tag(">=")),
-                    map(word_infallible(")", false), |(bound, err)| {
+                    map(word_infallible("", false), |(bound, err)| {
                         (
                             (
                                 bound
@@ -574,7 +574,7 @@ fn range_infallible(inp: &str) -> JResult<&str, UserInputLeaf> {
                 ),
                 (
                     value((), tag("<=")),
-                    map(word_infallible(")", false), |(bound, err)| {
+                    map(word_infallible("", false), |(bound, err)| {
                         (
                             (
                                 UserInputBound::Unbounded,
@@ -588,7 +588,7 @@ fn range_infallible(inp: &str) -> JResult<&str, UserInputLeaf> {
                 ),
                 (
                     value((), tag(">")),
-                    map(word_infallible(")", false), |(bound, err)| {
+                    map(word_infallible("", false), |(bound, err)| {
                         (
                             (
                                 bound
@@ -602,7 +602,7 @@ fn range_infallible(inp: &str) -> JResult<&str, UserInputLeaf> {
                 ),
                 (
                     value((), tag("<")),
-                    map(word_infallible(")", false), |(bound, err)| {
+                    map(word_infallible("", false), |(bound, err)| {
                         (
                             (
                                 UserInputBound::Unbounded,
@@ -1323,14 +1323,6 @@ mod test {
         test_parse_query_to_ast_helper("<a", "{\"*\" TO \"a\"}");
         test_parse_query_to_ast_helper("<=a", "{\"*\" TO \"a\"]");
         test_parse_query_to_ast_helper("<=bsd", "{\"*\" TO \"bsd\"]");
-
-        test_parse_query_to_ast_helper("(<=42)", "{\"*\" TO \"42\"]");
-        test_parse_query_to_ast_helper("(<=42 )", "{\"*\" TO \"42\"]");
-        test_parse_query_to_ast_helper("(age:>5)", "\"age\":{\"5\" TO \"*\"}");
-        test_parse_query_to_ast_helper(
-            "(title:bar AND age:>12)",
-            "(+\"title\":bar +\"age\":{\"12\" TO \"*\"})",
-        );
     }
 
     #[test]

src/aggregation/agg_result.rs

@@ -10,8 +10,7 @@ use serde::{Deserialize, Serialize};
 
 use super::bucket::GetDocCount;
 use super::metric::{
-    AverageMetricResult, CardinalityMetricResult, ExtendedStats, PercentilesMetricResult,
-    SingleMetricResult, Stats, TopHitsMetricResult,
+    ExtendedStats, PercentilesMetricResult, SingleMetricResult, Stats, TopHitsMetricResult,
 };
 use super::{AggregationError, Key};
 use crate::TantivyError;
@@ -82,8 +81,8 @@ impl AggregationResult {
 #[serde(untagged)]
 /// MetricResult
 pub enum MetricResult {
-    /// Average metric result with sum and count for multi-step merging.
-    Average(AverageMetricResult),
+    /// Average metric result.
+    Average(SingleMetricResult),
     /// Count metric result.
     Count(SingleMetricResult),
     /// Max metric result.
@@ -100,8 +99,8 @@ pub enum MetricResult {
     Percentiles(PercentilesMetricResult),
     /// Top hits metric result
     TopHits(TopHitsMetricResult),
-    /// Cardinality metric result with HLL sketch for multi-step merging.
-    Cardinality(CardinalityMetricResult),
+    /// Cardinality metric result
+    Cardinality(SingleMetricResult),
 }
 
 impl MetricResult {
@@ -120,7 +119,7 @@ impl MetricResult {
             MetricResult::TopHits(_) => Err(TantivyError::AggregationError(
                 AggregationError::InvalidRequest("top_hits can't be used to order".to_string()),
             )),
-            MetricResult::Cardinality(card) => Ok(card.value), // CardinalityMetricResult.value
+            MetricResult::Cardinality(card) => Ok(card.value),
         }
     }
 }

src/aggregation/agg_tests.rs

@@ -1359,10 +1359,10 @@ fn test_aggregation_on_json_object_mixed_types() {
         &serde_json::json!({
           "rangeagg": {
             "buckets": [
-              { "average_in_range": { "value": -20.5, "sum": -20.5, "count": 1 }, "doc_count": 1, "key": "*-3", "to": 3.0 },
-              { "average_in_range": { "value": 10.0, "sum": 10.0, "count": 1 }, "doc_count": 1, "from": 3.0, "key": "3-19", "to": 19.0 },
-              { "average_in_range": { "value": null, "sum": 0.0, "count": 0 }, "doc_count": 0, "from": 19.0, "key": "19-20", "to": 20.0 },
-              { "average_in_range": { "value": null, "sum": 0.0, "count": 0 }, "doc_count": 0, "from": 20.0, "key": "20-*" }
+              { "average_in_range": { "value": -20.5 }, "doc_count": 1, "key": "*-3", "to": 3.0 },
+              { "average_in_range": { "value": 10.0 }, "doc_count": 1, "from": 3.0, "key": "3-19", "to": 19.0 },
+              { "average_in_range": { "value": null }, "doc_count": 0, "from": 19.0, "key": "19-20", "to": 20.0 },
+              { "average_in_range": { "value": null }, "doc_count": 0, "from": 20.0, "key": "20-*" }
             ]
           },
           "termagg": {

src/aggregation/bucket/filter.rs

@@ -838,7 +838,7 @@ mod tests {
         let expected = json!({
             "electronics": {
                 "doc_count": 2,
-                "avg_price": { "value": 899.0, "sum": 1798.0, "count": 2 }  // (999 + 799) / 2
+                "avg_price": { "value": 899.0 }  // (999 + 799) / 2
             }
         });
 
@@ -868,7 +868,7 @@ mod tests {
         let expected = json!({
             "furniture": {
                 "doc_count": 0,
-                "avg_price": { "value": null, "sum": 0.0, "count": 0 }
+                "avg_price": { "value": null }
             }
         });
 
@@ -904,7 +904,7 @@ mod tests {
         let expected = json!({
             "electronics": {
                 "doc_count": 2,
-                "avg_price": { "value": 899.0, "sum": 1798.0, "count": 2 }
+                "avg_price": { "value": 899.0 }
             },
             "in_stock": {
                 "doc_count": 3,  // apple, samsung, penguin
@@ -1000,7 +1000,7 @@ mod tests {
         let expected = json!({
             "premium_electronics": {
                 "doc_count": 1,  // Only apple (999) is >= 800 in tantivy's range semantics
-                "avg_rating": { "value": 4.5, "sum": 4.5, "count": 1 }
+                "avg_rating": { "value": 4.5 }
             }
         });
 
@@ -1032,7 +1032,7 @@ mod tests {
         let expected = json!({
             "in_stock": {
                 "doc_count": 3,  // apple, samsung, penguin
-                "avg_price": { "value": 607.67, "sum": 1823.0, "count": 3 }  // (999 + 799 + 25) / 3 ≈ 607.67
+                "avg_price": { "value": 607.67 }  // (999 + 799 + 25) / 3 ≈ 607.67
             },
             "out_of_stock": {
                 "doc_count": 1,  // nike
@@ -1183,7 +1183,7 @@ mod tests {
                 "doc_count": 4,
                 "electronics_branch": {
                     "doc_count": 2,
-                    "avg_price": { "value": 899.0, "sum": 1798.0, "count": 2 }
+                    "avg_price": { "value": 899.0 }
                 },
                 "in_stock_branch": {
                     "doc_count": 3,
@@ -1259,7 +1259,7 @@ mod tests {
                     "doc_count": 2,  // apple (999), samsung (799)
                     "electronics": {
                         "doc_count": 2,  // both are electronics
-                        "avg_rating": { "value": 4.35, "sum": 8.7, "count": 2 }  // (4.5 + 4.2) / 2
+                        "avg_rating": { "value": 4.35 }  // (4.5 + 4.2) / 2
                     },
                     "in_stock": {
                         "doc_count": 2,  // both are in stock
@@ -1321,12 +1321,12 @@ mod tests {
                         {
                             "key": "samsung",
                             "doc_count": 1,
-                            "avg_price": { "value": 799.0, "sum": 799.0, "count": 1 }
+                            "avg_price": { "value": 799.0 }
                         },
                         {
                             "key": "apple",
                             "doc_count": 1,
-                            "avg_price": { "value": 999.0, "sum": 999.0, "count": 1 }
+                            "avg_price": { "value": 999.0 }
                         }
                     ],
                     "sum_other_doc_count": 0,
@@ -1370,7 +1370,7 @@ mod tests {
                     "sum": 1798.0,
                     "avg": 899.0
                 },
-                "rating_avg": { "value": 4.35, "sum": 8.7, "count": 2 },
+                "rating_avg": { "value": 4.35 },
                 "count": { "value": 2.0 }
             }
         });
@@ -1411,7 +1411,7 @@ mod tests {
         let expected = json!({
             "electronics": {
                 "doc_count": 0,
-                "avg_price": { "value": null, "sum": 0.0, "count": 0 }
+                "avg_price": { "value": null }
             }
         });
 
@@ -1698,15 +1698,13 @@ mod tests {
         let filter_expected = json!({
             "electronics": {
                 "doc_count": 2,
-                "avg_price": { "value": 899.0, "sum": 1798.0, "count": 2 }
+                "avg_price": { "value": 899.0 }
             }
         });
 
         let separate_expected = json!({
             "result": {
-                "value": 899.0,
-                "sum": 1798.0,
-                "count": 2
+                "value": 899.0
             }
         });
 

src/aggregation/bucket/histogram/histogram.rs

@@ -1222,9 +1222,7 @@ mod tests {
             res["histogram"]["buckets"][0],
             json!({
                 "avg": {
-                    "value": Value::Null,
-                    "sum": 0.0,
-                    "count": 0
+                    "value": Value::Null
                 },
                 "doc_count": 0,
                 "key": 2.0,

src/aggregation/intermediate_agg_result.rs

@@ -19,9 +19,8 @@ use super::bucket::{
     GetDocCount, Order, OrderTarget, RangeAggregation, TermsAggregation,
 };
 use super::metric::{
-    AverageMetricResult, CardinalityMetricResult, IntermediateAverage, IntermediateCount,
-    IntermediateExtendedStats, IntermediateMax, IntermediateMin, IntermediateStats,
-    IntermediateSum, PercentilesCollector, TopHitsTopNComputer,
+    IntermediateAverage, IntermediateCount, IntermediateExtendedStats, IntermediateMax,
+    IntermediateMin, IntermediateStats, IntermediateSum, PercentilesCollector, TopHitsTopNComputer,
 };
 use super::segment_agg_result::AggregationLimitsGuard;
 use super::{format_date, AggregationError, Key, SerializedKey};
@@ -326,11 +325,7 @@ impl IntermediateMetricResult {
     fn into_final_metric_result(self, req: &Aggregation) -> MetricResult {
         match self {
             IntermediateMetricResult::Average(intermediate_avg) => {
-                MetricResult::Average(AverageMetricResult {
-                    value: intermediate_avg.finalize(),
-                    sum: intermediate_avg.sum(),
-                    count: intermediate_avg.count(),
-                })
+                MetricResult::Average(intermediate_avg.finalize().into())
             }
             IntermediateMetricResult::Count(intermediate_count) => {
                 MetricResult::Count(intermediate_count.finalize().into())
@@ -358,11 +353,7 @@ impl IntermediateMetricResult {
                 MetricResult::TopHits(top_hits.into_final_result())
             }
             IntermediateMetricResult::Cardinality(cardinality) => {
-                let value = cardinality.finalize();
-                MetricResult::Cardinality(CardinalityMetricResult {
-                    value,
-                    sketch: Some(cardinality),
-                })
+                MetricResult::Cardinality(cardinality.finalize().into())
             }
         }
     }
@@ -829,7 +820,7 @@ impl IntermediateRangeBucketEntry {
         };
 
         // If we have a date type on the histogram buckets, we add the `key_as_string` field as
-        // rfc3339
+        // rfc339
         if column_type == Some(ColumnType::DateTime) {
             if let Some(val) = range_bucket_entry.to {
                 let key_as_string = format_date(val as i64)?;

src/aggregation/metric/average.rs

@@ -63,16 +63,6 @@ impl IntermediateAverage {
     pub fn finalize(&self) -> Option<f64> {
         self.stats.finalize().avg
     }
-
-    /// Returns the sum of all collected values.
-    pub fn sum(&self) -> f64 {
-        self.stats.sum
-    }
-
-    /// Returns the count of all collected values.
-    pub fn count(&self) -> u64 {
-        self.stats.count
-    }
 }
 
 #[cfg(test)]

src/aggregation/metric/cardinality.rs

@@ -340,7 +340,7 @@ impl PartialEq for CardinalityCollector {
 
 impl CardinalityCollector {
     /// Compute the final cardinality estimate.
-    pub fn finalize(&self) -> Option<f64> {
+    pub fn finalize(self) -> Option<f64> {
         Some(self.sketch.clone().count().trunc())
     }
 

src/aggregation/metric/mod.rs

@@ -93,41 +93,6 @@ impl From<Option<f64>> for SingleMetricResult {
     }
 }
 
-/// Average metric result with intermediate data for merging.
-///
-/// Unlike [`SingleMetricResult`], this struct includes the raw `sum` and `count`
-/// values that can be used for multi-step query merging.
-#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
-pub struct AverageMetricResult {
-    /// The computed average value. None if no documents matched.
-    pub value: Option<f64>,
-    /// The sum of all values (for multi-step merging).
-    pub sum: f64,
-    /// The count of all values (for multi-step merging).
-    pub count: u64,
-}
-
-/// Cardinality metric result with computed value and raw HLL sketch for multi-step merging.
-///
-/// The `value` field contains the computed cardinality estimate.
-/// The `sketch` field contains the serialized HyperLogLog++ sketch that can be used
-/// for merging results across multiple query steps.
-#[derive(Clone, Debug, Serialize, Deserialize)]
-pub struct CardinalityMetricResult {
-    /// The computed cardinality estimate.
-    pub value: Option<f64>,
-    /// The serialized HyperLogLog++ sketch for multi-step merging.
-    #[serde(skip_serializing_if = "Option::is_none")]
-    pub sketch: Option<CardinalityCollector>,
-}
-
-impl PartialEq for CardinalityMetricResult {
-    fn eq(&self, other: &Self) -> bool {
-        // Only compare values, not sketch (sketch comparison is complex)
-        self.value == other.value
-    }
-}
-
 /// This is the wrapper of percentile entries, which can be vector or hashmap
 /// depending on if it's keyed or not.
 #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
@@ -146,26 +111,13 @@ pub struct PercentileValuesVecEntry {
     value: f64,
 }
 
-/// Percentiles metric result with computed values and raw sketch for multi-step merging.
+/// Single-metric aggregations use this common result structure.
 ///
-/// The `values` field contains the computed percentile values.
-/// The `sketch` field contains the serialized DDSketch that can be used for merging
-/// results across multiple query steps.
-#[derive(Clone, Debug, Serialize, Deserialize)]
+/// Main reason to wrap it in value is to match elasticsearch output structure.
+#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
 pub struct PercentilesMetricResult {
-    /// The computed percentile values.
+    /// The result of the percentile metric.
     pub values: PercentileValues,
-    /// The serialized DDSketch for multi-step merging.
-    /// This is the raw sketch data that can be deserialized and merged with other sketches.
-    #[serde(skip_serializing_if = "Option::is_none")]
-    pub sketch: Option<PercentilesCollector>,
-}
-
-impl PartialEq for PercentilesMetricResult {
-    fn eq(&self, other: &Self) -> bool {
-        // Only compare values, not sketch (sketch comparison is complex)
-        self.values == other.values
-    }
 }
 
 /// The top_hits metric results entry
@@ -246,105 +198,4 @@ mod tests {
         assert_eq!(aggregations_res_json["price_min"]["value"], 0.0);
         assert_eq!(aggregations_res_json["price_sum"]["value"], 15.0);
     }
-
-    #[test]
-    fn test_average_returns_sum_and_count() {
-        let mut schema_builder = Schema::builder();
-        let field_options = NumericOptions::default().set_fast();
-        let field = schema_builder.add_f64_field("price", field_options);
-        let index = Index::create_in_ram(schema_builder.build());
-        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();
-
-        // Add documents with values 0, 1, 2, 3, 4, 5
-        // sum = 15, count = 6, avg = 2.5
-        for i in 0..6 {
-            index_writer
-                .add_document(doc!(
-                    field => i as f64,
-                ))
-                .unwrap();
-        }
-        index_writer.commit().unwrap();
-
-        let aggregations_json = r#"{ "price_avg": { "avg": { "field": "price" } } }"#;
-        let aggregations: Aggregations = serde_json::from_str(aggregations_json).unwrap();
-        let collector = AggregationCollector::from_aggs(aggregations, Default::default());
-        let reader = index.reader().unwrap();
-        let searcher = reader.searcher();
-        let aggregations_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();
-        let aggregations_res_json = serde_json::to_value(aggregations_res).unwrap();
-
-        // Verify all three fields are present and correct
-        assert_eq!(aggregations_res_json["price_avg"]["value"], 2.5);
-        assert_eq!(aggregations_res_json["price_avg"]["sum"], 15.0);
-        assert_eq!(aggregations_res_json["price_avg"]["count"], 6);
-    }
-
-    #[test]
-    fn test_percentiles_returns_sketch() {
-        let mut schema_builder = Schema::builder();
-        let field_options = NumericOptions::default().set_fast();
-        let field = schema_builder.add_f64_field("latency", field_options);
-        let index = Index::create_in_ram(schema_builder.build());
-        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();
-
-        // Add documents with latency values
-        for i in 0..100 {
-            index_writer
-                .add_document(doc!(
-                    field => i as f64,
-                ))
-                .unwrap();
-        }
-        index_writer.commit().unwrap();
-
-        let aggregations_json =
-            r#"{ "latency_percentiles": { "percentiles": { "field": "latency" } } }"#;
-        let aggregations: Aggregations = serde_json::from_str(aggregations_json).unwrap();
-        let collector = AggregationCollector::from_aggs(aggregations, Default::default());
-        let reader = index.reader().unwrap();
-        let searcher = reader.searcher();
-        let aggregations_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();
-        let aggregations_res_json = serde_json::to_value(aggregations_res).unwrap();
-
-        // Verify percentile values are present
-        assert!(aggregations_res_json["latency_percentiles"]["values"].is_object());
-        // Verify sketch is present (serialized DDSketch)
-        assert!(aggregations_res_json["latency_percentiles"]["sketch"].is_object());
-    }
-
-    #[test]
-    fn test_cardinality_returns_sketch() {
-        let mut schema_builder = Schema::builder();
-        let field_options = NumericOptions::default().set_fast();
-        let field = schema_builder.add_u64_field("user_id", field_options);
-        let index = Index::create_in_ram(schema_builder.build());
-        let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();
-
-        // Add documents with some duplicate user_ids
-        for i in 0..50 {
-            index_writer
-                .add_document(doc!(
-                    field => (i % 10) as u64,  // 10 unique values
-                ))
-                .unwrap();
-        }
-        index_writer.commit().unwrap();
-
-        let aggregations_json = r#"{ "unique_users": { "cardinality": { "field": "user_id" } } }"#;
-        let aggregations: Aggregations = serde_json::from_str(aggregations_json).unwrap();
-        let collector = AggregationCollector::from_aggs(aggregations, Default::default());
-        let reader = index.reader().unwrap();
-        let searcher = reader.searcher();
-        let aggregations_res: AggregationResults = searcher.search(&AllQuery, &collector).unwrap();
-        let aggregations_res_json = serde_json::to_value(aggregations_res).unwrap();
-
-        // Verify cardinality value is present and approximately correct
-        let cardinality = aggregations_res_json["unique_users"]["value"]
-            .as_f64()
-            .unwrap();
-        assert!(cardinality >= 9.0 && cardinality <= 11.0); // HLL is approximate
-                                                            // Verify sketch is present (serialized HyperLogLog++)
-        assert!(aggregations_res_json["unique_users"]["sketch"].is_object());
-    }
 }

src/aggregation/metric/percentiles.rs

@@ -178,9 +178,6 @@ fn format_percentile(percentile: f64) -> String {
 impl PercentilesCollector {
     /// Convert result into final result. This will query the quantils from the underlying quantil
     /// collector.
-    ///
-    /// The result includes both the computed percentile values and the raw DDSketch
-    /// for multi-step query merging.
     pub fn into_final_result(self, req: &PercentilesAggregationReq) -> PercentilesMetricResult {
         let percentiles: &[f64] = req
             .percents
@@ -213,15 +210,7 @@ impl PercentilesCollector {
                     .collect(),
             )
         };
-        PercentilesMetricResult {
-            values,
-            sketch: Some(self),
-        }
-    }
-
-    /// Returns a reference to the underlying DDSketch.
-    pub fn sketch(&self) -> &sketches_ddsketch::DDSketch {
-        &self.sketch
+        PercentilesMetricResult { values }
     }
 
     fn new() -> Self {

src/codec/mod.rs

@@ -0,0 +1,49 @@
+mod postings;
+mod standard;
+
+use std::borrow::Cow;
+
+use serde::{Deserialize, Serialize};
+pub use standard::StandardCodec;
+
+pub trait Codec: Clone + std::fmt::Debug + Send + Sync + 'static {
+    type PostingsCodec;
+
+    const NAME: &'static str;
+
+    fn from_json_props(json_value: &serde_json::Value) -> crate::Result<Self>;
+    fn to_json_props(&self) -> serde_json::Value;
+}
+
+#[derive(Serialize, Deserialize, Clone, Debug)]
+pub struct CodecConfiguration {
+    name: Cow<'static, str>,
+    #[serde(default, skip_serializing_if = "serde_json::Value::is_null")]
+    props: serde_json::Value,
+}
+
+impl CodecConfiguration {
+    pub fn from_codec<C: Codec>(codec: &C) -> Self {
+        CodecConfiguration {
+            name: Cow::Borrowed(C::NAME),
+            props: codec.to_json_props(),
+        }
+    }
+
+    pub fn to_codec<C: Codec>(&self) -> crate::Result<C> {
+        if self.name != C::NAME {
+            return Err(crate::TantivyError::InvalidArgument(format!(
+                "Codec name mismatch: expected {}, got {}",
+                C::NAME,
+                self.name
+            )));
+        }
+        C::from_json_props(&self.props)
+    }
+}
+
+impl Default for CodecConfiguration {
+    fn default() -> Self {
+        CodecConfiguration::from_codec(&StandardCodec)
+    }
+}

src/codec/postings/mod.rs

@@ -0,0 +1,23 @@
+use std::io;
+
+use crate::fieldnorm::FieldNormReader;
+use crate::schema::IndexRecordOption;
+use crate::{DocId, Score};
+
+pub trait PostingsCodec {
+    type PostingsSerializer: PostingsSerializer;
+}
+
+pub trait PostingsSerializer {
+    fn new(
+        avg_fieldnorm: Score,
+        mode: IndexRecordOption,
+        fieldnorm_reader: Option<FieldNormReader>,
+    ) -> Self;
+
+    fn new_term(&mut self, term_doc_freq: u32, record_term_freq: bool);
+
+    fn write_doc(&mut self, doc_id: DocId, term_freq: u32);
+
+    fn close_term(&mut self, doc_freq: u32, wrt: &mut impl io::Write) -> io::Result<()>;
+}

src/codec/standard/mod.rs

@@ -0,0 +1,29 @@
+use serde::{Deserialize, Serialize};
+
+use crate::codec::standard::postings::StandardPostingsCodec;
+use crate::codec::Codec;
+
+mod postings;
+
+#[derive(Debug, Default, Clone, Serialize, Deserialize)]
+pub struct StandardCodec;
+
+impl Codec for StandardCodec {
+    type PostingsCodec = StandardPostingsCodec;
+
+    const NAME: &'static str = "standard";
+
+    fn from_json_props(json_value: &serde_json::Value) -> crate::Result<Self> {
+        if !json_value.is_null() {
+            return Err(crate::TantivyError::InvalidArgument(format!(
+                "Codec property for the StandardCodec are unexpected. expected null, got {}",
+                json_value.as_str().unwrap_or("null")
+            )));
+        }
+        Ok(StandardCodec)
+    }
+
+    fn to_json_props(&self) -> serde_json::Value {
+        serde_json::Value::Null
+    }
+}

src/codec/standard/postings/block.rs

@@ -0,0 +1,50 @@
+use crate::postings::compression::COMPRESSION_BLOCK_SIZE;
+use crate::DocId;
+
+pub struct Block {
+    doc_ids: [DocId; COMPRESSION_BLOCK_SIZE],
+    term_freqs: [u32; COMPRESSION_BLOCK_SIZE],
+    len: usize,
+}
+
+impl Block {
+    pub fn new() -> Self {
+        Block {
+            doc_ids: [0u32; COMPRESSION_BLOCK_SIZE],
+            term_freqs: [0u32; COMPRESSION_BLOCK_SIZE],
+            len: 0,
+        }
+    }
+
+    pub fn doc_ids(&self) -> &[DocId] {
+        &self.doc_ids[..self.len]
+    }
+
+    pub fn term_freqs(&self) -> &[u32] {
+        &self.term_freqs[..self.len]
+    }
+
+    pub fn clear(&mut self) {
+        self.len = 0;
+    }
+
+    pub fn append_doc(&mut self, doc: DocId, term_freq: u32) {
+        let len = self.len;
+        self.doc_ids[len] = doc;
+        self.term_freqs[len] = term_freq;
+        self.len = len + 1;
+    }
+
+    pub fn is_full(&self) -> bool {
+        self.len == COMPRESSION_BLOCK_SIZE
+    }
+
+    pub fn is_empty(&self) -> bool {
+        self.len == 0
+    }
+
+    pub fn last_doc(&self) -> DocId {
+        assert_eq!(self.len, COMPRESSION_BLOCK_SIZE);
+        self.doc_ids[COMPRESSION_BLOCK_SIZE - 1]
+    }
+}

src/codec/standard/postings/mod.rs

@@ -0,0 +1,13 @@
+use crate::codec::postings::PostingsCodec;
+
+mod block;
+mod postings_serializer;
+mod skip;
+
+pub use postings_serializer::StandardPostingsSerializer;
+
+pub struct StandardPostingsCodec;
+
+impl PostingsCodec for StandardPostingsCodec {
+    type PostingsSerializer = StandardPostingsSerializer;
+}

src/codec/standard/postings/postings_serializer.rs

@@ -0,0 +1,187 @@
+use std::cmp::Ordering;
+use std::io::{self, Write as _};
+
+use common::{BinarySerializable as _, VInt};
+
+use crate::codec::postings::PostingsSerializer;
+use crate::codec::standard::postings::block::Block;
+use crate::codec::standard::postings::skip::SkipSerializer;
+use crate::fieldnorm::FieldNormReader;
+use crate::postings::compression::{BlockEncoder, VIntEncoder as _, COMPRESSION_BLOCK_SIZE};
+use crate::query::Bm25Weight;
+use crate::schema::IndexRecordOption;
+use crate::{DocId, Score};
+
+pub struct StandardPostingsSerializer {
+    last_doc_id_encoded: u32,
+
+    block_encoder: BlockEncoder,
+    block: Box<Block>,
+
+    postings_write: Vec<u8>,
+    skip_write: SkipSerializer,
+
+    mode: IndexRecordOption,
+    fieldnorm_reader: Option<FieldNormReader>,
+
+    bm25_weight: Option<Bm25Weight>,
+    avg_fieldnorm: Score, /* Average number of term in the field for that segment.
+                           * this value is used to compute the block wand information. */
+    term_has_freq: bool,
+}
+
+impl PostingsSerializer for StandardPostingsSerializer {
+    fn new(
+        avg_fieldnorm: Score,
+        mode: IndexRecordOption,
+        fieldnorm_reader: Option<FieldNormReader>,
+    ) -> StandardPostingsSerializer {
+        Self {
+            block_encoder: BlockEncoder::new(),
+            block: Box::new(Block::new()),
+
+            postings_write: Vec::new(),
+            skip_write: SkipSerializer::new(),
+
+            last_doc_id_encoded: 0u32,
+            mode,
+
+            fieldnorm_reader,
+            bm25_weight: None,
+            avg_fieldnorm,
+            term_has_freq: false,
+        }
+    }
+
+    fn new_term(&mut self, term_doc_freq: u32, record_term_freq: bool) {
+        self.bm25_weight = None;
+
+        self.term_has_freq = self.mode.has_freq() && record_term_freq;
+        if !self.term_has_freq {
+            return;
+        }
+
+        let num_docs_in_segment: u64 =
+            if let Some(fieldnorm_reader) = self.fieldnorm_reader.as_ref() {
+                fieldnorm_reader.num_docs() as u64
+            } else {
+                return;
+            };
+
+        if num_docs_in_segment == 0 {
+            return;
+        }
+
+        self.bm25_weight = Some(Bm25Weight::for_one_term_without_explain(
+            term_doc_freq as u64,
+            num_docs_in_segment,
+            self.avg_fieldnorm,
+        ));
+    }
+
+    fn write_doc(&mut self, doc_id: DocId, term_freq: u32) {
+        self.block.append_doc(doc_id, term_freq);
+        if self.block.is_full() {
+            self.write_block();
+        }
+    }
+
+    fn close_term(
+        &mut self,
+        doc_freq: u32,
+        output_write: &mut impl std::io::Write,
+    ) -> io::Result<()> {
+        if !self.block.is_empty() {
+            // we have doc ids waiting to be written
+            // this happens when the number of doc ids is
+            // not a perfect multiple of our block size.
+            //
+            // In that case, the remaining part is encoded
+            // using variable int encoding.
+            {
+                let block_encoded = self
+                    .block_encoder
+                    .compress_vint_sorted(self.block.doc_ids(), self.last_doc_id_encoded);
+                self.postings_write.write_all(block_encoded)?;
+            }
+            // ... Idem for term frequencies
+            if self.term_has_freq {
+                let block_encoded = self
+                    .block_encoder
+                    .compress_vint_unsorted(self.block.term_freqs());
+                self.postings_write.write_all(block_encoded)?;
+            }
+            self.block.clear();
+        }
+        if doc_freq >= COMPRESSION_BLOCK_SIZE as u32 {
+            let skip_data = self.skip_write.data();
+            VInt(skip_data.len() as u64).serialize(output_write)?;
+            output_write.write_all(skip_data)?;
+        }
+        output_write.write_all(&self.postings_write[..])?;
+        self.skip_write.clear();
+        self.postings_write.clear();
+        self.bm25_weight = None;
+        Ok(())
+    }
+}
+
+impl StandardPostingsSerializer {
+    fn write_block(&mut self) {
+        {
+            // encode the doc ids
+            let (num_bits, block_encoded): (u8, &[u8]) = self
+                .block_encoder
+                .compress_block_sorted(self.block.doc_ids(), self.last_doc_id_encoded);
+            self.last_doc_id_encoded = self.block.last_doc();
+            self.skip_write
+                .write_doc(self.last_doc_id_encoded, num_bits);
+            // last el block 0, offset block 1,
+            self.postings_write.extend(block_encoded);
+        }
+        if self.term_has_freq {
+            let (num_bits, block_encoded): (u8, &[u8]) = self
+                .block_encoder
+                .compress_block_unsorted(self.block.term_freqs(), true);
+            self.postings_write.extend(block_encoded);
+            self.skip_write.write_term_freq(num_bits);
+            if self.mode.has_positions() {
+                // We serialize the sum of term freqs within the skip information
+                // in order to navigate through positions.
+                let sum_freq = self.block.term_freqs().iter().cloned().sum();
+                self.skip_write.write_total_term_freq(sum_freq);
+            }
+            let mut blockwand_params = (0u8, 0u32);
+            if let Some(bm25_weight) = self.bm25_weight.as_ref() {
+                if let Some(fieldnorm_reader) = self.fieldnorm_reader.as_ref() {
+                    let docs = self.block.doc_ids().iter().cloned();
+                    let term_freqs = self.block.term_freqs().iter().cloned();
+                    let fieldnorms = docs.map(|doc| fieldnorm_reader.fieldnorm_id(doc));
+                    blockwand_params = fieldnorms
+                        .zip(term_freqs)
+                        .max_by(
+                            |(left_fieldnorm_id, left_term_freq),
+                             (right_fieldnorm_id, right_term_freq)| {
+                                let left_score =
+                                    bm25_weight.tf_factor(*left_fieldnorm_id, *left_term_freq);
+                                let right_score =
+                                    bm25_weight.tf_factor(*right_fieldnorm_id, *right_term_freq);
+                                left_score
+                                    .partial_cmp(&right_score)
+                                    .unwrap_or(Ordering::Equal)
+                            },
+                        )
+                        .unwrap();
+                }
+            }
+            let (fieldnorm_id, term_freq) = blockwand_params;
+            self.skip_write.write_blockwand_max(fieldnorm_id, term_freq);
+        }
+        self.block.clear();
+    }
+
+    fn clear(&mut self) {
+        self.block.clear();
+        self.last_doc_id_encoded = 0;
+    }
+}

src/codec/standard/postings/skip.rs

@@ -0,0 +1,448 @@
+use crate::directory::OwnedBytes;
+use crate::postings::compression::{compressed_block_size, COMPRESSION_BLOCK_SIZE};
+use crate::query::Bm25Weight;
+use crate::schema::IndexRecordOption;
+use crate::{DocId, Score, TERMINATED};
+
+// doc num bits uses the following encoding:
+// given 0b a b cdefgh
+//         |1|2|3|  4  |
+// - 1: unused
+// - 2: is delta-1 encoded. 0 if not, 1, if yes
+// - 3: unused
+// - 4: a 5 bit number in 0..32, the actual bitwidth. Bitpacking could in theory say this is 32
+//   (requiring a 6th bit), but the biggest doc_id we can want to encode is TERMINATED-1, which can
+//   be represented on 31b without delta encoding.
+fn encode_bitwidth(bitwidth: u8, delta_1: bool) -> u8 {
+    assert!(bitwidth < 32);
+    bitwidth | ((delta_1 as u8) << 6)
+}
+
+fn decode_bitwidth(raw_bitwidth: u8) -> (u8, bool) {
+    let delta_1 = ((raw_bitwidth >> 6) & 1) != 0;
+    let bitwidth = raw_bitwidth & 0x1f;
+    (bitwidth, delta_1)
+}
+
+#[inline]
+fn encode_block_wand_max_tf(max_tf: u32) -> u8 {
+    max_tf.min(u8::MAX as u32) as u8
+}
+
+#[inline]
+fn decode_block_wand_max_tf(max_tf_code: u8) -> u32 {
+    if max_tf_code == u8::MAX {
+        u32::MAX
+    } else {
+        max_tf_code as u32
+    }
+}
+
+#[inline]
+fn read_u32(data: &[u8]) -> u32 {
+    u32::from_le_bytes(data[..4].try_into().unwrap())
+}
+
+#[inline]
+fn write_u32(val: u32, buf: &mut Vec<u8>) {
+    buf.extend_from_slice(&val.to_le_bytes());
+}
+
+pub struct SkipSerializer {
+    buffer: Vec<u8>,
+}
+
+impl SkipSerializer {
+    pub fn new() -> SkipSerializer {
+        SkipSerializer { buffer: Vec::new() }
+    }
+
+    pub fn write_doc(&mut self, last_doc: DocId, doc_num_bits: u8) {
+        write_u32(last_doc, &mut self.buffer);
+        self.buffer.push(encode_bitwidth(doc_num_bits, true));
+    }
+
+    pub fn write_term_freq(&mut self, tf_num_bits: u8) {
+        self.buffer.push(tf_num_bits);
+    }
+
+    pub fn write_total_term_freq(&mut self, tf_sum: u32) {
+        write_u32(tf_sum, &mut self.buffer);
+    }
+
+    pub fn write_blockwand_max(&mut self, fieldnorm_id: u8, term_freq: u32) {
+        let block_wand_tf = encode_block_wand_max_tf(term_freq);
+        self.buffer
+            .extend_from_slice(&[fieldnorm_id, block_wand_tf]);
+    }
+
+    pub fn data(&self) -> &[u8] {
+        &self.buffer[..]
+    }
+
+    pub fn clear(&mut self) {
+        self.buffer.clear();
+    }
+}
+
+#[derive(Clone)]
+pub(crate) struct SkipReader {
+    last_doc_in_block: DocId,
+    pub(crate) last_doc_in_previous_block: DocId,
+    owned_read: OwnedBytes,
+    skip_info: IndexRecordOption,
+    byte_offset: usize,
+    remaining_docs: u32, // number of docs remaining, including the
+    // documents in the current block.
+    block_info: BlockInfo,
+
+    position_offset: u64,
+}
+
+#[derive(Clone, Eq, PartialEq, Copy, Debug)]
+pub(crate) enum BlockInfo {
+    BitPacked {
+        doc_num_bits: u8,
+        strict_delta_encoded: bool,
+        tf_num_bits: u8,
+        tf_sum: u32,
+        block_wand_fieldnorm_id: u8,
+        block_wand_term_freq: u32,
+    },
+    VInt {
+        num_docs: u32,
+    },
+}
+
+impl Default for BlockInfo {
+    fn default() -> Self {
+        BlockInfo::VInt { num_docs: 0u32 }
+    }
+}
+
+impl SkipReader {
+    pub fn new(data: OwnedBytes, doc_freq: u32, skip_info: IndexRecordOption) -> SkipReader {
+        let mut skip_reader = SkipReader {
+            last_doc_in_block: if doc_freq >= COMPRESSION_BLOCK_SIZE as u32 {
+                0
+            } else {
+                TERMINATED
+            },
+            last_doc_in_previous_block: 0u32,
+            owned_read: data,
+            skip_info,
+            block_info: BlockInfo::VInt { num_docs: doc_freq },
+            byte_offset: 0,
+            remaining_docs: doc_freq,
+            position_offset: 0u64,
+        };
+        if doc_freq >= COMPRESSION_BLOCK_SIZE as u32 {
+            skip_reader.read_block_info();
+        }
+        skip_reader
+    }
+
+    pub fn reset(&mut self, data: OwnedBytes, doc_freq: u32) {
+        self.last_doc_in_block = if doc_freq >= COMPRESSION_BLOCK_SIZE as u32 {
+            0
+        } else {
+            TERMINATED
+        };
+        self.last_doc_in_previous_block = 0u32;
+        self.owned_read = data;
+        self.block_info = BlockInfo::VInt { num_docs: doc_freq };
+        self.byte_offset = 0;
+        self.remaining_docs = doc_freq;
+        self.position_offset = 0u64;
+        if doc_freq >= COMPRESSION_BLOCK_SIZE as u32 {
+            self.read_block_info();
+        }
+    }
+
+    // Returns the block max score for this block if available.
+    //
+    // The block max score is available for all full bitpacked block,
+    // but no available for the last VInt encoded incomplete block.
+    pub fn block_max_score(&self, bm25_weight: &Bm25Weight) -> Option<Score> {
+        match self.block_info {
+            BlockInfo::BitPacked {
+                block_wand_fieldnorm_id,
+                block_wand_term_freq,
+                ..
+            } => Some(bm25_weight.score(block_wand_fieldnorm_id, block_wand_term_freq)),
+            BlockInfo::VInt { .. } => None,
+        }
+    }
+
+    pub(crate) fn last_doc_in_block(&self) -> DocId {
+        self.last_doc_in_block
+    }
+
+    pub fn position_offset(&self) -> u64 {
+        self.position_offset
+    }
+
+    #[inline]
+    pub fn byte_offset(&self) -> usize {
+        self.byte_offset
+    }
+
+    fn read_block_info(&mut self) {
+        let bytes = self.owned_read.as_slice();
+        let advance_len: usize;
+        self.last_doc_in_block = read_u32(bytes);
+        let (doc_num_bits, strict_delta_encoded) = decode_bitwidth(bytes[4]);
+        match self.skip_info {
+            IndexRecordOption::Basic => {
+                advance_len = 5;
+                self.block_info = BlockInfo::BitPacked {
+                    doc_num_bits,
+                    strict_delta_encoded,
+                    tf_num_bits: 0,
+                    tf_sum: 0,
+                    block_wand_fieldnorm_id: 0,
+                    block_wand_term_freq: 0,
+                };
+            }
+            IndexRecordOption::WithFreqs => {
+                let tf_num_bits = bytes[5];
+                let block_wand_fieldnorm_id = bytes[6];
+                let block_wand_term_freq = decode_block_wand_max_tf(bytes[7]);
+                advance_len = 8;
+                self.block_info = BlockInfo::BitPacked {
+                    doc_num_bits,
+                    strict_delta_encoded,
+                    tf_num_bits,
+                    tf_sum: 0,
+                    block_wand_fieldnorm_id,
+                    block_wand_term_freq,
+                };
+            }
+            IndexRecordOption::WithFreqsAndPositions => {
+                let tf_num_bits = bytes[5];
+                let tf_sum = read_u32(&bytes[6..10]);
+                let block_wand_fieldnorm_id = bytes[10];
+                let block_wand_term_freq = decode_block_wand_max_tf(bytes[11]);
+                advance_len = 12;
+                self.block_info = BlockInfo::BitPacked {
+                    doc_num_bits,
+                    strict_delta_encoded,
+                    tf_num_bits,
+                    tf_sum,
+                    block_wand_fieldnorm_id,
+                    block_wand_term_freq,
+                };
+            }
+        }
+        self.owned_read.advance(advance_len);
+    }
+
+    pub fn block_info(&self) -> BlockInfo {
+        self.block_info
+    }
+
+    /// Advance the skip reader to the block that may contain the target.
+    ///
+    /// If the target is larger than all documents, the skip_reader
+    /// then advance to the last Variable In block.
+    pub fn seek(&mut self, target: DocId) -> bool {
+        if self.last_doc_in_block() >= target {
+            return false;
+        }
+        loop {
+            self.advance();
+            if self.last_doc_in_block() >= target {
+                return true;
+            }
+        }
+    }
+
+    pub fn advance(&mut self) {
+        match self.block_info {
+            BlockInfo::BitPacked {
+                doc_num_bits,
+                tf_num_bits,
+                tf_sum,
+                ..
+            } => {
+                self.remaining_docs -= COMPRESSION_BLOCK_SIZE as u32;
+                self.byte_offset += compressed_block_size(doc_num_bits + tf_num_bits);
+                self.position_offset += tf_sum as u64;
+            }
+            BlockInfo::VInt { num_docs } => {
+                debug_assert_eq!(num_docs, self.remaining_docs);
+                self.remaining_docs = 0;
+                self.byte_offset = usize::MAX;
+            }
+        }
+        self.last_doc_in_previous_block = self.last_doc_in_block;
+        if self.remaining_docs >= COMPRESSION_BLOCK_SIZE as u32 {
+            self.read_block_info();
+        } else {
+            self.last_doc_in_block = TERMINATED;
+            self.block_info = BlockInfo::VInt {
+                num_docs: self.remaining_docs,
+            };
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+
+    use super::{
+        decode_bitwidth, encode_bitwidth, BlockInfo, IndexRecordOption, SkipReader, SkipSerializer,
+    };
+    use crate::directory::OwnedBytes;
+    use crate::postings::compression::COMPRESSION_BLOCK_SIZE;
+
+    #[test]
+    fn test_encode_block_wand_max_tf() {
+        for tf in 0..255 {
+            assert_eq!(super::encode_block_wand_max_tf(tf), tf as u8);
+        }
+        for &tf in &[255, 256, 1_000_000, u32::MAX] {
+            assert_eq!(super::encode_block_wand_max_tf(tf), 255);
+        }
+    }
+
+    #[test]
+    fn test_decode_block_wand_max_tf() {
+        for tf in 0..255 {
+            assert_eq!(super::decode_block_wand_max_tf(tf), tf as u32);
+        }
+        assert_eq!(super::decode_block_wand_max_tf(255), u32::MAX);
+    }
+
+    #[test]
+    fn test_skip_with_freq() {
+        let buf = {
+            let mut skip_serializer = SkipSerializer::new();
+            skip_serializer.write_doc(1u32, 2u8);
+            skip_serializer.write_term_freq(3u8);
+            skip_serializer.write_blockwand_max(13u8, 3u32);
+            skip_serializer.write_doc(5u32, 5u8);
+            skip_serializer.write_term_freq(2u8);
+            skip_serializer.write_blockwand_max(8u8, 2u32);
+            skip_serializer.data().to_owned()
+        };
+        let doc_freq = 3u32 + (COMPRESSION_BLOCK_SIZE * 2) as u32;
+        let mut skip_reader =
+            SkipReader::new(OwnedBytes::new(buf), doc_freq, IndexRecordOption::WithFreqs);
+        assert_eq!(skip_reader.last_doc_in_block(), 1u32);
+        assert_eq!(
+            skip_reader.block_info,
+            BlockInfo::BitPacked {
+                doc_num_bits: 2u8,
+                strict_delta_encoded: true,
+                tf_num_bits: 3u8,
+                tf_sum: 0,
+                block_wand_fieldnorm_id: 13,
+                block_wand_term_freq: 3
+            }
+        );
+        skip_reader.advance();
+        assert_eq!(skip_reader.last_doc_in_block(), 5u32);
+        assert_eq!(
+            skip_reader.block_info(),
+            BlockInfo::BitPacked {
+                doc_num_bits: 5u8,
+                strict_delta_encoded: true,
+                tf_num_bits: 2u8,
+                tf_sum: 0,
+                block_wand_fieldnorm_id: 8,
+                block_wand_term_freq: 2
+            }
+        );
+        skip_reader.advance();
+        assert_eq!(skip_reader.block_info(), BlockInfo::VInt { num_docs: 3u32 });
+        skip_reader.advance();
+        assert_eq!(skip_reader.block_info(), BlockInfo::VInt { num_docs: 0u32 });
+        skip_reader.advance();
+        assert_eq!(skip_reader.block_info(), BlockInfo::VInt { num_docs: 0u32 });
+    }
+
+    #[test]
+    fn test_skip_no_freq() {
+        let buf = {
+            let mut skip_serializer = SkipSerializer::new();
+            skip_serializer.write_doc(1u32, 2u8);
+            skip_serializer.write_doc(5u32, 5u8);
+            skip_serializer.data().to_owned()
+        };
+        let doc_freq = 3u32 + (COMPRESSION_BLOCK_SIZE * 2) as u32;
+        let mut skip_reader =
+            SkipReader::new(OwnedBytes::new(buf), doc_freq, IndexRecordOption::Basic);
+        assert_eq!(skip_reader.last_doc_in_block(), 1u32);
+        assert_eq!(
+            skip_reader.block_info(),
+            BlockInfo::BitPacked {
+                doc_num_bits: 2u8,
+                strict_delta_encoded: true,
+                tf_num_bits: 0,
+                tf_sum: 0u32,
+                block_wand_fieldnorm_id: 0,
+                block_wand_term_freq: 0
+            }
+        );
+        skip_reader.advance();
+        assert_eq!(skip_reader.last_doc_in_block(), 5u32);
+        assert_eq!(
+            skip_reader.block_info(),
+            BlockInfo::BitPacked {
+                doc_num_bits: 5u8,
+                strict_delta_encoded: true,
+                tf_num_bits: 0,
+                tf_sum: 0u32,
+                block_wand_fieldnorm_id: 0,
+                block_wand_term_freq: 0
+            }
+        );
+        skip_reader.advance();
+        assert_eq!(skip_reader.block_info(), BlockInfo::VInt { num_docs: 3u32 });
+        skip_reader.advance();
+        assert_eq!(skip_reader.block_info(), BlockInfo::VInt { num_docs: 0u32 });
+        skip_reader.advance();
+        assert_eq!(skip_reader.block_info(), BlockInfo::VInt { num_docs: 0u32 });
+    }
+
+    #[test]
+    fn test_skip_multiple_of_block_size() {
+        let buf = {
+            let mut skip_serializer = SkipSerializer::new();
+            skip_serializer.write_doc(1u32, 2u8);
+            skip_serializer.data().to_owned()
+        };
+        let doc_freq = COMPRESSION_BLOCK_SIZE as u32;
+        let mut skip_reader =
+            SkipReader::new(OwnedBytes::new(buf), doc_freq, IndexRecordOption::Basic);
+        assert_eq!(skip_reader.last_doc_in_block(), 1u32);
+        assert_eq!(
+            skip_reader.block_info(),
+            BlockInfo::BitPacked {
+                doc_num_bits: 2u8,
+                strict_delta_encoded: true,
+                tf_num_bits: 0,
+                tf_sum: 0u32,
+                block_wand_fieldnorm_id: 0,
+                block_wand_term_freq: 0
+            }
+        );
+        skip_reader.advance();
+        assert_eq!(skip_reader.block_info(), BlockInfo::VInt { num_docs: 0u32 });
+    }
+
+    #[test]
+    fn test_encode_decode_bitwidth() {
+        for bitwidth in 0..32 {
+            for delta_1 in [false, true] {
+                assert_eq!(
+                    (bitwidth, delta_1),
+                    decode_bitwidth(encode_bitwidth(bitwidth, delta_1))
+                );
+            }
+        }
+        assert_eq!(0b01000010, encode_bitwidth(0b10, true));
+        assert_eq!(0b00000010, encode_bitwidth(0b10, false));
+    }
+}

src/collector/facet_collector.rs

@@ -486,9 +486,9 @@ mod tests {
     use std::collections::BTreeSet;
 
     use columnar::Dictionary;
-    use rand::distr::Uniform;
+    use rand::distributions::Uniform;
     use rand::prelude::SliceRandom;
-    use rand::{rng, Rng};
+    use rand::{thread_rng, Rng};
 
     use super::{FacetCollector, FacetCounts};
     use crate::collector::facet_collector::compress_mapping;
@@ -731,7 +731,7 @@ mod tests {
         let schema = schema_builder.build();
         let index = Index::create_in_ram(schema);
 
-        let uniform = Uniform::new_inclusive(1, 100_000).unwrap();
+        let uniform = Uniform::new_inclusive(1, 100_000);
         let mut docs: Vec<TantivyDocument> =
             vec![("a", 10), ("b", 100), ("c", 7), ("d", 12), ("e", 21)]
                 .into_iter()
@@ -741,11 +741,14 @@ mod tests {
                     std::iter::repeat_n(doc, count)
                 })
                 .map(|mut doc| {
-                    doc.add_facet(facet_field, &format!("/facet/{}", rng().sample(uniform)));
+                    doc.add_facet(
+                        facet_field,
+                        &format!("/facet/{}", thread_rng().sample(uniform)),
+                    );
                     doc
                 })
                 .collect();
-        docs[..].shuffle(&mut rng());
+        docs[..].shuffle(&mut thread_rng());
 
         let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();
         for doc in docs {
@@ -819,8 +822,8 @@ mod tests {
 #[cfg(all(test, feature = "unstable"))]
 mod bench {
 
-    use rand::rng;
     use rand::seq::SliceRandom;
+    use rand::thread_rng;
     use test::Bencher;
 
     use crate::collector::FacetCollector;
@@ -843,7 +846,7 @@ mod bench {
             }
         }
         // 40425 docs
-        docs[..].shuffle(&mut rng());
+        docs[..].shuffle(&mut thread_rng());
 
         let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();
         for doc in docs {

src/collector/sort_key_top_collector.rs

@@ -160,7 +160,7 @@ mod tests {
         expected: &[(crate::Score, usize)],
     ) {
         let mut vals: Vec<(crate::Score, usize)> = (0..10).map(|val| (val as f32, val)).collect();
-        vals.shuffle(&mut rand::rng());
+        vals.shuffle(&mut rand::thread_rng());
         let vals_merged = merge_top_k(vals.into_iter(), doc_range, ComparatorEnum::from(order));
         assert_eq!(&vals_merged, expected);
     }

src/directory/mmap_directory/mod.rs

@@ -676,7 +676,7 @@ mod tests {
             let num_segments = reader.searcher().segment_readers().len();
             assert!(num_segments <= 4);
             let num_components_except_deletes_and_tempstore =
-                crate::index::SegmentComponent::iterator().len() - 1;
+                crate::index::SegmentComponent::iterator().len() - 2;
             let max_num_mmapped = num_components_except_deletes_and_tempstore * num_segments;
             assert_eventually(|| {
                 let num_mmapped = mmap_directory.get_cache_info().mmapped.len();

src/docset.rs

@@ -51,55 +51,31 @@ pub trait DocSet: Send {
         doc
     }
 
-    /// !!!Dragons ahead!!!
-    /// In spirit, this is an approximate and dangerous version of `seek`.
-    ///
-    /// It can leave the DocSet in an `invalid` state and might return a
-    /// lower bound of what the result of Seek would have been.
-    ///
-    ///
-    /// More accurately it returns either:
-    /// - Found if the target is in the docset. In that case, the DocSet is left in a valid state.
-    /// - SeekLowerBound(seek_lower_bound) if the target is not in the docset. In that case, The
-    ///   DocSet can be the left in a invalid state. The DocSet should then only receives call to
-    ///   `seek_danger(..)` until it returns `Found`, and get back to a valid state.
-    ///
-    /// `seek_lower_bound` can be any `DocId` (in the docset or not) as long as it is in
-    /// `(target .. seek_result] U {TERMINATED}` where `seek_result` is the first document in the
-    /// docset greater than to `target`.
-    ///
-    /// `seek_danger` may return `SeekLowerBound(TERMINATED)`.
-    ///
-    /// Calling `seek_danger` with TERMINATED as a target is allowed,
-    /// and should always return NewTarget(TERMINATED) or anything larger as TERMINATED is NOT in
-    /// the DocSet.
+    /// Seeks to the target if possible and returns true if the target is in the DocSet.
     ///
     /// DocSets that already have an efficient `seek` method don't need to implement
-    /// `seek_danger`.
+    /// `seek_into_the_danger_zone`. All wrapper DocSets should forward
+    /// `seek_into_the_danger_zone` to the underlying DocSet.
     ///
-    /// Consecutive calls to seek_danger are guaranteed to have strictly increasing `target`
-    /// values.
-    fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {
-        if target >= TERMINATED {
-            debug_assert!(target == TERMINATED);
-            // No need to advance.
-            return SeekDangerResult::SeekLowerBound(target);
-        }
-
-        // The default implementation does not include any
-        // `danger zone` behavior.
-        //
-        // It does not leave the scorer in an invalid state.
-        // For this reason, we can safely call `self.doc()`.
-        let mut doc = self.doc();
-        if doc < target {
-            doc = self.seek(target);
-        }
-        if doc == target {
-            SeekDangerResult::Found
-        } else {
-            SeekDangerResult::SeekLowerBound(doc)
+    /// ## API Behaviour
+    /// If `seek_into_the_danger_zone` is returning true, a call to `doc()` has to return target.
+    /// If `seek_into_the_danger_zone` is returning false, a call to `doc()` may return any doc
+    /// between the last doc that matched and target or a doc that is a valid next hit after
+    /// target. The DocSet is considered to be in an invalid state until
+    /// `seek_into_the_danger_zone` returns true again.
+    ///
+    /// `target` needs to be equal or larger than `doc` when in a valid state.
+    ///
+    /// Consecutive calls are not allowed to have decreasing `target` values.
+    ///
+    /// # Warning
+    /// This is an advanced API used by intersection. The API contract is tricky, avoid using it.
+    fn seek_into_the_danger_zone(&mut self, target: DocId) -> bool {
+        let current_doc = self.doc();
+        if current_doc < target {
+            self.seek(target);
         }
+        self.doc() == target
     }
 
     /// Fills a given mutable buffer with the next doc ids from the
@@ -190,17 +166,6 @@ pub trait DocSet: Send {
     }
 }
 
-#[derive(Clone, Copy, Debug, PartialEq, Eq)]
-pub enum SeekDangerResult {
-    /// The target was found in the DocSet.
-    Found,
-    /// The target was not found in the DocSet.
-    /// We return a range in which the value could be.
-    /// The given target can be any DocId, that is <= than the first document
-    /// in the docset after the target.
-    SeekLowerBound(DocId),
-}
-
 impl DocSet for &mut dyn DocSet {
     fn advance(&mut self) -> u32 {
         (**self).advance()
@@ -210,8 +175,8 @@ impl DocSet for &mut dyn DocSet {
         (**self).seek(target)
     }
 
-    fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {
-        (**self).seek_danger(target)
+    fn seek_into_the_danger_zone(&mut self, target: DocId) -> bool {
+        (**self).seek_into_the_danger_zone(target)
     }
 
     fn doc(&self) -> u32 {
@@ -246,9 +211,9 @@ impl<TDocSet: DocSet + ?Sized> DocSet for Box<TDocSet> {
         unboxed.seek(target)
     }
 
-    fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {
+    fn seek_into_the_danger_zone(&mut self, target: DocId) -> bool {
         let unboxed: &mut TDocSet = self.borrow_mut();
-        unboxed.seek_danger(target)
+        unboxed.seek_into_the_danger_zone(target)
     }
 
     fn fill_buffer(&mut self, buffer: &mut [DocId; COLLECT_BLOCK_BUFFER_LEN]) -> usize {

src/fastfield/alive_bitset.rs

@@ -162,7 +162,7 @@ mod tests {
 mod bench {
 
     use rand::prelude::IteratorRandom;
-    use rand::rng;
+    use rand::thread_rng;
     use test::Bencher;
 
     use super::AliveBitSet;
@@ -176,7 +176,7 @@ mod bench {
     }
 
     fn remove_rand(raw: &mut Vec<u32>) {
-        let i = (0..raw.len()).choose(&mut rng()).unwrap();
+        let i = (0..raw.len()).choose(&mut thread_rng()).unwrap();
         raw.remove(i);
     }
 

src/fastfield/mod.rs

@@ -879,7 +879,7 @@ mod tests {
         const ONE_HOUR_IN_MICROSECS: i64 = 3_600 * 1_000_000;
         let times: Vec<DateTime> = std::iter::repeat_with(|| {
             // +- One hour.
-            let t = T0 + rng.random_range(-ONE_HOUR_IN_MICROSECS..ONE_HOUR_IN_MICROSECS);
+            let t = T0 + rng.gen_range(-ONE_HOUR_IN_MICROSECS..ONE_HOUR_IN_MICROSECS);
             DateTime::from_timestamp_micros(t)
         })
         .take(1_000)

src/functional_test.rs

@@ -1,6 +1,6 @@
 use std::collections::HashSet;
 
-use rand::{rng, Rng};
+use rand::{thread_rng, Rng};
 
 use crate::indexer::index_writer::MEMORY_BUDGET_NUM_BYTES_MIN;
 use crate::schema::*;
@@ -29,7 +29,7 @@ fn test_functional_store() -> crate::Result<()> {
     let index = Index::create_in_ram(schema);
     let reader = index.reader()?;
 
-    let mut rng = rng();
+    let mut rng = thread_rng();
 
     let mut index_writer: IndexWriter =
         index.writer_with_num_threads(3, 3 * MEMORY_BUDGET_NUM_BYTES_MIN)?;
@@ -38,9 +38,9 @@ fn test_functional_store() -> crate::Result<()> {
 
     let mut doc_id = 0u64;
     for _iteration in 0..get_num_iterations() {
-        let num_docs: usize = rng.random_range(0..4);
+        let num_docs: usize = rng.gen_range(0..4);
         if !doc_set.is_empty() {
-            let doc_to_remove_id = rng.random_range(0..doc_set.len());
+            let doc_to_remove_id = rng.gen_range(0..doc_set.len());
             let removed_doc_id = doc_set.swap_remove(doc_to_remove_id);
             index_writer.delete_term(Term::from_field_u64(id_field, removed_doc_id));
         }
@@ -70,10 +70,10 @@ const LOREM: &str = "Doc Lorem ipsum dolor sit amet, consectetur adipiscing elit
                      cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat \
                      non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.";
 fn get_text() -> String {
-    use rand::seq::IndexedRandom;
-    let mut rng = rng();
+    use rand::seq::SliceRandom;
+    let mut rng = thread_rng();
     let tokens: Vec<_> = LOREM.split(' ').collect();
-    let random_val = rng.random_range(0..20);
+    let random_val = rng.gen_range(0..20);
 
     (0..random_val)
         .map(|_| tokens.choose(&mut rng).unwrap())
@@ -101,7 +101,7 @@ fn test_functional_indexing_unsorted() -> crate::Result<()> {
     let index = Index::create_from_tempdir(schema)?;
     let reader = index.reader()?;
 
-    let mut rng = rng();
+    let mut rng = thread_rng();
 
     let mut index_writer: IndexWriter =
         index.writer_with_num_threads(3, 3 * MEMORY_BUDGET_NUM_BYTES_MIN)?;
@@ -110,7 +110,7 @@ fn test_functional_indexing_unsorted() -> crate::Result<()> {
     let mut uncommitted_docs: HashSet<u64> = HashSet::new();
 
     for _ in 0..get_num_iterations() {
-        let random_val = rng.random_range(0..20);
+        let random_val = rng.gen_range(0..20);
         if random_val == 0 {
             index_writer.commit()?;
             committed_docs.extend(&uncommitted_docs);

src/index/index.rs

@@ -8,6 +8,7 @@ use std::thread::available_parallelism;
 use super::segment::Segment;
 use super::segment_reader::merge_field_meta_data;
 use super::{FieldMetadata, IndexSettings};
+use crate::codec::{CodecConfiguration, StandardCodec};
 use crate::core::{Executor, META_FILEPATH};
 use crate::directory::error::OpenReadError;
 #[cfg(feature = "mmap")]
@@ -59,6 +60,7 @@ fn save_new_metas(
     schema: Schema,
     index_settings: IndexSettings,
     directory: &dyn Directory,
+    codec: CodecConfiguration,
 ) -> crate::Result<()> {
     save_metas(
         &IndexMeta {
@@ -67,6 +69,7 @@ fn save_new_metas(
             schema,
             opstamp: 0u64,
             payload: None,
+            codec,
         },
         directory,
     )?;
@@ -101,18 +104,21 @@ fn save_new_metas(
 /// };
 /// let index = Index::builder().schema(schema).settings(settings).create_in_ram();
 /// ```
-pub struct IndexBuilder {
+pub struct IndexBuilder<Codec: crate::codec::Codec = StandardCodec> {
     schema: Option<Schema>,
     index_settings: IndexSettings,
     tokenizer_manager: TokenizerManager,
     fast_field_tokenizer_manager: TokenizerManager,
+    codec: Codec,
 }
-impl Default for IndexBuilder {
+
+impl Default for IndexBuilder<StandardCodec> {
     fn default() -> Self {
         IndexBuilder::new()
     }
 }
-impl IndexBuilder {
+
+impl IndexBuilder<StandardCodec> {
     /// Creates a new `IndexBuilder`
     pub fn new() -> Self {
         Self {
@@ -120,6 +126,21 @@ impl IndexBuilder {
             index_settings: IndexSettings::default(),
             tokenizer_manager: TokenizerManager::default(),
             fast_field_tokenizer_manager: TokenizerManager::default(),
+            codec: StandardCodec,
+        }
+    }
+}
+
+impl<Codec: crate::codec::Codec> IndexBuilder<Codec> {
+    /// Set the codec
+    #[must_use]
+    pub fn codec<NewCodec: crate::codec::Codec>(self, codec: NewCodec) -> IndexBuilder<NewCodec> {
+        IndexBuilder {
+            schema: self.schema,
+            index_settings: self.index_settings,
+            tokenizer_manager: self.tokenizer_manager,
+            fast_field_tokenizer_manager: self.fast_field_tokenizer_manager,
+            codec,
         }
     }
 
@@ -154,7 +175,7 @@ impl IndexBuilder {
     /// The index will be allocated in anonymous memory.
     /// This is useful for indexing small set of documents
     /// for instances like unit test or temporary in memory index.
-    pub fn create_in_ram(self) -> Result<Index, TantivyError> {
+    pub fn create_in_ram(self) -> Result<Index<Codec>, TantivyError> {
         let ram_directory = RamDirectory::create();
         self.create(ram_directory)
     }
@@ -165,7 +186,7 @@ impl IndexBuilder {
     /// If a previous index was in this directory, it returns an
     /// [`TantivyError::IndexAlreadyExists`] error.
     #[cfg(feature = "mmap")]
-    pub fn create_in_dir<P: AsRef<Path>>(self, directory_path: P) -> crate::Result<Index> {
+    pub fn create_in_dir<P: AsRef<Path>>(self, directory_path: P) -> crate::Result<Index<Codec>> {
         let mmap_directory: Box<dyn Directory> = Box::new(MmapDirectory::open(directory_path)?);
         if Index::exists(&*mmap_directory)? {
             return Err(TantivyError::IndexAlreadyExists);
@@ -186,7 +207,7 @@ impl IndexBuilder {
         self,
         dir: impl Into<Box<dyn Directory>>,
         mem_budget: usize,
-    ) -> crate::Result<SingleSegmentIndexWriter<D>> {
+    ) -> crate::Result<SingleSegmentIndexWriter<Codec, D>> {
         let index = self.create(dir)?;
         let index_simple_writer = SingleSegmentIndexWriter::new(index, mem_budget)?;
         Ok(index_simple_writer)
@@ -202,7 +223,7 @@ impl IndexBuilder {
     /// For other unit tests, prefer the [`RamDirectory`], see:
     /// [`IndexBuilder::create_in_ram()`].
     #[cfg(feature = "mmap")]
-    pub fn create_from_tempdir(self) -> crate::Result<Index> {
+    pub fn create_from_tempdir(self) -> crate::Result<Index<Codec>> {
         let mmap_directory: Box<dyn Directory> = Box::new(MmapDirectory::create_from_tempdir()?);
         self.create(mmap_directory)
     }
@@ -215,12 +236,15 @@ impl IndexBuilder {
     }
 
     /// Opens or creates a new index in the provided directory
-    pub fn open_or_create<T: Into<Box<dyn Directory>>>(self, dir: T) -> crate::Result<Index> {
+    pub fn open_or_create<T: Into<Box<dyn Directory>>>(
+        self,
+        dir: T,
+    ) -> crate::Result<Index<Codec>> {
         let dir: Box<dyn Directory> = dir.into();
         if !Index::exists(&*dir)? {
             return self.create(dir);
         }
-        let mut index = Index::open(dir)?;
+        let mut index: Index<Codec> = Index::<Codec>::open_with_codec(dir)?;
         index.set_tokenizers(self.tokenizer_manager.clone());
         if index.schema() == self.get_expect_schema()? {
             Ok(index)
@@ -244,18 +268,26 @@ impl IndexBuilder {
     /// Creates a new index given an implementation of the trait `Directory`.
     ///
     /// If a directory previously existed, it will be erased.
-    fn create<T: Into<Box<dyn Directory>>>(self, dir: T) -> crate::Result<Index> {
+    pub fn create<T: Into<Box<dyn Directory>>>(self, dir: T) -> crate::Result<Index<Codec>> {
+        self.create_avoid_monomorphization(dir.into())
+    }
+
+    fn create_avoid_monomorphization(self, dir: Box<dyn Directory>) -> crate::Result<Index<Codec>> {
         self.validate()?;
         let dir = dir.into();
         let directory = ManagedDirectory::wrap(dir)?;
+        let codec: CodecConfiguration = CodecConfiguration::from_codec(&self.codec);
         save_new_metas(
             self.get_expect_schema()?,
             self.index_settings.clone(),
             &directory,
+            codec,
         )?;
-        let mut metas = IndexMeta::with_schema(self.get_expect_schema()?);
+        let schema = self.get_expect_schema()?;
+        let mut metas = IndexMeta::with_schema_and_codec(schema, &self.codec);
         metas.index_settings = self.index_settings;
-        let mut index = Index::open_from_metas(directory, &metas, SegmentMetaInventory::default());
+        let mut index: Index<Codec> =
+            Index::<Codec>::open_from_metas(directory, &metas, SegmentMetaInventory::default())?;
         index.set_tokenizers(self.tokenizer_manager);
         index.set_fast_field_tokenizers(self.fast_field_tokenizer_manager);
         Ok(index)
@@ -264,7 +296,7 @@ impl IndexBuilder {
 
 /// Search Index
 #[derive(Clone)]
-pub struct Index {
+pub struct Index<Codec: crate::codec::Codec = crate::codec::StandardCodec> {
     directory: ManagedDirectory,
     schema: Schema,
     settings: IndexSettings,
@@ -272,6 +304,7 @@ pub struct Index {
     tokenizers: TokenizerManager,
     fast_field_tokenizers: TokenizerManager,
     inventory: SegmentMetaInventory,
+    codec: Codec,
 }
 
 impl Index {
@@ -279,41 +312,6 @@ impl Index {
     pub fn builder() -> IndexBuilder {
         IndexBuilder::new()
     }
-    /// Examines the directory to see if it contains an index.
-    ///
-    /// Effectively, it only checks for the presence of the `meta.json` file.
-    pub fn exists(dir: &dyn Directory) -> Result<bool, OpenReadError> {
-        dir.exists(&META_FILEPATH)
-    }
-
-    /// Accessor to the search executor.
-    ///
-    /// This pool is used by default when calling `searcher.search(...)`
-    /// to perform search on the individual segments.
-    ///
-    /// By default the executor is single thread, and simply runs in the calling thread.
-    pub fn search_executor(&self) -> &Executor {
-        &self.executor
-    }
-
-    /// Replace the default single thread search executor pool
-    /// by a thread pool with a given number of threads.
-    pub fn set_multithread_executor(&mut self, num_threads: usize) -> crate::Result<()> {
-        self.executor = Executor::multi_thread(num_threads, "tantivy-search-")?;
-        Ok(())
-    }
-
-    /// Custom thread pool by a outer thread pool.
-    pub fn set_executor(&mut self, executor: Executor) {
-        self.executor = executor;
-    }
-
-    /// Replace the default single thread search executor pool
-    /// by a thread pool with as many threads as there are CPUs on the system.
-    pub fn set_default_multithread_executor(&mut self) -> crate::Result<()> {
-        let default_num_threads = available_parallelism()?.get();
-        self.set_multithread_executor(default_num_threads)
-    }
 
     /// Creates a new index using the [`RamDirectory`].
     ///
@@ -324,6 +322,13 @@ impl Index {
         IndexBuilder::new().schema(schema).create_in_ram().unwrap()
     }
 
+    /// Examines the directory to see if it contains an index.
+    ///
+    /// Effectively, it only checks for the presence of the `meta.json` file.
+    pub fn exists(directory: &dyn Directory) -> Result<bool, OpenReadError> {
+        directory.exists(&META_FILEPATH)
+    }
+
     /// Creates a new index in a given filepath.
     /// The index will use the [`MmapDirectory`].
     ///
@@ -370,20 +375,107 @@ impl Index {
         schema: Schema,
         settings: IndexSettings,
     ) -> crate::Result<Index> {
-        let dir: Box<dyn Directory> = dir.into();
+        Self::create_to_avoid_monomorphization(dir.into(), schema, settings)
+    }
+
+    fn create_to_avoid_monomorphization(
+        dir: Box<dyn Directory>,
+        schema: Schema,
+        settings: IndexSettings,
+    ) -> crate::Result<Index> {
         let mut builder = IndexBuilder::new().schema(schema);
         builder = builder.settings(settings);
         builder.create(dir)
     }
 
+    /// Opens a new directory from an index path.
+    #[cfg(feature = "mmap")]
+    pub fn open_in_dir<P: AsRef<Path>>(directory_path: P) -> crate::Result<Index> {
+        Self::open_in_dir_to_avoid_monomorphization(directory_path.as_ref())
+    }
+
+    #[inline(never)]
+    fn open_in_dir_to_avoid_monomorphization(directory_path: &Path) -> crate::Result<Index> {
+        let mmap_directory = MmapDirectory::open(directory_path)?;
+        Index::open(mmap_directory)
+    }
+
+    /// Open the index using the provided directory
+    pub fn open<T: Into<Box<dyn Directory>>>(directory: T) -> crate::Result<Index> {
+        Index::<StandardCodec>::open_with_codec(directory.into())
+    }
+}
+
+impl<Codec: crate::codec::Codec> Index<Codec> {
+    /// Returns a version of this index with the standard codec.
+    /// This is useful when you need to pass the index to APIs that
+    /// don't care about the codec (e.g., for reading).
+    pub(crate) fn with_standard_codec(&self) -> Index<StandardCodec> {
+        Index {
+            directory: self.directory.clone(),
+            schema: self.schema.clone(),
+            settings: self.settings.clone(),
+            executor: self.executor.clone(),
+            tokenizers: self.tokenizers.clone(),
+            fast_field_tokenizers: self.fast_field_tokenizers.clone(),
+            inventory: self.inventory.clone(),
+            codec: StandardCodec::default(),
+        }
+    }
+
+    /// Open the index using the provided directory
+    #[inline(never)]
+    pub fn open_with_codec(directory: Box<dyn Directory>) -> crate::Result<Index<Codec>> {
+        let directory = ManagedDirectory::wrap(directory)?;
+        let inventory = SegmentMetaInventory::default();
+        let metas = load_metas(&directory, &inventory)?;
+        let index: Index<Codec> = Index::<Codec>::open_from_metas(directory, &metas, inventory)?;
+        Ok(index)
+    }
+
+    /// Accessor to the codec.
+    pub fn codec(&self) -> &Codec {
+        &self.codec
+    }
+
+    /// Accessor to the search executor.
+    ///
+    /// This pool is used by default when calling `searcher.search(...)`
+    /// to perform search on the individual segments.
+    ///
+    /// By default the executor is single thread, and simply runs in the calling thread.
+    pub fn search_executor(&self) -> &Executor {
+        &self.executor
+    }
+
+    /// Replace the default single thread search executor pool
+    /// by a thread pool with a given number of threads.
+    pub fn set_multithread_executor(&mut self, num_threads: usize) -> crate::Result<()> {
+        self.executor = Executor::multi_thread(num_threads, "tantivy-search-")?;
+        Ok(())
+    }
+
+    /// Custom thread pool by a outer thread pool.
+    pub fn set_executor(&mut self, executor: Executor) {
+        self.executor = executor;
+    }
+
+    /// Replace the default single thread search executor pool
+    /// by a thread pool with as many threads as there are CPUs on the system.
+    pub fn set_default_multithread_executor(&mut self) -> crate::Result<()> {
+        let default_num_threads = available_parallelism()?.get();
+        self.set_multithread_executor(default_num_threads)
+    }
+
     /// Creates a new index given a directory and an [`IndexMeta`].
-    fn open_from_metas(
+    fn open_from_metas<C: crate::codec::Codec>(
         directory: ManagedDirectory,
         metas: &IndexMeta,
         inventory: SegmentMetaInventory,
-    ) -> Index {
+    ) -> crate::Result<Index<C>> {
         let schema = metas.schema.clone();
-        Index {
+        let codec = metas.codec.to_codec::<C>()?;
+        Ok(Index {
             settings: metas.index_settings.clone(),
             directory,
             schema,
@@ -391,7 +483,8 @@ impl Index {
             fast_field_tokenizers: TokenizerManager::default(),
             executor: Executor::single_thread(),
             inventory,
-        }
+            codec,
+        })
     }
 
     /// Setter for the tokenizer manager.
@@ -447,7 +540,7 @@ impl Index {
     /// Create a default [`IndexReader`] for the given index.
     ///
     /// See [`Index.reader_builder()`].
-    pub fn reader(&self) -> crate::Result<IndexReader> {
+    pub fn reader(&self) -> crate::Result<IndexReader<Codec>> {
         self.reader_builder().try_into()
     }
 
@@ -455,17 +548,10 @@ impl Index {
     ///
     /// Most project should create at most one reader for a given index.
     /// This method is typically called only once per `Index` instance.
-    pub fn reader_builder(&self) -> IndexReaderBuilder {
+    pub fn reader_builder(&self) -> IndexReaderBuilder<Codec> {
         IndexReaderBuilder::new(self.clone())
     }
 
-    /// Opens a new directory from an index path.
-    #[cfg(feature = "mmap")]
-    pub fn open_in_dir<P: AsRef<Path>>(directory_path: P) -> crate::Result<Index> {
-        let mmap_directory = MmapDirectory::open(directory_path)?;
-        Index::open(mmap_directory)
-    }
-
     /// Returns the list of the segment metas tracked by the index.
     ///
     /// Such segments can of course be part of the index,
@@ -506,16 +592,6 @@ impl Index {
         self.inventory.new_segment_meta(segment_id, max_doc)
     }
 
-    /// Open the index using the provided directory
-    pub fn open<T: Into<Box<dyn Directory>>>(directory: T) -> crate::Result<Index> {
-        let directory = directory.into();
-        let directory = ManagedDirectory::wrap(directory)?;
-        let inventory = SegmentMetaInventory::default();
-        let metas = load_metas(&directory, &inventory)?;
-        let index = Index::open_from_metas(directory, &metas, inventory);
-        Ok(index)
-    }
-
     /// Reads the index meta file from the directory.
     pub fn load_metas(&self) -> crate::Result<IndexMeta> {
         load_metas(self.directory(), &self.inventory)
@@ -539,7 +615,7 @@ impl Index {
     pub fn writer_with_options<D: Document>(
         &self,
         options: IndexWriterOptions,
-    ) -> crate::Result<IndexWriter<D>> {
+    ) -> crate::Result<IndexWriter<Codec, D>> {
         let directory_lock = self
             .directory
             .acquire_lock(&INDEX_WRITER_LOCK)
@@ -581,7 +657,7 @@ impl Index {
         &self,
         num_threads: usize,
         overall_memory_budget_in_bytes: usize,
-    ) -> crate::Result<IndexWriter<D>> {
+    ) -> crate::Result<IndexWriter<Codec, D>> {
         let memory_arena_in_bytes_per_thread = overall_memory_budget_in_bytes / num_threads;
         let options = IndexWriterOptions::builder()
             .num_worker_threads(num_threads)
@@ -595,7 +671,7 @@ impl Index {
     /// That index writer only simply has a single thread and a memory budget of 15 MB.
     /// Using a single thread gives us a deterministic allocation of DocId.
     #[cfg(test)]
-    pub fn writer_for_tests<D: Document>(&self) -> crate::Result<IndexWriter<D>> {
+    pub fn writer_for_tests<D: Document>(&self) -> crate::Result<IndexWriter<Codec, D>> {
         self.writer_with_num_threads(1, MEMORY_BUDGET_NUM_BYTES_MIN)
     }
 
@@ -613,7 +689,7 @@ impl Index {
     pub fn writer<D: Document>(
         &self,
         memory_budget_in_bytes: usize,
-    ) -> crate::Result<IndexWriter<D>> {
+    ) -> crate::Result<IndexWriter<Codec, D>> {
         let mut num_threads = std::cmp::min(available_parallelism()?.get(), MAX_NUM_THREAD);
         let memory_budget_num_bytes_per_thread = memory_budget_in_bytes / num_threads;
         if memory_budget_num_bytes_per_thread < MEMORY_BUDGET_NUM_BYTES_MIN {
@@ -640,7 +716,7 @@ impl Index {
     }
 
     /// Returns the list of segments that are searchable
-    pub fn searchable_segments(&self) -> crate::Result<Vec<Segment>> {
+    pub fn searchable_segments(&self) -> crate::Result<Vec<Segment<Codec>>> {
         Ok(self
             .searchable_segment_metas()?
             .into_iter()
@@ -649,12 +725,12 @@ impl Index {
     }
 
     #[doc(hidden)]
-    pub fn segment(&self, segment_meta: SegmentMeta) -> Segment {
+    pub fn segment(&self, segment_meta: SegmentMeta) -> Segment<Codec> {
         Segment::for_index(self.clone(), segment_meta)
     }
 
     /// Creates a new segment.
-    pub fn new_segment(&self) -> Segment {
+    pub fn new_segment(&self) -> Segment<Codec> {
         let segment_meta = self
             .inventory
             .new_segment_meta(SegmentId::generate_random(), 0);

src/index/index_meta.rs

@@ -1,10 +1,13 @@
 use std::collections::HashSet;
 use std::fmt;
 use std::path::PathBuf;
+use std::sync::atomic::AtomicBool;
+use std::sync::Arc;
 
 use serde::{Deserialize, Serialize};
 
 use super::SegmentComponent;
+use crate::codec::{Codec, CodecConfiguration};
 use crate::index::SegmentId;
 use crate::schema::Schema;
 use crate::store::Compressor;
@@ -35,6 +38,7 @@ impl SegmentMetaInventory {
         let inner = InnerSegmentMeta {
             segment_id,
             max_doc,
+            include_temp_doc_store: Arc::new(AtomicBool::new(true)),
             deletes: None,
         };
         SegmentMeta::from(self.inventory.track(inner))
@@ -82,6 +86,15 @@ impl SegmentMeta {
         self.tracked.segment_id
     }
 
+    /// Removes the Component::TempStore from the alive list and
+    /// therefore marks the temp docstore file to be deleted by
+    /// the garbage collection.
+    pub fn untrack_temp_docstore(&self) {
+        self.tracked
+            .include_temp_doc_store
+            .store(false, std::sync::atomic::Ordering::Relaxed);
+    }
+
     /// Returns the number of deleted documents.
     pub fn num_deleted_docs(&self) -> u32 {
         self.tracked
@@ -99,9 +112,20 @@ impl SegmentMeta {
     /// is by removing all files that have been created by tantivy
     /// and are not used by any segment anymore.
     pub fn list_files(&self) -> HashSet<PathBuf> {
-        SegmentComponent::iterator()
-            .map(|component| self.relative_path(*component))
-            .collect::<HashSet<PathBuf>>()
+        if self
+            .tracked
+            .include_temp_doc_store
+            .load(std::sync::atomic::Ordering::Relaxed)
+        {
+            SegmentComponent::iterator()
+                .map(|component| self.relative_path(*component))
+                .collect::<HashSet<PathBuf>>()
+        } else {
+            SegmentComponent::iterator()
+                .filter(|comp| *comp != &SegmentComponent::TempStore)
+                .map(|component| self.relative_path(*component))
+                .collect::<HashSet<PathBuf>>()
+        }
     }
 
     /// Returns the relative path of a component of our segment.
@@ -115,6 +139,7 @@ impl SegmentMeta {
             SegmentComponent::Positions => ".pos".to_string(),
             SegmentComponent::Terms => ".term".to_string(),
             SegmentComponent::Store => ".store".to_string(),
+            SegmentComponent::TempStore => ".store.temp".to_string(),
             SegmentComponent::FastFields => ".fast".to_string(),
             SegmentComponent::FieldNorms => ".fieldnorm".to_string(),
             SegmentComponent::Delete => format!(".{}.del", self.delete_opstamp().unwrap_or(0)),
@@ -159,6 +184,7 @@ impl SegmentMeta {
             segment_id: inner_meta.segment_id,
             max_doc,
             deletes: None,
+            include_temp_doc_store: Arc::new(AtomicBool::new(true)),
         });
         SegmentMeta { tracked }
     }
@@ -177,6 +203,7 @@ impl SegmentMeta {
         let tracked = self.tracked.map(move |inner_meta| InnerSegmentMeta {
             segment_id: inner_meta.segment_id,
             max_doc: inner_meta.max_doc,
+            include_temp_doc_store: Arc::new(AtomicBool::new(true)),
             deletes: Some(delete_meta),
         });
         SegmentMeta { tracked }
@@ -188,6 +215,14 @@ struct InnerSegmentMeta {
     segment_id: SegmentId,
     max_doc: u32,
     pub deletes: Option<DeleteMeta>,
+    /// If you want to avoid the SegmentComponent::TempStore file to be covered by
+    /// garbage collection and deleted, set this to true. This is used during merge.
+    #[serde(skip)]
+    #[serde(default = "default_temp_store")]
+    pub(crate) include_temp_doc_store: Arc<AtomicBool>,
+}
+fn default_temp_store() -> Arc<AtomicBool> {
+    Arc::new(AtomicBool::new(false))
 }
 
 impl InnerSegmentMeta {
@@ -286,6 +321,7 @@ pub struct IndexMeta {
     /// This payload is entirely unused by tantivy.
     #[serde(skip_serializing_if = "Option::is_none")]
     pub payload: Option<String>,
+    pub codec: CodecConfiguration,
 }
 
 #[derive(Deserialize, Debug)]
@@ -297,6 +333,8 @@ struct UntrackedIndexMeta {
     pub opstamp: Opstamp,
     #[serde(skip_serializing_if = "Option::is_none")]
     pub payload: Option<String>,
+    #[serde(default)]
+    pub codec: CodecConfiguration,
 }
 
 impl UntrackedIndexMeta {
@@ -311,6 +349,7 @@ impl UntrackedIndexMeta {
             schema: self.schema,
             opstamp: self.opstamp,
             payload: self.payload,
+            codec: self.codec,
         }
     }
 }
@@ -321,13 +360,14 @@ impl IndexMeta {
     ///
     /// This new index does not contains any segments.
     /// Opstamp will the value `0u64`.
-    pub fn with_schema(schema: Schema) -> IndexMeta {
+    pub fn with_schema_and_codec<C: Codec>(schema: Schema, codec: &C) -> IndexMeta {
         IndexMeta {
             index_settings: IndexSettings::default(),
             segments: vec![],
             schema,
             opstamp: 0u64,
             payload: None,
+            codec: CodecConfiguration::from_codec(codec),
         }
     }
 
@@ -378,11 +418,12 @@ mod tests {
             schema,
             opstamp: 0u64,
             payload: None,
+            codec: Default::default(),
         };
         let json = serde_json::ser::to_string(&index_metas).expect("serialization failed");
         assert_eq!(
             json,
-            r#"{"index_settings":{"docstore_compression":"none","docstore_blocksize":16384},"segments":[],"schema":[{"name":"text","type":"text","options":{"indexing":{"record":"position","fieldnorms":true,"tokenizer":"default"},"stored":false,"fast":false}}],"opstamp":0}"#
+            r#"{"index_settings":{"docstore_compression":"none","docstore_blocksize":16384},"segments":[],"schema":[{"name":"text","type":"text","options":{"indexing":{"record":"position","fieldnorms":true,"tokenizer":"default"},"stored":false,"fast":false}}],"opstamp":0,"codec":{"name":"standard"}}"#
         );
 
         let deser_meta: UntrackedIndexMeta = serde_json::from_str(&json).unwrap();

src/index/segment.rs

@@ -2,6 +2,7 @@ use std::fmt;
 use std::path::PathBuf;
 
 use super::SegmentComponent;
+use crate::codec::StandardCodec;
 use crate::directory::error::{OpenReadError, OpenWriteError};
 use crate::directory::{Directory, FileSlice, WritePtr};
 use crate::index::{Index, SegmentId, SegmentMeta};
@@ -10,25 +11,25 @@ use crate::Opstamp;
 
 /// A segment is a piece of the index.
 #[derive(Clone)]
-pub struct Segment {
-    index: Index,
+pub struct Segment<C: crate::codec::Codec = StandardCodec> {
+    index: Index<C>,
     meta: SegmentMeta,
 }
 
-impl fmt::Debug for Segment {
-    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
+impl<C: crate::codec::Codec> fmt::Debug for Segment<C> {
+    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         write!(f, "Segment({:?})", self.id().uuid_string())
     }
 }
 
-impl Segment {
+impl<C: crate::codec::Codec> Segment<C> {
     /// Creates a new segment given an `Index` and a `SegmentId`
-    pub(crate) fn for_index(index: Index, meta: SegmentMeta) -> Segment {
+    pub(crate) fn for_index(index: Index<C>, meta: SegmentMeta) -> Segment<C> {
         Segment { index, meta }
     }
 
     /// Returns the index the segment belongs to.
-    pub fn index(&self) -> &Index {
+    pub fn index(&self) -> &Index<C> {
         &self.index
     }
 
@@ -46,7 +47,7 @@ impl Segment {
     ///
     /// This method is only used when updating `max_doc` from 0
     /// as we finalize a fresh new segment.
-    pub fn with_max_doc(self, max_doc: u32) -> Segment {
+    pub fn with_max_doc(self, max_doc: u32) -> Segment<C> {
         Segment {
             index: self.index,
             meta: self.meta.with_max_doc(max_doc),
@@ -55,7 +56,7 @@ impl Segment {
 
     #[doc(hidden)]
     #[must_use]
-    pub fn with_delete_meta(self, num_deleted_docs: u32, opstamp: Opstamp) -> Segment {
+    pub fn with_delete_meta(self, num_deleted_docs: u32, opstamp: Opstamp) -> Segment<C> {
         Segment {
             index: self.index,
             meta: self.meta.with_delete_meta(num_deleted_docs, opstamp),

src/index/segment_component.rs

@@ -23,6 +23,8 @@ pub enum SegmentComponent {
     /// Accessing a document from the store is relatively slow, as it
     /// requires to decompress the entire block it belongs to.
     Store,
+    /// Temporary storage of the documents, before streamed to `Store`.
+    TempStore,
     /// Bitset describing which document of the segment is alive.
     /// (It was representing deleted docs but changed to represent alive docs from v0.17)
     Delete,
@@ -31,13 +33,14 @@ pub enum SegmentComponent {
 impl SegmentComponent {
     /// Iterates through the components.
     pub fn iterator() -> slice::Iter<'static, SegmentComponent> {
-        static SEGMENT_COMPONENTS: [SegmentComponent; 7] = [
+        static SEGMENT_COMPONENTS: [SegmentComponent; 8] = [
             SegmentComponent::Postings,
             SegmentComponent::Positions,
             SegmentComponent::FastFields,
             SegmentComponent::FieldNorms,
             SegmentComponent::Terms,
             SegmentComponent::Store,
+            SegmentComponent::TempStore,
             SegmentComponent::Delete,
         ];
         SEGMENT_COMPONENTS.iter()

src/index/segment_reader.rs

@@ -140,13 +140,13 @@ impl SegmentReader {
     }
 
     /// Open a new segment for reading.
-    pub fn open(segment: &Segment) -> crate::Result<SegmentReader> {
+    pub fn open<C: crate::codec::Codec>(segment: &Segment<C>) -> crate::Result<SegmentReader> {
         Self::open_with_custom_alive_set(segment, None)
     }
 
     /// Open a new segment for reading.
-    pub fn open_with_custom_alive_set(
-        segment: &Segment,
+    pub fn open_with_custom_alive_set<C: crate::codec::Codec>(
+        segment: &Segment<C>,
         custom_bitset: Option<AliveBitSet>,
     ) -> crate::Result<SegmentReader> {
         let termdict_file = segment.open_read(SegmentComponent::Terms)?;

src/indexer/index_writer.rs

@@ -9,6 +9,7 @@ use smallvec::smallvec;
 use super::operation::{AddOperation, UserOperation};
 use super::segment_updater::SegmentUpdater;
 use super::{AddBatch, AddBatchReceiver, AddBatchSender, PreparedCommit};
+use crate::codec::{Codec, StandardCodec};
 use crate::directory::{DirectoryLock, GarbageCollectionResult, TerminatingWrite};
 use crate::error::TantivyError;
 use crate::fastfield::write_alive_bitset;
@@ -68,12 +69,12 @@ pub struct IndexWriterOptions {
 /// indexing queue.
 /// Each indexing thread builds its own independent [`Segment`], via
 /// a `SegmentWriter` object.
-pub struct IndexWriter<D: Document = TantivyDocument> {
+pub struct IndexWriter<C: Codec = StandardCodec, D: Document = TantivyDocument> {
     // the lock is just used to bind the
     // lifetime of the lock with that of the IndexWriter.
     _directory_lock: Option<DirectoryLock>,
 
-    index: Index,
+    index: Index<C>,
 
     options: IndexWriterOptions,
 
@@ -82,7 +83,7 @@ pub struct IndexWriter<D: Document = TantivyDocument> {
     index_writer_status: IndexWriterStatus<D>,
     operation_sender: AddBatchSender<D>,
 
-    segment_updater: SegmentUpdater,
+    segment_updater: SegmentUpdater<C>,
 
     worker_id: usize,
 
@@ -128,8 +129,8 @@ fn compute_deleted_bitset(
 /// is `==` target_opstamp.
 /// For instance, there was no delete operation between the state of the `segment_entry` and
 /// the `target_opstamp`, `segment_entry` is not updated.
-pub fn advance_deletes(
-    mut segment: Segment,
+pub fn advance_deletes<C: Codec>(
+    mut segment: Segment<C>,
     segment_entry: &mut SegmentEntry,
     target_opstamp: Opstamp,
 ) -> crate::Result<()> {
@@ -179,11 +180,11 @@ pub fn advance_deletes(
     Ok(())
 }
 
-fn index_documents<D: Document>(
+fn index_documents<C: crate::codec::Codec, D: Document>(
     memory_budget: usize,
-    segment: Segment,
+    segment: Segment<C>,
     grouped_document_iterator: &mut dyn Iterator<Item = AddBatch<D>>,
-    segment_updater: &SegmentUpdater,
+    segment_updater: &SegmentUpdater<C>,
     mut delete_cursor: DeleteCursor,
 ) -> crate::Result<()> {
     let mut segment_writer = SegmentWriter::for_segment(memory_budget, segment.clone())?;
@@ -218,7 +219,7 @@ fn index_documents<D: Document>(
     let alive_bitset_opt = apply_deletes(&segment_with_max_doc, &mut delete_cursor, &doc_opstamps)?;
 
     let meta = segment_with_max_doc.meta().clone();
-
+    meta.untrack_temp_docstore();
     // update segment_updater inventory to remove tempstore
     let segment_entry = SegmentEntry::new(meta, delete_cursor, alive_bitset_opt);
     segment_updater.schedule_add_segment(segment_entry).wait()?;
@@ -226,8 +227,8 @@ fn index_documents<D: Document>(
 }
 
 /// `doc_opstamps` is required to be non-empty.
-fn apply_deletes(
-    segment: &Segment,
+fn apply_deletes<C: crate::codec::Codec>(
+    segment: &Segment<C>,
     delete_cursor: &mut DeleteCursor,
     doc_opstamps: &[Opstamp],
 ) -> crate::Result<Option<BitSet>> {
@@ -262,7 +263,7 @@ fn apply_deletes(
     })
 }
 
-impl<D: Document> IndexWriter<D> {
+impl<C: Codec, D: Document> IndexWriter<C, D> {
     /// Create a new index writer. Attempts to acquire a lockfile.
     ///
     /// The lockfile should be deleted on drop, but it is possible
@@ -278,7 +279,7 @@ impl<D: Document> IndexWriter<D> {
     /// If the memory arena per thread is too small or too big, returns
     /// `TantivyError::InvalidArgument`
     pub(crate) fn new(
-        index: &Index,
+        index: &Index<C>,
         options: IndexWriterOptions,
         directory_lock: DirectoryLock,
     ) -> crate::Result<Self> {
@@ -345,7 +346,7 @@ impl<D: Document> IndexWriter<D> {
     }
 
     /// Accessor to the index.
-    pub fn index(&self) -> &Index {
+    pub fn index(&self) -> &Index<C> {
         &self.index
     }
 
@@ -393,7 +394,7 @@ impl<D: Document> IndexWriter<D> {
     /// It is safe to start writing file associated with the new `Segment`.
     /// These will not be garbage collected as long as an instance object of
     /// `SegmentMeta` object associated with the new `Segment` is "alive".
-    pub fn new_segment(&self) -> Segment {
+    pub fn new_segment(&self) -> Segment<C> {
         self.index.new_segment()
     }
 
@@ -615,7 +616,7 @@ impl<D: Document> IndexWriter<D> {
     /// It is also possible to add a payload to the `commit`
     /// using this API.
     /// See [`PreparedCommit::set_payload()`].
-    pub fn prepare_commit(&mut self) -> crate::Result<PreparedCommit<'_, D>> {
+    pub fn prepare_commit(&mut self) -> crate::Result<PreparedCommit<'_, C, D>> {
         // Here, because we join all of the worker threads,
         // all of the segment update for this commit have been
         // sent.
@@ -665,7 +666,7 @@ impl<D: Document> IndexWriter<D> {
         self.prepare_commit()?.commit()
     }
 
-    pub(crate) fn segment_updater(&self) -> &SegmentUpdater {
+    pub(crate) fn segment_updater(&self) -> &SegmentUpdater<C> {
         &self.segment_updater
     }
 
@@ -804,7 +805,7 @@ impl<D: Document> IndexWriter<D> {
     }
 }
 
-impl<D: Document> Drop for IndexWriter<D> {
+impl<C: Codec, D: Document> Drop for IndexWriter<C, D> {
     fn drop(&mut self) {
         self.segment_updater.kill();
         self.drop_sender();

src/indexer/merger.rs

@@ -145,7 +145,10 @@ fn extract_fast_field_required_columns(schema: &Schema) -> Vec<(String, ColumnTy
 }
 
 impl IndexMerger {
-    pub fn open(schema: Schema, segments: &[Segment]) -> crate::Result<IndexMerger> {
+    pub fn open<C: crate::codec::Codec>(
+        schema: Schema,
+        segments: &[Segment<C>],
+    ) -> crate::Result<IndexMerger> {
         let alive_bitset = segments.iter().map(|_| None).collect_vec();
         Self::open_with_custom_alive_set(schema, segments, alive_bitset)
     }
@@ -162,9 +165,9 @@ impl IndexMerger {
     // This can be used to merge but also apply an additional filter.
     // One use case is demux, which is basically taking a list of
     // segments and partitions them e.g. by a value in a field.
-    pub fn open_with_custom_alive_set(
+    pub fn open_with_custom_alive_set<C: crate::codec::Codec>(
         schema: Schema,
-        segments: &[Segment],
+        segments: &[Segment<C>],
         alive_bitset_opt: Vec<Option<AliveBitSet>>,
     ) -> crate::Result<IndexMerger> {
         let mut readers = vec![];
@@ -525,7 +528,10 @@ impl IndexMerger {
     ///
     /// # Returns
     /// The number of documents in the resulting segment.
-    pub fn write(&self, mut serializer: SegmentSerializer) -> crate::Result<u32> {
+    pub fn write<C: crate::codec::Codec>(
+        &self,
+        mut serializer: SegmentSerializer<C>,
+    ) -> crate::Result<u32> {
         let doc_id_mapping = self.get_doc_id_from_concatenated_data()?;
         debug!("write-fieldnorms");
         if let Some(fieldnorms_serializer) = serializer.extract_fieldnorms_serializer() {

src/indexer/prepared_commit.rs

@@ -1,16 +1,17 @@
 use super::IndexWriter;
+use crate::codec::Codec;
 use crate::schema::document::Document;
 use crate::{FutureResult, Opstamp, TantivyDocument};
 
 /// A prepared commit
-pub struct PreparedCommit<'a, D: Document = TantivyDocument> {
-    index_writer: &'a mut IndexWriter<D>,
+pub struct PreparedCommit<'a, C: Codec, D: Document = TantivyDocument> {
+    index_writer: &'a mut IndexWriter<C, D>,
     payload: Option<String>,
     opstamp: Opstamp,
 }
 
-impl<'a, D: Document> PreparedCommit<'a, D> {
-    pub(crate) fn new(index_writer: &'a mut IndexWriter<D>, opstamp: Opstamp) -> Self {
+impl<'a, C: Codec, D: Document> PreparedCommit<'a, C, D> {
+    pub(crate) fn new(index_writer: &'a mut IndexWriter<C, D>, opstamp: Opstamp) -> Self {
         Self {
             index_writer,
             payload: None,

src/indexer/segment_serializer.rs

@@ -8,17 +8,17 @@ use crate::store::StoreWriter;
 
 /// Segment serializer is in charge of laying out on disk
 /// the data accumulated and sorted by the `SegmentWriter`.
-pub struct SegmentSerializer {
-    segment: Segment,
+pub struct SegmentSerializer<C: crate::codec::Codec> {
+    segment: Segment<C>,
     pub(crate) store_writer: StoreWriter,
     fast_field_write: WritePtr,
     fieldnorms_serializer: Option<FieldNormsSerializer>,
     postings_serializer: InvertedIndexSerializer,
 }
 
-impl SegmentSerializer {
+impl<C: crate::codec::Codec> SegmentSerializer<C> {
     /// Creates a new `SegmentSerializer`.
-    pub fn for_segment(mut segment: Segment) -> crate::Result<SegmentSerializer> {
+    pub fn for_segment(mut segment: Segment<C>) -> crate::Result<SegmentSerializer<C>> {
         let settings = segment.index().settings().clone();
         let store_writer = {
             let store_write = segment.open_write(SegmentComponent::Store)?;
@@ -50,7 +50,7 @@ impl SegmentSerializer {
         self.store_writer.mem_usage()
     }
 
-    pub fn segment(&self) -> &Segment {
+    pub fn segment(&self) -> &Segment<C> {
         &self.segment
     }
 

src/indexer/segment_updater.rs

@@ -10,6 +10,7 @@ use std::sync::{Arc, RwLock};
 use rayon::{ThreadPool, ThreadPoolBuilder};
 
 use super::segment_manager::SegmentManager;
+use crate::codec::{Codec, CodecConfiguration};
 use crate::core::META_FILEPATH;
 use crate::directory::{Directory, DirectoryClone, GarbageCollectionResult};
 use crate::fastfield::AliveBitSet;
@@ -61,10 +62,10 @@ pub(crate) fn save_metas(metas: &IndexMeta, directory: &dyn Directory) -> crate:
 // We voluntarily pass a merge_operation ref to guarantee that
 // the merge_operation is alive during the process
 #[derive(Clone)]
-pub(crate) struct SegmentUpdater(Arc<InnerSegmentUpdater>);
+pub(crate) struct SegmentUpdater<C: Codec>(Arc<InnerSegmentUpdater<C>>);
 
-impl Deref for SegmentUpdater {
-    type Target = InnerSegmentUpdater;
+impl<C: Codec> Deref for SegmentUpdater<C> {
+    type Target = InnerSegmentUpdater<C>;
 
     #[inline]
     fn deref(&self) -> &Self::Target {
@@ -72,8 +73,8 @@ impl Deref for SegmentUpdater {
     }
 }
 
-fn garbage_collect_files(
-    segment_updater: SegmentUpdater,
+fn garbage_collect_files<C: Codec>(
+    segment_updater: SegmentUpdater<C>,
 ) -> crate::Result<GarbageCollectionResult> {
     info!("Running garbage collection");
     let mut index = segment_updater.index.clone();
@@ -84,8 +85,8 @@ fn garbage_collect_files(
 
 /// Merges a list of segments the list of segment givens in the `segment_entries`.
 /// This function happens in the calling thread and is computationally expensive.
-fn merge(
-    index: &Index,
+fn merge<Codec: crate::codec::Codec>(
+    index: &Index<Codec>,
     mut segment_entries: Vec<SegmentEntry>,
     target_opstamp: Opstamp,
 ) -> crate::Result<Option<SegmentEntry>> {
@@ -108,7 +109,7 @@ fn merge(
 
     let delete_cursor = segment_entries[0].delete_cursor().clone();
 
-    let segments: Vec<Segment> = segment_entries
+    let segments: Vec<Segment<Codec>> = segment_entries
         .iter()
         .map(|segment_entry| index.segment(segment_entry.meta().clone()))
         .collect();
@@ -139,10 +140,10 @@ fn merge(
 /// meant to work if you have an `IndexWriter` running for the origin indices, or
 /// the destination `Index`.
 #[doc(hidden)]
-pub fn merge_indices<T: Into<Box<dyn Directory>>>(
-    indices: &[Index],
-    output_directory: T,
-) -> crate::Result<Index> {
+pub fn merge_indices<Codec: crate::codec::Codec>(
+    indices: &[Index<Codec>],
+    output_directory: Box<dyn Directory>,
+) -> crate::Result<Index<Codec>> {
     if indices.is_empty() {
         // If there are no indices to merge, there is no need to do anything.
         return Err(crate::TantivyError::InvalidArgument(
@@ -163,7 +164,7 @@ pub fn merge_indices<T: Into<Box<dyn Directory>>>(
         ));
     }
 
-    let mut segments: Vec<Segment> = Vec::new();
+    let mut segments: Vec<Segment<Codec>> = Vec::new();
     for index in indices {
         segments.extend(index.searchable_segments()?);
     }
@@ -185,12 +186,12 @@ pub fn merge_indices<T: Into<Box<dyn Directory>>>(
 /// meant to work if you have an `IndexWriter` running for the origin indices, or
 /// the destination `Index`.
 #[doc(hidden)]
-pub fn merge_filtered_segments<T: Into<Box<dyn Directory>>>(
-    segments: &[Segment],
+pub fn merge_filtered_segments<Codec: crate::codec::Codec, T: Into<Box<dyn Directory>>>(
+    segments: &[Segment<Codec>],
     target_settings: IndexSettings,
     filter_doc_ids: Vec<Option<AliveBitSet>>,
     output_directory: T,
-) -> crate::Result<Index> {
+) -> crate::Result<Index<Codec>> {
     if segments.is_empty() {
         // If there are no indices to merge, there is no need to do anything.
         return Err(crate::TantivyError::InvalidArgument(
@@ -211,11 +212,12 @@ pub fn merge_filtered_segments<T: Into<Box<dyn Directory>>>(
         ));
     }
 
-    let mut merged_index = Index::create(
-        output_directory,
-        target_schema.clone(),
-        target_settings.clone(),
-    )?;
+    let mut merged_index: Index<Codec> = Index::builder()
+        .schema(target_schema.clone())
+        .codec(segments[0].index().codec().clone())
+        .settings(target_settings.clone())
+        .create(output_directory.into())?;
+
     let merged_segment = merged_index.new_segment();
     let merged_segment_id = merged_segment.id();
     let merger: IndexMerger =
@@ -235,6 +237,7 @@ pub fn merge_filtered_segments<T: Into<Box<dyn Directory>>>(
             ))
             .trim_end()
     );
+    let codec_configuration = CodecConfiguration::from_codec(segments[0].index().codec());
 
     let index_meta = IndexMeta {
         index_settings: target_settings, // index_settings of all segments should be the same
@@ -242,6 +245,7 @@ pub fn merge_filtered_segments<T: Into<Box<dyn Directory>>>(
         schema: target_schema,
         opstamp: 0u64,
         payload: Some(stats),
+        codec: codec_configuration,
     };
 
     // save the meta.json
@@ -250,7 +254,7 @@ pub fn merge_filtered_segments<T: Into<Box<dyn Directory>>>(
     Ok(merged_index)
 }
 
-pub(crate) struct InnerSegmentUpdater {
+pub(crate) struct InnerSegmentUpdater<C: Codec> {
     // we keep a copy of the current active IndexMeta to
     // avoid loading the file every time we need it in the
     // `SegmentUpdater`.
@@ -261,7 +265,7 @@ pub(crate) struct InnerSegmentUpdater {
     pool: ThreadPool,
     merge_thread_pool: ThreadPool,
 
-    index: Index,
+    index: Index<C>,
     segment_manager: SegmentManager,
     merge_policy: RwLock<Arc<dyn MergePolicy>>,
     killed: AtomicBool,
@@ -269,13 +273,13 @@ pub(crate) struct InnerSegmentUpdater {
     merge_operations: MergeOperationInventory,
 }
 
-impl SegmentUpdater {
+impl<Codec: crate::codec::Codec> SegmentUpdater<Codec> {
     pub fn create(
-        index: Index,
+        index: Index<Codec>,
         stamper: Stamper,
         delete_cursor: &DeleteCursor,
         num_merge_threads: usize,
-    ) -> crate::Result<SegmentUpdater> {
+    ) -> crate::Result<Self> {
         let segments = index.searchable_segment_metas()?;
         let segment_manager = SegmentManager::from_segments(segments, delete_cursor);
         let pool = ThreadPoolBuilder::new()
@@ -404,12 +408,14 @@ impl SegmentUpdater {
             //
             // Segment 1 from disk 1, Segment 1 from disk 2, etc.
             committed_segment_metas.sort_by_key(|segment_meta| -(segment_meta.max_doc() as i32));
+            let codec = CodecConfiguration::from_codec(index.codec());
             let index_meta = IndexMeta {
                 index_settings: index.settings().clone(),
                 segments: committed_segment_metas,
                 schema: index.schema(),
                 opstamp,
                 payload: commit_message,
+                codec,
             };
             // TODO add context to the error.
             save_metas(&index_meta, directory.box_clone().borrow_mut())?;
@@ -443,7 +449,7 @@ impl SegmentUpdater {
         opstamp: Opstamp,
         payload: Option<String>,
     ) -> FutureResult<Opstamp> {
-        let segment_updater: SegmentUpdater = self.clone();
+        let segment_updater: SegmentUpdater<Codec> = self.clone();
         self.schedule_task(move || {
             let segment_entries = segment_updater.purge_deletes(opstamp)?;
             segment_updater.segment_manager.commit(segment_entries);
@@ -702,6 +708,7 @@ impl SegmentUpdater {
 #[cfg(test)]
 mod tests {
     use super::merge_indices;
+    use crate::codec::StandardCodec;
     use crate::collector::TopDocs;
     use crate::directory::RamDirectory;
     use crate::fastfield::AliveBitSet;
@@ -915,7 +922,7 @@ mod tests {
 
     #[test]
     fn test_merge_empty_indices_array() {
-        let merge_result = merge_indices(&[], RamDirectory::default());
+        let merge_result = merge_indices::<StandardCodec>(&[], Box::new(RamDirectory::default()));
         assert!(merge_result.is_err());
     }
 
@@ -942,7 +949,10 @@ mod tests {
         };
 
         // mismatched schema index list
-        let result = merge_indices(&[first_index, second_index], RamDirectory::default());
+        let result = merge_indices(
+            &[first_index, second_index],
+            Box::new(RamDirectory::default()),
+        );
         assert!(result.is_err());
 
         Ok(())

src/indexer/segment_writer.rs

@@ -4,6 +4,7 @@ use itertools::Itertools;
 use tokenizer_api::BoxTokenStream;
 
 use super::operation::AddOperation;
+use crate::codec::Codec;
 use crate::fastfield::FastFieldsWriter;
 use crate::fieldnorm::{FieldNormReaders, FieldNormsWriter};
 use crate::index::{Segment, SegmentComponent};
@@ -45,11 +46,11 @@ fn compute_initial_table_size(per_thread_memory_budget: usize) -> crate::Result<
 ///
 /// They creates the postings list in anonymous memory.
 /// The segment is laid on disk when the segment gets `finalized`.
-pub struct SegmentWriter {
+pub struct SegmentWriter<Codec: crate::codec::Codec> {
     pub(crate) max_doc: DocId,
     pub(crate) ctx: IndexingContext,
     pub(crate) per_field_postings_writers: PerFieldPostingsWriter,
-    pub(crate) segment_serializer: SegmentSerializer,
+    pub(crate) segment_serializer: SegmentSerializer<Codec>,
     pub(crate) fast_field_writers: FastFieldsWriter,
     pub(crate) fieldnorms_writer: FieldNormsWriter,
     pub(crate) json_path_writer: JsonPathWriter,
@@ -60,7 +61,7 @@ pub struct SegmentWriter {
     schema: Schema,
 }
 
-impl SegmentWriter {
+impl<Codec: crate::codec::Codec> SegmentWriter<Codec> {
     /// Creates a new `SegmentWriter`
     ///
     /// The arguments are defined as follows
@@ -70,7 +71,10 @@ impl SegmentWriter {
     ///   behavior as a memory limit.
     /// - segment: The segment being written
     /// - schema
-    pub fn for_segment(memory_budget_in_bytes: usize, segment: Segment) -> crate::Result<Self> {
+    pub fn for_segment(
+        memory_budget_in_bytes: usize,
+        segment: Segment<Codec>,
+    ) -> crate::Result<Self> {
         let schema = segment.schema();
         let tokenizer_manager = segment.index().tokenizers().clone();
         let tokenizer_manager_fast_field = segment.index().fast_field_tokenizer().clone();
@@ -386,13 +390,13 @@ impl SegmentWriter {
 /// to the `SegmentSerializer`.
 ///
 /// `doc_id_map` is used to map to the new doc_id order.
-fn remap_and_write(
+fn remap_and_write<C: Codec>(
     schema: Schema,
     per_field_postings_writers: &PerFieldPostingsWriter,
     ctx: IndexingContext,
     fast_field_writers: FastFieldsWriter,
     fieldnorms_writer: &FieldNormsWriter,
-    mut serializer: SegmentSerializer,
+    mut serializer: SegmentSerializer<C>,
 ) -> crate::Result<()> {
     debug!("remap-and-write");
     if let Some(fieldnorms_serializer) = serializer.extract_fieldnorms_serializer() {

src/indexer/single_segment_index_writer.rs

@@ -1,5 +1,6 @@
 use std::marker::PhantomData;
 
+use crate::codec::CodecConfiguration;
 use crate::indexer::operation::AddOperation;
 use crate::indexer::segment_updater::save_metas;
 use crate::indexer::SegmentWriter;
@@ -7,22 +8,22 @@ use crate::schema::document::Document;
 use crate::{Directory, Index, IndexMeta, Opstamp, Segment, TantivyDocument};
 
 #[doc(hidden)]
-pub struct SingleSegmentIndexWriter<D: Document = TantivyDocument> {
-    segment_writer: SegmentWriter,
-    segment: Segment,
+pub struct SingleSegmentIndexWriter<Codec: crate::codec::Codec, D: Document = TantivyDocument> {
+    segment_writer: SegmentWriter<Codec>,
+    segment: Segment<Codec>,
     opstamp: Opstamp,
-    _phantom: PhantomData<D>,
+    _doc: PhantomData<D>,
 }
 
-impl<D: Document> SingleSegmentIndexWriter<D> {
-    pub fn new(index: Index, mem_budget: usize) -> crate::Result<Self> {
+impl<Codec: crate::codec::Codec, D: Document> SingleSegmentIndexWriter<Codec, D> {
+    pub fn new(index: Index<Codec>, mem_budget: usize) -> crate::Result<Self> {
         let segment = index.new_segment();
         let segment_writer = SegmentWriter::for_segment(mem_budget, segment.clone())?;
         Ok(Self {
             segment_writer,
             segment,
             opstamp: 0,
-            _phantom: PhantomData,
+            _doc: PhantomData,
         })
     }
 
@@ -37,10 +38,10 @@ impl<D: Document> SingleSegmentIndexWriter<D> {
             .add_document(AddOperation { opstamp, document })
     }
 
-    pub fn finalize(self) -> crate::Result<Index> {
+    pub fn finalize(self) -> crate::Result<Index<Codec>> {
         let max_doc = self.segment_writer.max_doc();
         self.segment_writer.finalize()?;
-        let segment: Segment = self.segment.with_max_doc(max_doc);
+        let segment: Segment<Codec> = self.segment.with_max_doc(max_doc);
         let index = segment.index();
         let index_meta = IndexMeta {
             index_settings: index.settings().clone(),
@@ -48,6 +49,7 @@ impl<D: Document> SingleSegmentIndexWriter<D> {
             schema: index.schema(),
             opstamp: 0,
             payload: None,
+            codec: CodecConfiguration::from_codec(index.codec()),
         };
         save_metas(&index_meta, index.directory())?;
         index.directory().sync_directory()?;

src/lib.rs

@@ -166,6 +166,7 @@ mod functional_test;
 
 #[macro_use]
 mod macros;
+pub mod codec;
 mod future_result;
 
 // Re-exports
@@ -377,7 +378,7 @@ pub mod tests {
 
     use common::{BinarySerializable, FixedSize};
     use query_grammar::{UserInputAst, UserInputLeaf, UserInputLiteral};
-    use rand::distr::{Bernoulli, Uniform};
+    use rand::distributions::{Bernoulli, Uniform};
     use rand::rngs::StdRng;
     use rand::{Rng, SeedableRng};
     use time::OffsetDateTime;
@@ -428,7 +429,7 @@ pub mod tests {
     pub fn generate_nonunique_unsorted(max_value: u32, n_elems: usize) -> Vec<u32> {
         let seed: [u8; 32] = [1; 32];
         StdRng::from_seed(seed)
-            .sample_iter(&Uniform::new(0u32, max_value).unwrap())
+            .sample_iter(&Uniform::new(0u32, max_value))
             .take(n_elems)
             .collect::<Vec<u32>>()
     }

src/postings/block_segment_postings.rs

@@ -303,10 +303,10 @@ impl BlockSegmentPostings {
     }
 
     pub(crate) fn load_block(&mut self) {
+        let offset = self.skip_reader.byte_offset();
         if self.block_is_loaded() {
             return;
         }
-        let offset = self.skip_reader.byte_offset();
         match self.skip_reader.block_info() {
             BlockInfo::BitPacked {
                 doc_num_bits,

src/postings/compression/mod.rs

@@ -397,10 +397,7 @@ mod bench {
         let mut seed: [u8; 32] = [0; 32];
         seed[31] = seed_val;
         let mut rng = StdRng::from_seed(seed);
-        (0u32..)
-            .filter(|_| rng.random_bool(ratio))
-            .take(n)
-            .collect()
+        (0u32..).filter(|_| rng.gen_bool(ratio)).take(n).collect()
     }
 
     pub fn generate_array(n: usize, ratio: f64) -> Vec<u32> {

src/postings/json_postings_writer.rs

@@ -22,12 +22,6 @@ pub(crate) struct JsonPostingsWriter<Rec: Recorder> {
     non_str_posting_writer: SpecializedPostingsWriter<DocIdRecorder>,
 }
 
-impl<Rec: Recorder> From<JsonPostingsWriter<Rec>> for Box<dyn PostingsWriter> {
-    fn from(json_postings_writer: JsonPostingsWriter<Rec>) -> Box<dyn PostingsWriter> {
-        Box::new(json_postings_writer)
-    }
-}
-
 impl<Rec: Recorder> PostingsWriter for JsonPostingsWriter<Rec> {
     #[inline]
     fn subscribe(

src/postings/mod.rs

@@ -604,13 +604,13 @@ mod bench {
             let mut index_writer: IndexWriter = index.writer_for_tests().unwrap();
             for _ in 0..posting_list_size {
                 let mut doc = TantivyDocument::default();
-                if rng.random_bool(1f64 / 15f64) {
+                if rng.gen_bool(1f64 / 15f64) {
                     doc.add_text(text_field, "a");
                 }
-                if rng.random_bool(1f64 / 10f64) {
+                if rng.gen_bool(1f64 / 10f64) {
                     doc.add_text(text_field, "b");
                 }
-                if rng.random_bool(1f64 / 5f64) {
+                if rng.gen_bool(1f64 / 5f64) {
                     doc.add_text(text_field, "c");
                 }
                 doc.add_text(text_field, "d");

src/postings/per_field_postings_writer.rs

@@ -1,16 +1,15 @@
 use crate::postings::json_postings_writer::JsonPostingsWriter;
-use crate::postings::postings_writer::SpecializedPostingsWriter;
+use crate::postings::postings_writer::{PostingsWriterEnum, SpecializedPostingsWriter};
 use crate::postings::recorder::{DocIdRecorder, TermFrequencyRecorder, TfAndPositionRecorder};
-use crate::postings::PostingsWriter;
 use crate::schema::{Field, FieldEntry, FieldType, IndexRecordOption, Schema};
 
 pub(crate) struct PerFieldPostingsWriter {
-    per_field_postings_writers: Vec<Box<dyn PostingsWriter>>,
+    per_field_postings_writers: Vec<PostingsWriterEnum>,
 }
 
 impl PerFieldPostingsWriter {
     pub fn for_schema(schema: &Schema) -> Self {
-        let per_field_postings_writers = schema
+        let per_field_postings_writers: Vec<PostingsWriterEnum> = schema
             .fields()
             .map(|(_, field_entry)| posting_writer_from_field_entry(field_entry))
             .collect();
@@ -19,16 +18,16 @@ impl PerFieldPostingsWriter {
         }
     }
 
-    pub(crate) fn get_for_field(&self, field: Field) -> &dyn PostingsWriter {
-        self.per_field_postings_writers[field.field_id() as usize].as_ref()
+    pub(crate) fn get_for_field(&self, field: Field) -> &PostingsWriterEnum {
+        &self.per_field_postings_writers[field.field_id() as usize]
     }
 
-    pub(crate) fn get_for_field_mut(&mut self, field: Field) -> &mut dyn PostingsWriter {
-        self.per_field_postings_writers[field.field_id() as usize].as_mut()
+    pub(crate) fn get_for_field_mut(&mut self, field: Field) -> &mut PostingsWriterEnum {
+        &mut self.per_field_postings_writers[field.field_id() as usize]
     }
 }
 
-fn posting_writer_from_field_entry(field_entry: &FieldEntry) -> Box<dyn PostingsWriter> {
+fn posting_writer_from_field_entry(field_entry: &FieldEntry) -> PostingsWriterEnum {
     match *field_entry.field_type() {
         FieldType::Str(ref text_options) => text_options
             .get_indexing_options()
@@ -51,7 +50,7 @@ fn posting_writer_from_field_entry(field_entry: &FieldEntry) -> Box<dyn Postings
         | FieldType::Date(_)
         | FieldType::Bytes(_)
         | FieldType::IpAddr(_)
-        | FieldType::Facet(_) => Box::<SpecializedPostingsWriter<DocIdRecorder>>::default(),
+        | FieldType::Facet(_) => <SpecializedPostingsWriter<DocIdRecorder>>::default().into(),
         FieldType::JsonObject(ref json_object_options) => {
             if let Some(text_indexing_option) = json_object_options.get_text_indexing_options() {
                 match text_indexing_option.index_option() {

src/postings/postings_writer.rs

@@ -7,7 +7,10 @@ use stacker::Addr;
 use crate::fieldnorm::FieldNormReaders;
 use crate::indexer::indexing_term::IndexingTerm;
 use crate::indexer::path_to_unordered_id::OrderedPathId;
-use crate::postings::recorder::{BufferLender, Recorder};
+use crate::postings::json_postings_writer::JsonPostingsWriter;
+use crate::postings::recorder::{
+    BufferLender, DocIdRecorder, Recorder, TermFrequencyRecorder, TfAndPositionRecorder,
+};
 use crate::postings::{
     FieldSerializer, IndexingContext, InvertedIndexSerializer, PerFieldPostingsWriter,
 };
@@ -100,6 +103,141 @@ pub(crate) struct IndexingPosition {
     pub end_position: u32,
 }
 
+pub enum PostingsWriterEnum {
+    DocId(SpecializedPostingsWriter<DocIdRecorder>),
+    DocIdTf(SpecializedPostingsWriter<TermFrequencyRecorder>),
+    DocTfAndPosition(SpecializedPostingsWriter<TfAndPositionRecorder>),
+    JsonDocId(JsonPostingsWriter<DocIdRecorder>),
+    JsonDocIdTf(JsonPostingsWriter<TermFrequencyRecorder>),
+    JsonDocTfAndPosition(JsonPostingsWriter<TfAndPositionRecorder>),
+}
+
+impl From<SpecializedPostingsWriter<DocIdRecorder>> for PostingsWriterEnum {
+    fn from(doc_id_recorder_writer: SpecializedPostingsWriter<DocIdRecorder>) -> Self {
+        PostingsWriterEnum::DocId(doc_id_recorder_writer)
+    }
+}
+
+impl From<SpecializedPostingsWriter<TermFrequencyRecorder>> for PostingsWriterEnum {
+    fn from(doc_id_tf_recorder_writer: SpecializedPostingsWriter<TermFrequencyRecorder>) -> Self {
+        PostingsWriterEnum::DocIdTf(doc_id_tf_recorder_writer)
+    }
+}
+
+impl From<SpecializedPostingsWriter<TfAndPositionRecorder>> for PostingsWriterEnum {
+    fn from(
+        doc_id_tf_and_positions_recorder_writer: SpecializedPostingsWriter<TfAndPositionRecorder>,
+    ) -> Self {
+        PostingsWriterEnum::DocTfAndPosition(doc_id_tf_and_positions_recorder_writer)
+    }
+}
+
+impl From<JsonPostingsWriter<DocIdRecorder>> for PostingsWriterEnum {
+    fn from(doc_id_recorder_writer: JsonPostingsWriter<DocIdRecorder>) -> Self {
+        PostingsWriterEnum::JsonDocId(doc_id_recorder_writer)
+    }
+}
+
+impl From<JsonPostingsWriter<TermFrequencyRecorder>> for PostingsWriterEnum {
+    fn from(doc_id_tf_recorder_writer: JsonPostingsWriter<TermFrequencyRecorder>) -> Self {
+        PostingsWriterEnum::JsonDocIdTf(doc_id_tf_recorder_writer)
+    }
+}
+
+impl From<JsonPostingsWriter<TfAndPositionRecorder>> for PostingsWriterEnum {
+    fn from(
+        doc_id_tf_and_positions_recorder_writer: JsonPostingsWriter<TfAndPositionRecorder>,
+    ) -> Self {
+        PostingsWriterEnum::JsonDocTfAndPosition(doc_id_tf_and_positions_recorder_writer)
+    }
+}
+
+impl PostingsWriter for PostingsWriterEnum {
+    fn subscribe(&mut self, doc: DocId, pos: u32, term: &IndexingTerm, ctx: &mut IndexingContext) {
+        match self {
+            PostingsWriterEnum::DocId(writer) => writer.subscribe(doc, pos, term, ctx),
+            PostingsWriterEnum::DocIdTf(writer) => writer.subscribe(doc, pos, term, ctx),
+            PostingsWriterEnum::DocTfAndPosition(writer) => writer.subscribe(doc, pos, term, ctx),
+            PostingsWriterEnum::JsonDocId(writer) => writer.subscribe(doc, pos, term, ctx),
+            PostingsWriterEnum::JsonDocIdTf(writer) => writer.subscribe(doc, pos, term, ctx),
+            PostingsWriterEnum::JsonDocTfAndPosition(writer) => {
+                writer.subscribe(doc, pos, term, ctx)
+            }
+        }
+    }
+
+    fn serialize(
+        &self,
+        term_addrs: &[(Field, OrderedPathId, &[u8], Addr)],
+        ordered_id_to_path: &[&str],
+        ctx: &IndexingContext,
+        serializer: &mut FieldSerializer,
+    ) -> io::Result<()> {
+        match self {
+            PostingsWriterEnum::DocId(writer) => {
+                writer.serialize(term_addrs, ordered_id_to_path, ctx, serializer)
+            }
+            PostingsWriterEnum::DocIdTf(writer) => {
+                writer.serialize(term_addrs, ordered_id_to_path, ctx, serializer)
+            }
+            PostingsWriterEnum::DocTfAndPosition(writer) => {
+                writer.serialize(term_addrs, ordered_id_to_path, ctx, serializer)
+            }
+            PostingsWriterEnum::JsonDocId(writer) => {
+                writer.serialize(term_addrs, ordered_id_to_path, ctx, serializer)
+            }
+            PostingsWriterEnum::JsonDocIdTf(writer) => {
+                writer.serialize(term_addrs, ordered_id_to_path, ctx, serializer)
+            }
+            PostingsWriterEnum::JsonDocTfAndPosition(writer) => {
+                writer.serialize(term_addrs, ordered_id_to_path, ctx, serializer)
+            }
+        }
+    }
+
+    /// Tokenize a text and subscribe all of its token.
+    fn index_text(
+        &mut self,
+        doc_id: DocId,
+        token_stream: &mut dyn TokenStream,
+        term_buffer: &mut IndexingTerm,
+        ctx: &mut IndexingContext,
+        indexing_position: &mut IndexingPosition,
+    ) {
+        match self {
+            PostingsWriterEnum::DocId(writer) => {
+                writer.index_text(doc_id, token_stream, term_buffer, ctx, indexing_position)
+            }
+            PostingsWriterEnum::DocIdTf(writer) => {
+                writer.index_text(doc_id, token_stream, term_buffer, ctx, indexing_position)
+            }
+            PostingsWriterEnum::DocTfAndPosition(writer) => {
+                writer.index_text(doc_id, token_stream, term_buffer, ctx, indexing_position)
+            }
+            PostingsWriterEnum::JsonDocId(writer) => {
+                writer.index_text(doc_id, token_stream, term_buffer, ctx, indexing_position)
+            }
+            PostingsWriterEnum::JsonDocIdTf(writer) => {
+                writer.index_text(doc_id, token_stream, term_buffer, ctx, indexing_position)
+            }
+            PostingsWriterEnum::JsonDocTfAndPosition(writer) => {
+                writer.index_text(doc_id, token_stream, term_buffer, ctx, indexing_position)
+            }
+        }
+    }
+
+    fn total_num_tokens(&self) -> u64 {
+        match self {
+            PostingsWriterEnum::DocId(writer) => writer.total_num_tokens(),
+            PostingsWriterEnum::DocIdTf(writer) => writer.total_num_tokens(),
+            PostingsWriterEnum::DocTfAndPosition(writer) => writer.total_num_tokens(),
+            PostingsWriterEnum::JsonDocId(writer) => writer.total_num_tokens(),
+            PostingsWriterEnum::JsonDocIdTf(writer) => writer.total_num_tokens(),
+            PostingsWriterEnum::JsonDocTfAndPosition(writer) => writer.total_num_tokens(),
+        }
+    }
+}
+
 /// The `PostingsWriter` is in charge of receiving documenting
 /// and building a `Segment` in anonymous memory.
 ///
@@ -171,14 +309,6 @@ pub(crate) struct SpecializedPostingsWriter<Rec: Recorder> {
     _recorder_type: PhantomData<Rec>,
 }
 
-impl<Rec: Recorder> From<SpecializedPostingsWriter<Rec>> for Box<dyn PostingsWriter> {
-    fn from(
-        specialized_postings_writer: SpecializedPostingsWriter<Rec>,
-    ) -> Box<dyn PostingsWriter> {
-        Box::new(specialized_postings_writer)
-    }
-}
-
 impl<Rec: Recorder> SpecializedPostingsWriter<Rec> {
     #[inline]
     pub(crate) fn serialize_one_term(

src/postings/segment_postings.rs

@@ -168,20 +168,12 @@ impl DocSet for SegmentPostings {
         self.doc()
     }
 
-    #[inline]
     fn seek(&mut self, target: DocId) -> DocId {
         debug_assert!(self.doc() <= target);
         if self.doc() >= target {
             return self.doc();
         }
 
-        // As an optimization, if the block is already loaded, we can
-        // cheaply check the next doc.
-        self.cur = (self.cur + 1).min(COMPRESSION_BLOCK_SIZE - 1);
-        if self.doc() >= target {
-            return self.doc();
-        }
-
         // Delegate block-local search to BlockSegmentPostings::seek, which returns
         // the in-block index of the first doc >= target.
         self.cur = self.block_cursor.seek(target);

src/postings/serializer.rs

@@ -55,7 +55,9 @@ pub struct InvertedIndexSerializer {
 
 impl InvertedIndexSerializer {
     /// Open a new `InvertedIndexSerializer` for the given segment
-    pub fn open(segment: &mut Segment) -> crate::Result<InvertedIndexSerializer> {
+    pub fn open<C: crate::codec::Codec>(
+        segment: &mut Segment<C>,
+    ) -> crate::Result<InvertedIndexSerializer> {
         use crate::index::SegmentComponent::{Positions, Postings, Terms};
         let inv_index_serializer = InvertedIndexSerializer {
             terms_write: CompositeWrite::wrap(segment.open_write(Terms)?),

src/query/boolean_query/boolean_weight.rs

@@ -291,6 +291,18 @@ impl<TScoreCombiner: ScoreCombiner> BooleanWeight<TScoreCombiner> {
             }
         };
 
+        let exclude_scorer_opt: Option<Box<dyn Scorer>> = if exclude_scorers.is_empty() {
+            None
+        } else {
+            let exclude_specialized_scorer: SpecializedScorer =
+                scorer_union(exclude_scorers, DoNothingCombiner::default, num_docs);
+            Some(into_box_scorer(
+                exclude_specialized_scorer,
+                DoNothingCombiner::default,
+                num_docs,
+            ))
+        };
+
         let include_scorer = match (should_scorers, must_scorers) {
             (ShouldScorersCombinationMethod::Ignored, must_scorers) => {
                 // No SHOULD clauses (or they were absorbed into MUST).
@@ -368,23 +380,16 @@ impl<TScoreCombiner: ScoreCombiner> BooleanWeight<TScoreCombiner> {
                 }
             }
         };
-        if exclude_scorers.is_empty() {
-            return Ok(include_scorer);
-        }
-
-        let include_scorer_boxed = into_box_scorer(include_scorer, &score_combiner_fn, num_docs);
-        let scorer: Box<dyn Scorer> = if exclude_scorers.len() == 1 {
-            let exclude_scorer = exclude_scorers.pop().unwrap();
-            match exclude_scorer.downcast::<TermScorer>() {
-                // Cast to TermScorer succeeded
-                Ok(exclude_scorer) => Box::new(Exclude::new(include_scorer_boxed, *exclude_scorer)),
-                // We get back the original Box<dyn Scorer>
-                Err(exclude_scorer) => Box::new(Exclude::new(include_scorer_boxed, exclude_scorer)),
-            }
+        if let Some(exclude_scorer) = exclude_scorer_opt {
+            let include_scorer_boxed =
+                into_box_scorer(include_scorer, &score_combiner_fn, num_docs);
+            Ok(SpecializedScorer::Other(Box::new(Exclude::new(
+                include_scorer_boxed,
+                exclude_scorer,
+            ))))
         } else {
-            Box::new(Exclude::new(include_scorer_boxed, exclude_scorers))
-        };
-        Ok(SpecializedScorer::Other(scorer))
+            Ok(include_scorer)
+        }
     }
 }
 

src/query/boost_query.rs

@@ -1,6 +1,6 @@
 use std::fmt;
 
-use crate::docset::{SeekDangerResult, COLLECT_BLOCK_BUFFER_LEN};
+use crate::docset::COLLECT_BLOCK_BUFFER_LEN;
 use crate::fastfield::AliveBitSet;
 use crate::query::{EnableScoring, Explanation, Query, Scorer, Weight};
 use crate::{DocId, DocSet, Score, SegmentReader, Term};
@@ -104,8 +104,8 @@ impl<S: Scorer> DocSet for BoostScorer<S> {
     fn seek(&mut self, target: DocId) -> DocId {
         self.underlying.seek(target)
     }
-    fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {
-        self.underlying.seek_danger(target)
+    fn seek_into_the_danger_zone(&mut self, target: DocId) -> bool {
+        self.underlying.seek_into_the_danger_zone(target)
     }
 
     fn fill_buffer(&mut self, buffer: &mut [DocId; COLLECT_BLOCK_BUFFER_LEN]) -> usize {

src/query/disjunction.rs

@@ -1,7 +1,6 @@
 use std::cmp::Ordering;
 use std::collections::BinaryHeap;
 
-use crate::docset::SeekDangerResult;
 use crate::query::score_combiner::DoNothingCombiner;
 use crate::query::{ScoreCombiner, Scorer};
 use crate::{DocId, DocSet, Score, TERMINATED};
@@ -68,12 +67,10 @@ impl<T: Scorer> DocSet for ScorerWrapper<T> {
         self.current_doc = doc_id;
         doc_id
     }
-    fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {
-        let result = self.scorer.seek_danger(target);
-        if result == SeekDangerResult::Found {
-            self.current_doc = target;
-        }
-        result
+    fn seek_into_the_danger_zone(&mut self, target: DocId) -> bool {
+        let found = self.scorer.seek_into_the_danger_zone(target);
+        self.current_doc = self.scorer.doc();
+        found
     }
 
     fn doc(&self) -> DocId {

src/query/exclude.rs

@@ -1,71 +1,48 @@
-use crate::docset::{DocSet, SeekDangerResult, TERMINATED};
+use crate::docset::{DocSet, TERMINATED};
 use crate::query::Scorer;
 use crate::{DocId, Score};
 
-/// An exclusion set is a set of documents
-/// that should be excluded from a given DocSet.
+#[inline]
+fn is_within<TDocSetExclude: DocSet>(docset: &mut TDocSetExclude, doc: DocId) -> bool {
+    docset.doc() <= doc && docset.seek(doc) == doc
+}
+
+/// Filters a given `DocSet` by removing the docs from a given `DocSet`.
 ///
-/// It can be a single DocSet, or a Vec of DocSets.
-pub trait ExclusionSet: Send {
-    /// Returns `true` if the given `doc` is in the exclusion set.
-    fn contains(&mut self, doc: DocId) -> bool;
-}
-
-impl<TDocSet: DocSet> ExclusionSet for TDocSet {
-    #[inline]
-    fn contains(&mut self, doc: DocId) -> bool {
-        self.seek_danger(doc) == SeekDangerResult::Found
-    }
-}
-
-impl<TDocSet: DocSet> ExclusionSet for Vec<TDocSet> {
-    #[inline]
-    fn contains(&mut self, doc: DocId) -> bool {
-        for docset in self.iter_mut() {
-            if docset.seek_danger(doc) == SeekDangerResult::Found {
-                return true;
-            }
-        }
-        false
-    }
-}
-
-/// Filters a given `DocSet` by removing the docs from an exclusion set.
-///
-/// The excluding docsets have no impact on scoring.
-pub struct Exclude<TDocSet, TExclusionSet> {
+/// The excluding docset has no impact on scoring.
+pub struct Exclude<TDocSet, TDocSetExclude> {
     underlying_docset: TDocSet,
-    exclusion_set: TExclusionSet,
+    excluding_docset: TDocSetExclude,
 }
 
-impl<TDocSet, TExclusionSet> Exclude<TDocSet, TExclusionSet>
+impl<TDocSet, TDocSetExclude> Exclude<TDocSet, TDocSetExclude>
 where
     TDocSet: DocSet,
-    TExclusionSet: ExclusionSet,
+    TDocSetExclude: DocSet,
 {
     /// Creates a new `ExcludeScorer`
     pub fn new(
         mut underlying_docset: TDocSet,
-        mut exclusion_set: TExclusionSet,
-    ) -> Exclude<TDocSet, TExclusionSet> {
+        mut excluding_docset: TDocSetExclude,
+    ) -> Exclude<TDocSet, TDocSetExclude> {
         while underlying_docset.doc() != TERMINATED {
             let target = underlying_docset.doc();
-            if !exclusion_set.contains(target) {
+            if !is_within(&mut excluding_docset, target) {
                 break;
             }
             underlying_docset.advance();
         }
         Exclude {
             underlying_docset,
-            exclusion_set,
+            excluding_docset,
         }
     }
 }
 
-impl<TDocSet, TExclusionSet> DocSet for Exclude<TDocSet, TExclusionSet>
+impl<TDocSet, TDocSetExclude> DocSet for Exclude<TDocSet, TDocSetExclude>
 where
     TDocSet: DocSet,
-    TExclusionSet: ExclusionSet,
+    TDocSetExclude: DocSet,
 {
     fn advance(&mut self) -> DocId {
         loop {
@@ -73,7 +50,7 @@ where
             if candidate == TERMINATED {
                 return TERMINATED;
             }
-            if !self.exclusion_set.contains(candidate) {
+            if !is_within(&mut self.excluding_docset, candidate) {
                 return candidate;
             }
         }
@@ -84,7 +61,7 @@ where
         if candidate == TERMINATED {
             return TERMINATED;
         }
-        if !self.exclusion_set.contains(candidate) {
+        if !is_within(&mut self.excluding_docset, candidate) {
             return candidate;
         }
         self.advance()
@@ -102,10 +79,10 @@ where
     }
 }
 
-impl<TScorer, TExclusionSet> Scorer for Exclude<TScorer, TExclusionSet>
+impl<TScorer, TDocSetExclude> Scorer for Exclude<TScorer, TDocSetExclude>
 where
     TScorer: Scorer,
-    TExclusionSet: ExclusionSet + 'static,
+    TDocSetExclude: DocSet + 'static,
 {
     #[inline]
     fn score(&mut self) -> Score {

src/query/intersection.rs

@@ -1,5 +1,5 @@
 use super::size_hint::estimate_intersection;
-use crate::docset::{DocSet, SeekDangerResult, TERMINATED};
+use crate::docset::{DocSet, TERMINATED};
 use crate::query::term_query::TermScorer;
 use crate::query::{EmptyScorer, Scorer};
 use crate::{DocId, Score};
@@ -84,14 +84,6 @@ impl<TDocSet: DocSet> Intersection<TDocSet, TDocSet> {
         docsets.sort_by_key(|docset| docset.cost());
         go_to_first_doc(&mut docsets);
         let left = docsets.remove(0);
-        debug_assert!({
-            let doc = left.doc();
-            if doc == TERMINATED {
-                true
-            } else {
-                docsets.iter().all(|docset| docset.doc() == doc)
-            }
-        });
         let right = docsets.remove(0);
         Intersection {
             left,
@@ -116,61 +108,46 @@ impl<TDocSet: DocSet, TOtherDocSet: DocSet> DocSet for Intersection<TDocSet, TOt
     #[inline]
     fn advance(&mut self) -> DocId {
         let (left, right) = (&mut self.left, &mut self.right);
+        let mut candidate = left.advance();
+        if candidate == TERMINATED {
+            return TERMINATED;
+        }
 
-        // Invariant:
-        // - candidate is always <= to the next document in the intersection.
-        // - candidate strictly increases at every occurence of the loop.
-        let mut candidate = left.doc() + 1;
+        loop {
+            // In the first part we look for a document in the intersection
+            // of the two rarest `DocSet` in the intersection.
 
-        // Termination: candidate strictly increases.
-        'outer: while candidate < TERMINATED {
-            // As we enter the loop, we should always have candidate < next_doc.
-
-            candidate = left.seek(candidate);
-
-            // Left is positionned on `candidate`.
-            debug_assert_eq!(left.doc(), candidate);
-
-            if let SeekDangerResult::SeekLowerBound(seek_lower_bound) = right.seek_danger(candidate)
-            {
-                debug_assert!(
-                    seek_lower_bound == TERMINATED || seek_lower_bound > candidate,
-                    "seek_lower_bound {seek_lower_bound} must be greater than candidate \
-                     {candidate}"
-                );
-                candidate = seek_lower_bound;
-                continue;
-            }
-
-            // Left and right are positionned on `candidate`.
-            debug_assert_eq!(right.doc(), candidate);
-
-            for other in &mut self.others {
-                if let SeekDangerResult::SeekLowerBound(seek_lower_bound) =
-                    other.seek_danger(candidate)
-                {
-                    // One of the scorer does not match, let's restart at the top of the loop.
-                    debug_assert!(
-                        seek_lower_bound == TERMINATED || seek_lower_bound > candidate,
-                        "seek_lower_bound {seek_lower_bound} must be greater than candidate \
-                         {candidate}"
-                    );
-                    candidate = seek_lower_bound;
-                    continue 'outer;
+            loop {
+                if right.seek_into_the_danger_zone(candidate) {
+                    break;
+                }
+                let right_doc = right.doc();
+                // TODO: Think about which value would make sense here
+                // It depends on the DocSet implementation, when a seek would outweigh an advance.
+                if right_doc > candidate.wrapping_add(100) {
+                    candidate = left.seek(right_doc);
+                } else {
+                    candidate = left.advance();
+                }
+                if candidate == TERMINATED {
+                    return TERMINATED;
                 }
             }
 
-            // At this point all scorers are in a valid state, aligned on the next document in the
-            // intersection.
-            debug_assert!(self.others.iter().all(|docset| docset.doc() == candidate));
-            return candidate;
+            debug_assert_eq!(left.doc(), right.doc());
+            // test the remaining scorers
+            if self
+                .others
+                .iter_mut()
+                .all(|docset| docset.seek_into_the_danger_zone(candidate))
+            {
+                debug_assert_eq!(candidate, self.left.doc());
+                debug_assert_eq!(candidate, self.right.doc());
+                debug_assert!(self.others.iter().all(|docset| docset.doc() == candidate));
+                return candidate;
+            }
+            candidate = left.advance();
         }
-
-        // We make sure our docset is in a valid state.
-        // In particular, we want .doc() to return TERMINATED.
-        left.seek(TERMINATED);
-
-        TERMINATED
     }
 
     fn seek(&mut self, target: DocId) -> DocId {
@@ -189,19 +166,13 @@ impl<TDocSet: DocSet, TOtherDocSet: DocSet> DocSet for Intersection<TDocSet, TOt
     ///
     /// Some implementations may choose to advance past the target if beneficial for performance.
     /// The return value is `true` if the target is in the docset, and `false` otherwise.
-    fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {
-        if let SeekDangerResult::SeekLowerBound(new_target) = self.left.seek_danger(target) {
-            return SeekDangerResult::SeekLowerBound(new_target);
-        }
-        if let SeekDangerResult::SeekLowerBound(new_target) = self.right.seek_danger(target) {
-            return SeekDangerResult::SeekLowerBound(new_target);
-        }
-        for docset in &mut self.others {
-            if let SeekDangerResult::SeekLowerBound(new_target) = docset.seek_danger(target) {
-                return SeekDangerResult::SeekLowerBound(new_target);
-            }
-        }
-        SeekDangerResult::Found
+    fn seek_into_the_danger_zone(&mut self, target: DocId) -> bool {
+        self.left.seek_into_the_danger_zone(target)
+            && self.right.seek_into_the_danger_zone(target)
+            && self
+                .others
+                .iter_mut()
+                .all(|docset| docset.seek_into_the_danger_zone(target))
     }
 
     #[inline]
@@ -244,12 +215,9 @@ mod tests {
     use proptest::prelude::*;
 
     use super::Intersection;
-    use crate::collector::Count;
     use crate::docset::{DocSet, TERMINATED};
     use crate::postings::tests::test_skip_against_unoptimized;
-    use crate::query::{QueryParser, VecDocSet};
-    use crate::schema::{Schema, TEXT};
-    use crate::Index;
+    use crate::query::VecDocSet;
 
     #[test]
     fn test_intersection() {
@@ -336,58 +304,6 @@ mod tests {
         assert_eq!(intersection.doc(), TERMINATED);
     }
 
-    #[test]
-    fn test_intersection_abc() {
-        let a = VecDocSet::from(vec![2, 3, 6]);
-        let b = VecDocSet::from(vec![1, 3, 5]);
-        let c = VecDocSet::from(vec![1, 3, 5]);
-        let mut intersection = Intersection::new(vec![c, b, a], 10);
-        let mut docs = Vec::new();
-        use crate::DocSet;
-        while intersection.doc() != TERMINATED {
-            docs.push(intersection.doc());
-            intersection.advance();
-        }
-        assert_eq!(&docs, &[3]);
-    }
-
-    #[test]
-    fn test_intersection_termination() {
-        use crate::query::score_combiner::DoNothingCombiner;
-        use crate::query::{BufferedUnionScorer, ConstScorer, VecDocSet};
-
-        let a1 = ConstScorer::new(VecDocSet::from(vec![0u32, 10000]), 1.0);
-        let a2 = ConstScorer::new(VecDocSet::from(vec![0u32, 10000]), 1.0);
-
-        let mut b_scorers = vec![];
-        for _ in 0..2 {
-            // Union matches 0 and 10000.
-            b_scorers.push(ConstScorer::new(VecDocSet::from(vec![0, 10000]), 1.0));
-        }
-        // That's the union of two scores matching 0, and 10_000.
-        let union = BufferedUnionScorer::build(b_scorers, DoNothingCombiner::default, 30000);
-
-        // Mismatching scorer: matches 0 and 20000. We then append more docs at the end to ensure it
-        // is last.
-        let mut m_docs = vec![0, 20000];
-        for i in 30000..30100 {
-            m_docs.push(i);
-        }
-        let m = ConstScorer::new(VecDocSet::from(m_docs), 1.0);
-
-        // Costs: A1=2, A2=2, Union=4, M=102.
-        // Sorted: A1, A2, Union, M.
-        // Left=A1, Right=A2, Others=[Union, M].
-        let mut intersection = crate::query::intersect_scorers(
-            vec![Box::new(a1), Box::new(a2), Box::new(union), Box::new(m)],
-            40000,
-        );
-
-        while intersection.doc() != TERMINATED {
-            intersection.advance();
-        }
-    }
-
     // Strategy to generate sorted and deduplicated vectors of u32 document IDs
     fn sorted_deduped_vec(max_val: u32, max_size: usize) -> impl Strategy<Value = Vec<u32>> {
         prop::collection::vec(0..max_val, 0..max_size).prop_map(|mut vec| {
@@ -419,30 +335,6 @@ mod tests {
             }
             assert_eq!(intersection.doc(), TERMINATED);
         }
-    }
 
-    #[test]
-    fn test_bug_2811_intersection_candidate_should_increase() {
-        let mut schema_builder = Schema::builder();
-        let text_field = schema_builder.add_text_field("text", TEXT);
-        let schema = schema_builder.build();
-
-        let index = Index::create_in_ram(schema);
-        let mut writer = index.writer_for_tests().unwrap();
-        writer
-            .add_document(doc!(text_field=>"hello happy tax"))
-            .unwrap();
-        writer.add_document(doc!(text_field=>"hello")).unwrap();
-        writer.add_document(doc!(text_field=>"hello")).unwrap();
-        writer.add_document(doc!(text_field=>"happy tax")).unwrap();
-
-        writer.commit().unwrap();
-        let query_parser = QueryParser::for_index(&index, Vec::new());
-        let query = query_parser
-            .parse_query(r#"+text:hello +text:"happy tax""#)
-            .unwrap();
-        let searcher = index.reader().unwrap().searcher();
-        let c = searcher.search(&*query, &Count).unwrap();
-        assert_eq!(c, 1);
     }
 }

src/query/mod.rs

@@ -43,7 +43,7 @@ pub use self::boost_query::{BoostQuery, BoostWeight};
 pub use self::const_score_query::{ConstScoreQuery, ConstScorer};
 pub use self::disjunction_max_query::DisjunctionMaxQuery;
 pub use self::empty_query::{EmptyQuery, EmptyScorer, EmptyWeight};
-pub use self::exclude::{Exclude, ExclusionSet};
+pub use self::exclude::Exclude;
 pub use self::exist_query::ExistsQuery;
 pub use self::explanation::Explanation;
 #[cfg(test)]

src/query/phrase_prefix_query/phrase_prefix_scorer.rs

@@ -1,4 +1,4 @@
-use crate::docset::{DocSet, SeekDangerResult, TERMINATED};
+use crate::docset::{DocSet, TERMINATED};
 use crate::fieldnorm::FieldNormReader;
 use crate::postings::Postings;
 use crate::query::bm25::Bm25Weight;
@@ -194,16 +194,11 @@ impl<TPostings: Postings> DocSet for PhrasePrefixScorer<TPostings> {
         self.advance()
     }
 
-    fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {
-        let seek_res = self.phrase_scorer.seek_danger(target);
-        if seek_res != SeekDangerResult::Found {
-            return seek_res;
-        }
-        // The intersection matched. Now let's see if we match the prefix.
-        if self.matches_prefix() {
-            SeekDangerResult::Found
+    fn seek_into_the_danger_zone(&mut self, target: DocId) -> bool {
+        if self.phrase_scorer.seek_into_the_danger_zone(target) {
+            self.matches_prefix()
         } else {
-            SeekDangerResult::SeekLowerBound(target + 1)
+            false
         }
     }
 

src/query/phrase_query/phrase_scorer.rs

@@ -1,6 +1,6 @@
 use std::cmp::Ordering;
 
-use crate::docset::{DocSet, SeekDangerResult, TERMINATED};
+use crate::docset::{DocSet, TERMINATED};
 use crate::fieldnorm::FieldNormReader;
 use crate::postings::Postings;
 use crate::query::bm25::Bm25Weight;
@@ -530,23 +530,12 @@ impl<TPostings: Postings> DocSet for PhraseScorer<TPostings> {
         self.advance()
     }
 
-    fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {
-        debug_assert!(
-            target >= self.doc(),
-            "target ({}) should be greater than or equal to doc ({})",
-            target,
-            self.doc()
-        );
-        let seek_res = self.intersection_docset.seek_danger(target);
-        if seek_res != SeekDangerResult::Found {
-            return seek_res;
-        }
-        // The intersection matched. Now let's see if we match the phrase.
-        if self.phrase_match() {
-            SeekDangerResult::Found
-        } else {
-            SeekDangerResult::SeekLowerBound(target + 1)
+    fn seek_into_the_danger_zone(&mut self, target: DocId) -> bool {
+        debug_assert!(target >= self.doc());
+        if self.intersection_docset.seek_into_the_danger_zone(target) && self.phrase_match() {
+            return true;
         }
+        false
     }
 
     fn doc(&self) -> DocId {

src/query/phrase_query/regex_phrase_weight.rs

@@ -311,7 +311,7 @@ mod tests {
         #![proptest_config(ProptestConfig::with_cases(50))]
         #[test]
         fn test_phrase_regex_with_random_strings(mut random_strings in proptest::collection::vec("[c-z ]{0,10}", 1..100), num_occurrences in 1..150_usize) {
-            let mut rng = rand::rng();
+            let mut rng = rand::thread_rng();
 
             // Insert "aaa ccc" the specified number of times into the list
             for _ in 0..num_occurrences {

src/query/range_query/range_query.rs

@@ -429,7 +429,7 @@ mod tests {
                 docs.push(doc);
             }
 
-            docs.shuffle(&mut rand::rng());
+            docs.shuffle(&mut rand::thread_rng());
             let mut docs_it = docs.into_iter();
             for doc in (&mut docs_it).take(50) {
                 index_writer.add_document(doc)?;

src/query/range_query/range_query_fastfield.rs

@@ -491,7 +491,7 @@ mod tests {
     use common::DateTime;
     use proptest::prelude::*;
     use rand::rngs::StdRng;
-    use rand::seq::IndexedRandom;
+    use rand::seq::SliceRandom;
     use rand::SeedableRng;
     use time::format_description::well_known::Rfc3339;
     use time::OffsetDateTime;

src/query/reqopt_scorer.rs

@@ -1,6 +1,6 @@
 use std::marker::PhantomData;
 
-use crate::docset::{DocSet, SeekDangerResult};
+use crate::docset::DocSet;
 use crate::query::score_combiner::ScoreCombiner;
 use crate::query::Scorer;
 use crate::{DocId, Score};
@@ -56,9 +56,9 @@ where
         self.req_scorer.seek(target)
     }
 
-    fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {
+    fn seek_into_the_danger_zone(&mut self, target: DocId) -> bool {
         self.score_cache = None;
-        self.req_scorer.seek_danger(target)
+        self.req_scorer.seek_into_the_danger_zone(target)
     }
 
     fn doc(&self) -> DocId {

src/query/term_query/term_scorer.rs

@@ -105,7 +105,6 @@ impl DocSet for TermScorer {
 
     #[inline]
     fn seek(&mut self, target: DocId) -> DocId {
-        debug_assert!(target >= self.doc());
         self.postings.seek(target)
     }
 
@@ -305,10 +304,10 @@ mod tests {
         let mut writer: IndexWriter =
             index.writer_with_num_threads(3, 3 * MEMORY_BUDGET_NUM_BYTES_MIN)?;
         use rand::Rng;
-        let mut rng = rand::rng();
+        let mut rng = rand::thread_rng();
         writer.set_merge_policy(Box::new(NoMergePolicy));
         for _ in 0..3_000 {
-            let term_freq = rng.random_range(1..10000);
+            let term_freq = rng.gen_range(1..10000);
             let words: Vec<&str> = std::iter::repeat_n("bbbb", term_freq).collect();
             let text = words.join(" ");
             writer.add_document(doc!(text_field=>text))?;

src/query/union/buffered_union.rs

@@ -1,6 +1,6 @@
 use common::TinySet;
 
-use crate::docset::{DocSet, SeekDangerResult, TERMINATED};
+use crate::docset::{DocSet, TERMINATED};
 use crate::query::score_combiner::{DoNothingCombiner, ScoreCombiner};
 use crate::query::size_hint::estimate_union;
 use crate::query::Scorer;
@@ -225,47 +225,25 @@ where
         }
     }
 
-    fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {
-        if target >= TERMINATED {
-            return SeekDangerResult::SeekLowerBound(TERMINATED);
-        }
+    fn seek_into_the_danger_zone(&mut self, target: DocId) -> bool {
         if self.is_in_horizon(target) {
             // Our value is within the buffered horizon and the docset may already have been
             // processed and removed, so we need to use seek, which uses the regular advance.
-            let seek_doc = self.seek(target);
-            if seek_doc == target {
-                return SeekDangerResult::Found;
-            } else {
-                return SeekDangerResult::SeekLowerBound(seek_doc);
-            };
-        }
-
-        // The docsets are not in the buffered range, so we can use seek_into_the_danger_zone
-        // of the underlying docsets
-        let mut is_hit = false;
-        let mut min_new_target = TERMINATED;
-
-        for docset in self.docsets.iter_mut() {
-            match docset.seek_danger(target) {
-                SeekDangerResult::Found => {
-                    is_hit = true;
-                    break;
-                }
-                SeekDangerResult::SeekLowerBound(new_target) => {
-                    min_new_target = min_new_target.min(new_target);
-                }
-            }
-        }
-
-        // The API requires the DocSet to be in a valid state when `seek_into_the_danger_zone`
-        // returns Found.
-        if is_hit {
-            // The doc is found. Let's make sure we position the union on the target
-            // to bring it back to a valid state.
-            self.seek(target);
-            SeekDangerResult::Found
+            self.seek(target) == target
         } else {
-            SeekDangerResult::SeekLowerBound(min_new_target)
+            // The docsets are not in the buffered range, so we can use seek_into_the_danger_zone
+            // of the underlying docsets
+            let is_hit = self
+                .docsets
+                .iter_mut()
+                .any(|docset| docset.seek_into_the_danger_zone(target));
+
+            // The API requires the DocSet to be in a valid state when `seek_into_the_danger_zone`
+            // returns true.
+            if is_hit {
+                self.seek(target);
+            }
+            is_hit
         }
     }
 

src/query/union/mod.rs

@@ -14,7 +14,7 @@ mod tests {
     use common::BitSet;
 
     use super::{SimpleUnion, *};
-    use crate::docset::{DocSet, SeekDangerResult, TERMINATED};
+    use crate::docset::{DocSet, TERMINATED};
     use crate::postings::tests::test_skip_against_unoptimized;
     use crate::query::score_combiner::DoNothingCombiner;
     use crate::query::union::bitset_union::BitSetPostingUnion;
@@ -254,27 +254,6 @@ mod tests {
             vec![1, 2, 3, 7, 8, 9, 99, 100, 101, 500, 20000],
         );
     }
-
-    #[test]
-    fn test_buffered_union_seek_into_danger_zone_terminated() {
-        let scorer1 = ConstScorer::new(VecDocSet::from(vec![1, 2]), 1.0);
-        let scorer2 = ConstScorer::new(VecDocSet::from(vec![2, 3]), 1.0);
-
-        let mut union_scorer =
-            BufferedUnionScorer::build(vec![scorer1, scorer2], DoNothingCombiner::default, 100);
-
-        // Advance to end
-        while union_scorer.doc() != TERMINATED {
-            union_scorer.advance();
-        }
-
-        assert_eq!(union_scorer.doc(), TERMINATED);
-
-        assert_eq!(
-            union_scorer.seek_danger(TERMINATED),
-            SeekDangerResult::SeekLowerBound(TERMINATED)
-        );
-    }
 }
 
 #[cfg(all(test, feature = "unstable"))]

src/query/vec_docset.rs

@@ -17,9 +17,6 @@ pub struct VecDocSet {
 
 impl From<Vec<DocId>> for VecDocSet {
     fn from(doc_ids: Vec<DocId>) -> VecDocSet {
-        // We do not use `slice::is_sorted`, as we want to check for doc ids to be strictly
-        // sorted.
-        assert!(doc_ids.windows(2).all(|w| w[0] < w[1]));
         VecDocSet { doc_ids, cursor: 0 }
     }
 }

src/reader/mod.rs

@@ -7,6 +7,7 @@ use arc_swap::ArcSwap;
 pub use warming::Warmer;
 
 use self::warming::WarmingState;
+use crate::codec::Codec;
 use crate::core::searcher::{SearcherGeneration, SearcherInner};
 use crate::directory::{Directory, WatchCallback, WatchHandle, META_LOCK};
 use crate::store::DOCSTORE_CACHE_CAPACITY;
@@ -38,17 +39,17 @@ pub enum ReloadPolicy {
 /// - number of warming threads, for parallelizing warming work
 /// - The cache size of the underlying doc store readers.
 #[derive(Clone)]
-pub struct IndexReaderBuilder {
+pub struct IndexReaderBuilder<C: Codec = crate::codec::StandardCodec> {
     reload_policy: ReloadPolicy,
-    index: Index,
+    index: Index<C>,
     warmers: Vec<Weak<dyn Warmer>>,
     num_warming_threads: usize,
     doc_store_cache_num_blocks: usize,
 }
 
-impl IndexReaderBuilder {
+impl<C: Codec> IndexReaderBuilder<C> {
     #[must_use]
-    pub(crate) fn new(index: Index) -> IndexReaderBuilder {
+    pub(crate) fn new(index: Index<C>) -> IndexReaderBuilder<C> {
         IndexReaderBuilder {
             reload_policy: ReloadPolicy::OnCommitWithDelay,
             index,
@@ -63,7 +64,7 @@ impl IndexReaderBuilder {
     /// Building the reader is a non-trivial operation that requires
     /// to open different segment readers. It may take hundreds of milliseconds
     /// of time and it may return an error.
-    pub fn try_into(self) -> crate::Result<IndexReader> {
+    pub fn try_into(self) -> crate::Result<IndexReader<C>> {
         let searcher_generation_inventory = Inventory::default();
         let warming_state = WarmingState::new(
             self.num_warming_threads,
@@ -106,7 +107,7 @@ impl IndexReaderBuilder {
     ///
     /// See [`ReloadPolicy`] for more details.
     #[must_use]
-    pub fn reload_policy(mut self, reload_policy: ReloadPolicy) -> IndexReaderBuilder {
+    pub fn reload_policy(mut self, reload_policy: ReloadPolicy) -> IndexReaderBuilder<C> {
         self.reload_policy = reload_policy;
         self
     }
@@ -118,14 +119,14 @@ impl IndexReaderBuilder {
     pub fn doc_store_cache_num_blocks(
         mut self,
         doc_store_cache_num_blocks: usize,
-    ) -> IndexReaderBuilder {
+    ) -> IndexReaderBuilder<C> {
         self.doc_store_cache_num_blocks = doc_store_cache_num_blocks;
         self
     }
 
     /// Set the [`Warmer`]s that are invoked when reloading searchable segments.
     #[must_use]
-    pub fn warmers(mut self, warmers: Vec<Weak<dyn Warmer>>) -> IndexReaderBuilder {
+    pub fn warmers(mut self, warmers: Vec<Weak<dyn Warmer>>) -> IndexReaderBuilder<C> {
         self.warmers = warmers;
         self
     }
@@ -135,33 +136,33 @@ impl IndexReaderBuilder {
     /// This allows parallelizing warming work when there are multiple [`Warmer`] registered with
     /// the [`IndexReader`].
     #[must_use]
-    pub fn num_warming_threads(mut self, num_warming_threads: usize) -> IndexReaderBuilder {
+    pub fn num_warming_threads(mut self, num_warming_threads: usize) -> IndexReaderBuilder<C> {
         self.num_warming_threads = num_warming_threads;
         self
     }
 }
 
-impl TryInto<IndexReader> for IndexReaderBuilder {
+impl<C: Codec> TryInto<IndexReader<C>> for IndexReaderBuilder<C> {
     type Error = crate::TantivyError;
 
-    fn try_into(self) -> crate::Result<IndexReader> {
+    fn try_into(self) -> crate::Result<IndexReader<C>> {
         IndexReaderBuilder::try_into(self)
     }
 }
 
-struct InnerIndexReader {
+struct InnerIndexReader<C: Codec> {
     doc_store_cache_num_blocks: usize,
-    index: Index,
+    index: Index<C>,
     warming_state: WarmingState,
     searcher: arc_swap::ArcSwap<SearcherInner>,
     searcher_generation_counter: Arc<AtomicU64>,
     searcher_generation_inventory: Inventory<SearcherGeneration>,
 }
 
-impl InnerIndexReader {
+impl<C: Codec> InnerIndexReader<C> {
     fn new(
         doc_store_cache_num_blocks: usize,
-        index: Index,
+        index: Index<C>,
         warming_state: WarmingState,
         // The searcher_generation_inventory is not used as source, but as target to track the
         // loaded segments.
@@ -189,7 +190,7 @@ impl InnerIndexReader {
     ///
     /// This function acquires a lock to prevent GC from removing files
     /// as we are opening our index.
-    fn open_segment_readers(index: &Index) -> crate::Result<Vec<SegmentReader>> {
+    fn open_segment_readers(index: &Index<C>) -> crate::Result<Vec<SegmentReader>> {
         // Prevents segment files from getting deleted while we are in the process of opening them
         let _meta_lock = index.directory().acquire_lock(&META_LOCK)?;
         let searchable_segments = index.searchable_segments()?;
@@ -212,7 +213,7 @@ impl InnerIndexReader {
     }
 
     fn create_searcher(
-        index: &Index,
+        index: &Index<C>,
         doc_store_cache_num_blocks: usize,
         warming_state: &WarmingState,
         searcher_generation_counter: &Arc<AtomicU64>,
@@ -226,9 +227,10 @@ impl InnerIndexReader {
         );
 
         let schema = index.schema();
+        // SearcherInner uses Index<StandardCodec> since the codec doesn't affect reading
         let searcher = Arc::new(SearcherInner::new(
             schema,
-            index.clone(),
+            index.with_standard_codec(),
             segment_readers,
             searcher_generation,
             doc_store_cache_num_blocks,
@@ -264,14 +266,14 @@ impl InnerIndexReader {
 ///
 /// `IndexReader` just wraps an `Arc`.
 #[derive(Clone)]
-pub struct IndexReader {
-    inner: Arc<InnerIndexReader>,
+pub struct IndexReader<C: Codec = crate::codec::StandardCodec> {
+    inner: Arc<InnerIndexReader<C>>,
     _watch_handle_opt: Option<WatchHandle>,
 }
 
-impl IndexReader {
+impl<C: Codec> IndexReader<C> {
     #[cfg(test)]
-    pub(crate) fn index(&self) -> Index {
+    pub(crate) fn index(&self) -> Index<C> {
         self.inner.index.clone()
     }
 

src/space_usage/mod.rs

@@ -124,6 +124,7 @@ impl SegmentSpaceUsage {
             FieldNorms => PerField(self.fieldnorms().clone()),
             Terms => PerField(self.termdict().clone()),
             SegmentComponent::Store => ComponentSpaceUsage::Store(self.store().clone()),
+            SegmentComponent::TempStore => ComponentSpaceUsage::Store(self.store().clone()),
             Delete => Basic(self.deletes()),
         }
     }

src/termdict/fst_termdict/merger.rs

@@ -95,7 +95,7 @@ impl<'a> TermMerger<'a> {
 #[cfg(all(test, feature = "unstable"))]
 mod bench {
     use rand::distributions::Alphanumeric;
-    use rand::{rng, Rng};
+    use rand::{thread_rng, Rng};
     use test::{self, Bencher};
 
     use super::TermMerger;
@@ -117,9 +117,9 @@ mod bench {
         let buffer: Vec<u8> = {
             let mut terms = vec![];
             for _i in 0..num_terms {
-                let rand_string: String = rng()
+                let rand_string: String = thread_rng()
                     .sample_iter(&Alphanumeric)
-                    .take(rng().random_range(30..42))
+                    .take(thread_rng().gen_range(30..42))
                     .map(char::from)
                     .collect();
                 terms.push(rand_string);

sstable/Cargo.toml

@@ -25,7 +25,7 @@ zstd-compression = ["zstd"]
 proptest = "1"
 criterion = { version = "0.5", default-features = false }
 names = "0.14"
-rand = "0.9"
+rand = "0.8"
 
 [[bench]]
 name = "stream_bench"

sstable/benches/stream_bench.rs

@@ -10,9 +10,9 @@ use tantivy_sstable::{Dictionary, MonotonicU64SSTable};
 const CHARSET: &[u8] = b"abcdefghij";
 
 fn generate_key(rng: &mut impl Rng) -> String {
-    let len = rng.random_range(3..12);
+    let len = rng.gen_range(3..12);
     std::iter::from_fn(|| {
-        let idx = rng.random_range(0..CHARSET.len());
+        let idx = rng.gen_range(0..CHARSET.len());
         Some(CHARSET[idx] as char)
     })
     .take(len)

stacker/Cargo.toml

@@ -23,12 +23,12 @@ name = "hashmap"
 path = "example/hashmap.rs"
 
 [dev-dependencies]
-rand = "0.9"
+rand = "0.8.5"
 zipf = "7.0.0"
 rustc-hash = "2.1.0"
 proptest = "1.2.0"
 binggan = { version = "0.14.0" }
-rand_distr = "0.5"
+rand_distr = "0.4.3"
 
 [features]
 compare_hash_only = ["ahash"] # Compare hash only, not the key in the Hashmap

stacker/benches/bench.rs

@@ -90,10 +90,10 @@ fn bench_vint() {
             }
             // benchmark zipfs distribution numbers
             {
-                use rand::distr::Distribution;
+                use rand::distributions::Distribution;
                 use rand::rngs::StdRng;
                 let mut rng = StdRng::from_seed([3u8; 32]);
-                let zipf = rand_distr::Zipf::new(10_000.0f64, 1.03).unwrap();
+                let zipf = zipf::ZipfDistribution::new(10_000, 1.03).unwrap();
                 let numbers: Vec<[u8; 8]> = (0..num_numbers)
                     .map(|_| zipf.sample(&mut rng).to_le_bytes())
                     .collect();

stacker/fuzz_test/Cargo.toml

@@ -7,8 +7,8 @@ edition = "2021"
 
 [dependencies]
 ahash = "0.8.7"
-rand = "0.9"
-rand_distr = "0.5"
+rand = "0.8.5"
+rand_distr = "0.4.3"
 tantivy-stacker = { version = "0.2.0", path = ".." }
 
 [workspace]

stacker/fuzz_test/src/main.rs

@@ -14,7 +14,7 @@ fn test_with_seed(seed: u64) {
     let mut hash_map = AHashMap::new();
     let mut arena_hashmap = ArenaHashMap::default();
     let mut rng = StdRng::seed_from_u64(seed);
-    let key_count = rng.random_range(1_000..=1_000_000);
+    let key_count = rng.gen_range(1_000..=1_000_000);
     let exp = Exp::new(0.05).unwrap();
 
     for _ in 0..key_count {