pub method on Term

this will remove the DocumentDeserialize (maybe added later in a
different form)

.claude/skills/rationalize-deps/SKILL.md

@@ -0,0 +1,125 @@
+---
+name: rationalize-deps
+description: Analyze Cargo.toml dependencies and attempt to remove unused features to reduce compile times and binary size
+---
+
+# Rationalize Dependencies
+
+This skill analyzes Cargo.toml dependencies to identify and remove unused features.
+
+## Overview
+
+Many crates enable features by default that may not be needed. This skill:
+1. Identifies dependencies with default features enabled
+2. Tests if `default-features = false` works
+3. Identifies which specific features are actually needed
+4. Verifies compilation after changes
+
+## Step 1: Identify the target
+
+Ask the user which crate(s) to analyze:
+- A specific crate name (e.g., "tokio", "serde")
+- A specific workspace member (e.g., "quickwit-search")
+- "all" to scan the entire workspace
+
+## Step 2: Analyze current dependencies
+
+For the workspace Cargo.toml (`quickwit/Cargo.toml`), list dependencies that:
+- Do NOT have `default-features = false`
+- Have default features that might be unnecessary
+
+Run: `cargo tree -p <crate> -f "{p} {f}" --edges features` to see what features are actually used.
+
+## Step 3: For each candidate dependency
+
+### 3a: Check the crate's default features
+
+Look up the crate on crates.io or check its Cargo.toml to understand:
+- What features are enabled by default
+- What each feature provides
+
+Use: `cargo metadata --format-version=1 | jq '.packages[] | select(.name == "<crate>") | .features'`
+
+### 3b: Try disabling default features
+
+Modify the dependency in `quickwit/Cargo.toml`:
+
+From:
+```toml
+some-crate = { version = "1.0" }
+```
+
+To:
+```toml
+some-crate = { version = "1.0", default-features = false }
+```
+
+### 3c: Run cargo check
+
+Run: `cargo check --workspace` (or target specific packages for faster feedback)
+
+If compilation fails:
+1. Read the error messages to identify which features are needed
+2. Add only the required features explicitly:
+   ```toml
+   some-crate = { version = "1.0", default-features = false, features = ["needed-feature"] }
+   ```
+3. Re-run cargo check
+
+### 3d: Binary search for minimal features
+
+If there are many default features, use binary search:
+1. Start with no features
+2. If it fails, add half the default features
+3. Continue until you find the minimal set
+
+## Step 4: Document findings
+
+For each dependency analyzed, report:
+- Original configuration
+- New configuration (if changed)
+- Features that were removed
+- Any features that are required
+
+## Step 5: Verify full build
+
+After all changes, run:
+```bash
+cargo check --workspace --all-targets
+cargo test --workspace --no-run
+```
+
+## Common Patterns
+
+### Serde
+Often only needs `derive`:
+```toml
+serde = { version = "1.0", default-features = false, features = ["derive", "std"] }
+```
+
+### Tokio
+Identify which runtime features are actually used:
+```toml
+tokio = { version = "1.0", default-features = false, features = ["rt-multi-thread", "macros", "sync"] }
+```
+
+### Reqwest
+Often doesn't need all TLS backends:
+```toml
+reqwest = { version = "0.11", default-features = false, features = ["rustls-tls", "json"] }
+```
+
+## Rollback
+
+If changes cause issues:
+```bash
+git checkout quickwit/Cargo.toml
+cargo check --workspace
+```
+
+## Tips
+
+- Start with large crates that have many default features (tokio, reqwest, hyper)
+- Use `cargo bloat --crates` to identify large dependencies
+- Check `cargo tree -d` for duplicate dependencies that might indicate feature conflicts
+- Some features are needed only for tests - consider using `[dev-dependencies]` features

.claude/skills/simple-pr/SKILL.md

@@ -0,0 +1,60 @@
+---
+name: simple-pr
+description: Create a simple PR from staged changes with an auto-generated commit message
+disable-model-invocation: true
+---
+
+# Simple PR
+
+Follow these steps to create a simple PR from staged changes:
+
+## Step 1: Check workspace state
+
+Run: `git status`
+
+Verify that all changes have been staged (no unstaged changes). If there are unstaged changes, abort and ask the user to stage their changes first with `git add`.
+
+Also verify that we are on the `main` branch. If not, abort and ask the user to switch to main first.
+
+## Step 2: Ensure main is up to date
+
+Run: `git pull origin main`
+
+This ensures we're working from the latest code.
+
+## Step 3: Review staged changes
+
+Run: `git diff --cached`
+
+Review the staged changes to understand what the PR will contain.
+
+## Step 4: Generate commit message
+
+Based on the staged changes, generate a concise commit message (1-2 sentences) that describes the "why" rather than the "what".
+
+Display the proposed commit message to the user and ask for confirmation before proceeding.
+
+## Step 5: Create a new branch
+
+Get the git username: `git config user.name | tr ' ' '-' | tr '[:upper:]' '[:lower:]'`
+
+Create a short, descriptive branch name based on the changes (e.g., `fix-typo-in-readme`, `add-retry-logic`, `update-deps`).
+
+Create and checkout the branch: `git checkout -b {username}/{short-descriptive-name}`
+
+## Step 6: Commit changes
+
+Commit with the message from step 3:
+```
+git commit -m "{commit-message}"
+```
+
+## Step 7: Push and open a PR
+
+Push the branch and open a PR:
+```
+git push -u origin {branch-name}
+gh pr create --title "{commit-message-title}" --body "{longer-description-if-needed}"
+```
+
+Report the PR URL to the user when complete.

Cargo.toml

@@ -15,7 +15,7 @@ rust-version = "1.85"
 exclude = ["benches/*.json", "benches/*.txt"]
 
 [dependencies]
-oneshot = "0.1.7"
+oneshot = "0.1.13"
 base64 = "0.22.0"
 byteorder = "1.4.3"
 crc32fast = "1.3.2"
@@ -193,3 +193,12 @@ harness = false
 [[bench]]
 name = "str_search_and_get"
 harness = false
+
+[[bench]]
+name = "merge_segments"
+harness = false
+
+[[bench]]
+name = "regex_all_terms"
+harness = false
+

benches/merge_segments.rs

@@ -0,0 +1,224 @@
+// Benchmarks segment merging
+//
+// Notes:
+// - Input segments are kept intact (no deletes / no IndexWriter merge).
+// - Output is written to a `NullDirectory` that discards all files except
+//  fieldnorms (needed for merging).
+
+use std::collections::HashMap;
+use std::io::{self, Write};
+use std::path::{Path, PathBuf};
+use std::sync::{Arc, RwLock};
+
+use binggan::{black_box, BenchRunner};
+use rand::prelude::*;
+use rand::rngs::StdRng;
+use rand::SeedableRng;
+use tantivy::directory::error::{DeleteError, OpenReadError, OpenWriteError};
+use tantivy::directory::{
+    AntiCallToken, Directory, FileHandle, OwnedBytes, TerminatingWrite, WatchCallback, WatchHandle,
+    WritePtr,
+};
+use tantivy::indexer::{merge_filtered_segments, NoMergePolicy};
+use tantivy::schema::{Schema, TEXT};
+use tantivy::{doc, HasLen, Index, IndexSettings, Segment};
+
+#[derive(Clone, Default, Debug)]
+struct NullDirectory {
+    blobs: Arc<RwLock<HashMap<PathBuf, OwnedBytes>>>,
+}
+
+struct NullWriter;
+
+impl Write for NullWriter {
+    fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
+        Ok(buf.len())
+    }
+
+    fn flush(&mut self) -> io::Result<()> {
+        Ok(())
+    }
+}
+
+impl TerminatingWrite for NullWriter {
+    fn terminate_ref(&mut self, _token: AntiCallToken) -> io::Result<()> {
+        Ok(())
+    }
+}
+
+struct InMemoryWriter {
+    path: PathBuf,
+    buffer: Vec<u8>,
+    blobs: Arc<RwLock<HashMap<PathBuf, OwnedBytes>>>,
+}
+
+impl Write for InMemoryWriter {
+    fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
+        self.buffer.extend_from_slice(buf);
+        Ok(buf.len())
+    }
+
+    fn flush(&mut self) -> io::Result<()> {
+        Ok(())
+    }
+}
+
+impl TerminatingWrite for InMemoryWriter {
+    fn terminate_ref(&mut self, _token: AntiCallToken) -> io::Result<()> {
+        let bytes = OwnedBytes::new(std::mem::take(&mut self.buffer));
+        self.blobs.write().unwrap().insert(self.path.clone(), bytes);
+        Ok(())
+    }
+}
+
+#[derive(Debug, Default)]
+struct NullFileHandle;
+impl HasLen for NullFileHandle {
+    fn len(&self) -> usize {
+        0
+    }
+}
+impl FileHandle for NullFileHandle {
+    fn read_bytes(&self, _range: std::ops::Range<usize>) -> io::Result<OwnedBytes> {
+        unimplemented!()
+    }
+}
+
+impl Directory for NullDirectory {
+    fn get_file_handle(&self, path: &Path) -> Result<Arc<dyn FileHandle>, OpenReadError> {
+        if let Some(bytes) = self.blobs.read().unwrap().get(path) {
+            return Ok(Arc::new(bytes.clone()));
+        }
+        Ok(Arc::new(NullFileHandle))
+    }
+
+    fn delete(&self, _path: &Path) -> Result<(), DeleteError> {
+        Ok(())
+    }
+
+    fn exists(&self, _path: &Path) -> Result<bool, OpenReadError> {
+        Ok(true)
+    }
+
+    fn open_write(&self, path: &Path) -> Result<WritePtr, OpenWriteError> {
+        let path_buf = path.to_path_buf();
+        if path.to_string_lossy().ends_with(".fieldnorm") {
+            let writer = InMemoryWriter {
+                path: path_buf,
+                buffer: Vec::new(),
+                blobs: Arc::clone(&self.blobs),
+            };
+            Ok(io::BufWriter::new(Box::new(writer)))
+        } else {
+            Ok(io::BufWriter::new(Box::new(NullWriter)))
+        }
+    }
+
+    fn atomic_read(&self, path: &Path) -> Result<Vec<u8>, OpenReadError> {
+        if let Some(bytes) = self.blobs.read().unwrap().get(path) {
+            return Ok(bytes.as_slice().to_vec());
+        }
+        Err(OpenReadError::FileDoesNotExist(path.to_path_buf()))
+    }
+
+    fn atomic_write(&self, _path: &Path, _data: &[u8]) -> io::Result<()> {
+        Ok(())
+    }
+
+    fn sync_directory(&self) -> io::Result<()> {
+        Ok(())
+    }
+
+    fn watch(&self, _watch_callback: WatchCallback) -> tantivy::Result<WatchHandle> {
+        Ok(WatchHandle::empty())
+    }
+}
+
+struct MergeScenario {
+    #[allow(dead_code)]
+    index: Index,
+    segments: Vec<Segment>,
+    settings: IndexSettings,
+    label: String,
+}
+
+fn build_index(
+    num_segments: usize,
+    docs_per_segment: usize,
+    tokens_per_doc: usize,
+    vocab_size: usize,
+) -> MergeScenario {
+    let mut schema_builder = Schema::builder();
+    let body = schema_builder.add_text_field("body", TEXT);
+    let schema = schema_builder.build();
+    let index = Index::create_in_ram(schema.clone());
+
+    assert!(vocab_size > 0);
+    let total_tokens = num_segments * docs_per_segment * tokens_per_doc;
+    let use_unique_terms = vocab_size >= total_tokens;
+    let mut rng = StdRng::from_seed([7u8; 32]);
+    let mut next_token_id: u64 = 0;
+
+    {
+        let mut writer = index.writer_with_num_threads(1, 256_000_000).unwrap();
+        writer.set_merge_policy(Box::new(NoMergePolicy));
+        for _ in 0..num_segments {
+            for _ in 0..docs_per_segment {
+                let mut tokens = Vec::with_capacity(tokens_per_doc);
+                for _ in 0..tokens_per_doc {
+                    let token_id = if use_unique_terms {
+                        let id = next_token_id;
+                        next_token_id += 1;
+                        id
+                    } else {
+                        rng.random_range(0..vocab_size as u64)
+                    };
+                    tokens.push(format!("term_{token_id}"));
+                }
+                writer.add_document(doc!(body => tokens.join(" "))).unwrap();
+            }
+            writer.commit().unwrap();
+        }
+    }
+
+    let segments = index.searchable_segments().unwrap();
+    let settings = index.settings().clone();
+    let label = format!(
+        "segments={}, docs/seg={}, tokens/doc={}, vocab={}",
+        num_segments, docs_per_segment, tokens_per_doc, vocab_size
+    );
+
+    MergeScenario {
+        index,
+        segments,
+        settings,
+        label,
+    }
+}
+
+fn main() {
+    let scenarios = vec![
+        build_index(8, 50_000, 12, 8),
+        build_index(16, 50_000, 12, 8),
+        build_index(16, 100_000, 12, 8),
+        build_index(8, 50_000, 8, 8 * 50_000 * 8),
+    ];
+
+    let mut runner = BenchRunner::new();
+    for scenario in scenarios {
+        let mut group = runner.new_group();
+        group.set_name(format!("merge_segments inv_index — {}", scenario.label));
+        let segments = scenario.segments.clone();
+        let settings = scenario.settings.clone();
+        group.register("merge", move |_| {
+            let output_dir = NullDirectory::default();
+            let filter_doc_ids = vec![None; segments.len()];
+            let merged_index =
+                merge_filtered_segments(&segments, settings.clone(), filter_doc_ids, output_dir)
+                    .unwrap();
+            black_box(merged_index);
+        });
+
+        group.run();
+    }
+}

benches/regex_all_terms.rs

@@ -0,0 +1,113 @@
+// Benchmarks regex query that matches all terms in a synthetic index.
+//
+// Corpus model:
+// - N unique terms: t000000, t000001, ...
+// - M docs
+// - K tokens per doc: doc i gets terms derived from (i, token_index)
+//
+// Query:
+// - Regex "t.*" to match all terms
+//
+// Run with:
+// - cargo bench --bench regex_all_terms
+//
+
+use std::fmt::Write;
+
+use binggan::{black_box, BenchRunner};
+use tantivy::collector::Count;
+use tantivy::query::RegexQuery;
+use tantivy::schema::{Schema, TEXT};
+use tantivy::{doc, Index, ReloadPolicy};
+
+const HEAP_SIZE_BYTES: usize = 200_000_000;
+
+#[derive(Clone, Copy)]
+struct BenchConfig {
+    num_terms: usize,
+    num_docs: usize,
+    tokens_per_doc: usize,
+}
+
+fn main() {
+    let configs = default_configs();
+
+    let mut runner = BenchRunner::new();
+    for config in configs {
+        let (index, text_field) = build_index(config, HEAP_SIZE_BYTES);
+        let reader = index
+            .reader_builder()
+            .reload_policy(ReloadPolicy::Manual)
+            .try_into()
+            .expect("reader");
+        let searcher = reader.searcher();
+        let query = RegexQuery::from_pattern("t.*", text_field).expect("regex query");
+
+        let mut group = runner.new_group();
+        group.set_name(format!(
+            "regex_all_terms_t{}_d{}_k{}",
+            config.num_terms, config.num_docs, config.tokens_per_doc
+        ));
+        group.register("regex_count", move |_| {
+            let count = searcher.search(&query, &Count).expect("search");
+            black_box(count);
+        });
+        group.run();
+    }
+}
+
+fn default_configs() -> Vec<BenchConfig> {
+    vec![
+        BenchConfig {
+            num_terms: 10_000,
+            num_docs: 100_000,
+            tokens_per_doc: 1,
+        },
+        BenchConfig {
+            num_terms: 10_000,
+            num_docs: 100_000,
+            tokens_per_doc: 8,
+        },
+        BenchConfig {
+            num_terms: 100_000,
+            num_docs: 100_000,
+            tokens_per_doc: 1,
+        },
+        BenchConfig {
+            num_terms: 100_000,
+            num_docs: 100_000,
+            tokens_per_doc: 8,
+        },
+    ]
+}
+
+fn build_index(config: BenchConfig, heap_size_bytes: usize) -> (Index, tantivy::schema::Field) {
+    let mut schema_builder = Schema::builder();
+    let text_field = schema_builder.add_text_field("text", TEXT);
+    let schema = schema_builder.build();
+    let index = Index::create_in_ram(schema);
+
+    let term_width = config.num_terms.to_string().len();
+    {
+        let mut writer = index
+            .writer_with_num_threads(1, heap_size_bytes)
+            .expect("writer");
+        let mut buffer = String::new();
+        for doc_id in 0..config.num_docs {
+            buffer.clear();
+            for token_idx in 0..config.tokens_per_doc {
+                if token_idx > 0 {
+                    buffer.push(' ');
+                }
+                let term_id = (doc_id * config.tokens_per_doc + token_idx) % config.num_terms;
+                write!(&mut buffer, "t{term_id:0term_width$}").expect("write token");
+            }
+            writer
+                .add_document(doc!(text_field => buffer.as_str()))
+                .expect("add_document");
+        }
+        writer.commit().expect("commit");
+    }
+
+    (index, text_field)
+}

benches/str_search_and_get.rs

@@ -17,7 +17,6 @@ use rand::rngs::StdRng;
 use rand::SeedableRng;
 use tantivy::collector::{Count, DocSetCollector};
 use tantivy::query::RangeQuery;
-use tantivy::schema::document::TantivyDocument;
 use tantivy::schema::{Schema, Value, FAST, STORED, STRING};
 use tantivy::{doc, Index, ReloadPolicy, Searcher, Term};
 
@@ -406,7 +405,7 @@ impl FetchAllStringsFromDocTask {
 
         for doc_address in docs {
             // Get the document from the doc store (row store access)
-            if let Ok(doc) = self.searcher.doc::<TantivyDocument>(doc_address) {
+            if let Ok(doc) = self.searcher.doc(doc_address) {
                 // Extract string values from the stored field
                 if let Some(field_value) = doc.get_first(str_stored_field) {
                     if let Some(text) = field_value.as_value().as_str() {

common/src/writer.rs

@@ -62,7 +62,9 @@ impl<W: TerminatingWrite> TerminatingWrite for CountingWriter<W> {
 pub struct AntiCallToken(());
 
 /// Trait used to indicate when no more write need to be done on a writer
-pub trait TerminatingWrite: Write + Send + Sync {
+///
+/// Thread-safety is enforced at the call sites that require it.
+pub trait TerminatingWrite: Write {
     /// Indicate that the writer will no longer be used. Internally call terminate_ref.
     fn terminate(mut self) -> io::Result<()>
     where Self: Sized {

doc/src/json.md

@@ -60,7 +60,7 @@ At indexing, tantivy will try to interpret number and strings as different type
 priority order.
 
 Numbers will be interpreted as u64, i64 and f64 in that order.
-Strings will be interpreted as rfc3999 dates or simple strings.
+Strings will be interpreted as rfc3339 dates or simple strings.
 
 The first working type is picked and is the only term that is emitted for indexing.
 Note this interpretation happens on a per-document basis, and there is no effort to try to sniff
@@ -81,7 +81,7 @@ Will be interpreted as
 (my_path.my_segment, String, 233) or (my_path.my_segment, u64, 233)
 ```
 
-Likewise, we need to emit two tokens if the query contains an rfc3999 date.
+Likewise, we need to emit two tokens if the query contains an rfc3339 date.
 Indeed the date could have been actually a single token inside the text of a document at ingestion time. Generally speaking, we will always at least emit a string token in query parsing, and sometimes more.
 
 If one more json field is defined, things get even more complicated.

examples/custom_collector.rs

@@ -70,7 +70,7 @@ impl Collector for StatsCollector {
     fn for_segment(
         &self,
         _segment_local_id: u32,
-        segment_reader: &SegmentReader,
+        segment_reader: &dyn SegmentReader,
     ) -> tantivy::Result<StatsSegmentCollector> {
         let fast_field_reader = segment_reader.fast_fields().u64(&self.field)?;
         Ok(StatsSegmentCollector {

examples/date_time_field.rs

@@ -60,7 +60,7 @@ fn main() -> tantivy::Result<()> {
         let count_docs = searcher.search(&*query, &TopDocs::with_limit(4).order_by_score())?;
         assert_eq!(count_docs.len(), 1);
         for (_score, doc_address) in count_docs {
-            let retrieved_doc = searcher.doc::<TantivyDocument>(doc_address)?;
+            let retrieved_doc = searcher.doc(doc_address)?;
             assert!(retrieved_doc
                 .get_first(occurred_at)
                 .unwrap()

examples/faceted_search_with_tweaked_score.rs

@@ -65,7 +65,7 @@ fn main() -> tantivy::Result<()> {
         );
         let top_docs_by_custom_score =
             // Call TopDocs with a custom tweak score
-            TopDocs::with_limit(2).tweak_score(move |segment_reader: &SegmentReader| {
+            TopDocs::with_limit(2).tweak_score(move |segment_reader: &dyn SegmentReader| {
                 let ingredient_reader = segment_reader.facet_reader("ingredient").unwrap();
                 let facet_dict = ingredient_reader.facet_dict();
 
@@ -91,7 +91,7 @@ fn main() -> tantivy::Result<()> {
             .iter()
             .map(|(_, doc_id)| {
                 searcher
-                    .doc::<TantivyDocument>(*doc_id)
+                    .doc(*doc_id)
                     .unwrap()
                     .get_first(title)
                     .and_then(|v| v.as_str().map(|el| el.to_string()))

examples/phrase_prefix_search.rs

@@ -67,7 +67,7 @@ fn main() -> Result<()> {
     let mut titles = top_docs
         .into_iter()
         .map(|(_score, doc_address)| {
-            let doc = searcher.doc::<TantivyDocument>(doc_address)?;
+            let doc = searcher.doc(doc_address)?;
             let title = doc
                 .get_first(title)
                 .and_then(|v| v.as_str())

examples/snippet.rs

@@ -55,7 +55,7 @@ fn main() -> tantivy::Result<()> {
     let snippet_generator = SnippetGenerator::create(&searcher, &*query, body)?;
 
     for (score, doc_address) in top_docs {
-        let doc = searcher.doc::<TantivyDocument>(doc_address)?;
+        let doc = searcher.doc(doc_address)?;
         let snippet = snippet_generator.snippet_from_doc(&doc);
         println!("Document score {score}:");
         println!("title: {}", doc.get_first(title).unwrap().as_str().unwrap());

examples/warmer.rs

@@ -43,7 +43,7 @@ impl DynamicPriceColumn {
         }
     }
 
-    pub fn price_for_segment(&self, segment_reader: &SegmentReader) -> Option<Arc<Vec<Price>>> {
+    pub fn price_for_segment(&self, segment_reader: &dyn SegmentReader) -> Option<Arc<Vec<Price>>> {
         let segment_key = (segment_reader.segment_id(), segment_reader.delete_opstamp());
         self.price_cache.read().unwrap().get(&segment_key).cloned()
     }
@@ -157,7 +157,7 @@ fn main() -> tantivy::Result<()> {
     let query = query_parser.parse_query("cooking")?;
 
     let searcher = reader.searcher();
-    let score_by_price = move |segment_reader: &SegmentReader| {
+    let score_by_price = move |segment_reader: &dyn SegmentReader| {
         let price = price_dynamic_column
             .price_for_segment(segment_reader)
             .unwrap();

query-grammar/src/query_grammar.rs

@@ -560,7 +560,7 @@ fn range_infallible(inp: &str) -> JResult<&str, UserInputLeaf> {
             (
                 (
                     value((), tag(">=")),
-                    map(word_infallible("", false), |(bound, err)| {
+                    map(word_infallible(")", false), |(bound, err)| {
                         (
                             (
                                 bound
@@ -574,7 +574,7 @@ fn range_infallible(inp: &str) -> JResult<&str, UserInputLeaf> {
                 ),
                 (
                     value((), tag("<=")),
-                    map(word_infallible("", false), |(bound, err)| {
+                    map(word_infallible(")", false), |(bound, err)| {
                         (
                             (
                                 UserInputBound::Unbounded,
@@ -588,7 +588,7 @@ fn range_infallible(inp: &str) -> JResult<&str, UserInputLeaf> {
                 ),
                 (
                     value((), tag(">")),
-                    map(word_infallible("", false), |(bound, err)| {
+                    map(word_infallible(")", false), |(bound, err)| {
                         (
                             (
                                 bound
@@ -602,7 +602,7 @@ fn range_infallible(inp: &str) -> JResult<&str, UserInputLeaf> {
                 ),
                 (
                     value((), tag("<")),
-                    map(word_infallible("", false), |(bound, err)| {
+                    map(word_infallible(")", false), |(bound, err)| {
                         (
                             (
                                 UserInputBound::Unbounded,
@@ -704,7 +704,11 @@ fn regex(inp: &str) -> IResult<&str, UserInputLeaf> {
                 many1(alt((preceded(char('\\'), char('/')), none_of("/")))),
                 char('/'),
             ),
-            peek(alt((multispace1, eof))),
+            peek(alt((
+                value((), multispace1),
+                value((), char(')')),
+                value((), eof),
+            ))),
         ),
         |elements| UserInputLeaf::Regex {
             field: None,
@@ -721,8 +725,12 @@ fn regex_infallible(inp: &str) -> JResult<&str, UserInputLeaf> {
             opt_i_err(char('/'), "missing delimiter /"),
         ),
         opt_i_err(
-            peek(alt((multispace1, eof))),
-            "expected whitespace or end of input",
+            peek(alt((
+                value((), multispace1),
+                value((), char(')')),
+                value((), eof),
+            ))),
+            "expected whitespace, closing parenthesis, or end of input",
         ),
     )(inp)
     {
@@ -1323,6 +1331,14 @@ mod test {
         test_parse_query_to_ast_helper("<a", "{\"*\" TO \"a\"}");
         test_parse_query_to_ast_helper("<=a", "{\"*\" TO \"a\"]");
         test_parse_query_to_ast_helper("<=bsd", "{\"*\" TO \"bsd\"]");
+
+        test_parse_query_to_ast_helper("(<=42)", "{\"*\" TO \"42\"]");
+        test_parse_query_to_ast_helper("(<=42 )", "{\"*\" TO \"42\"]");
+        test_parse_query_to_ast_helper("(age:>5)", "\"age\":{\"5\" TO \"*\"}");
+        test_parse_query_to_ast_helper(
+            "(title:bar AND age:>12)",
+            "(+\"title\":bar +\"age\":{\"12\" TO \"*\"})",
+        );
     }
 
     #[test]
@@ -1699,6 +1715,10 @@ mod test {
         test_parse_query_to_ast_helper("foo:(A OR B)", "(?\"foo\":A ?\"foo\":B)");
         test_parse_query_to_ast_helper("foo:(A* OR B*)", "(?\"foo\":A* ?\"foo\":B*)");
         test_parse_query_to_ast_helper("foo:(*A OR *B)", "(?\"foo\":*A ?\"foo\":*B)");
+
+        // Regexes between parentheses
+        test_parse_query_to_ast_helper("foo:(/A.*/)", "\"foo\":/A.*/");
+        test_parse_query_to_ast_helper("foo:(/A.*/ OR /B.*/)", "(?\"foo\":/A.*/ ?\"foo\":/B.*/)");
     }
 
     #[test]

query-grammar/src/user_input_ast.rs

@@ -66,6 +66,7 @@ impl UserInputLeaf {
             }
             UserInputLeaf::Range { field, .. } if field.is_none() => *field = Some(default_field),
             UserInputLeaf::Set { field, .. } if field.is_none() => *field = Some(default_field),
+            UserInputLeaf::Regex { field, .. } if field.is_none() => *field = Some(default_field),
             _ => (), // field was already set, do nothing
         }
     }

src/aggregation/accessor_helpers.rs

@@ -57,7 +57,7 @@ pub(crate) fn get_numeric_or_date_column_types() -> &'static [ColumnType] {
 
 /// Get fast field reader or empty as default.
 pub(crate) fn get_ff_reader(
-    reader: &SegmentReader,
+    reader: &dyn SegmentReader,
     field_name: &str,
     allowed_column_types: Option<&[ColumnType]>,
 ) -> crate::Result<(columnar::Column<u64>, ColumnType)> {
@@ -74,7 +74,7 @@ pub(crate) fn get_ff_reader(
 }
 
 pub(crate) fn get_dynamic_columns(
-    reader: &SegmentReader,
+    reader: &dyn SegmentReader,
     field_name: &str,
 ) -> crate::Result<Vec<columnar::DynamicColumn>> {
     let ff_fields = reader.fast_fields().dynamic_column_handles(field_name)?;
@@ -90,7 +90,7 @@ pub(crate) fn get_dynamic_columns(
 ///
 /// Is guaranteed to return at least one column.
 pub(crate) fn get_all_ff_reader_or_empty(
-    reader: &SegmentReader,
+    reader: &dyn SegmentReader,
     field_name: &str,
     allowed_column_types: Option<&[ColumnType]>,
     fallback_type: ColumnType,

src/aggregation/agg_data.rs

@@ -469,7 +469,7 @@ impl AggKind {
 /// Build AggregationsData by walking the request tree.
 pub(crate) fn build_aggregations_data_from_req(
     aggs: &Aggregations,
-    reader: &SegmentReader,
+    reader: &dyn SegmentReader,
     segment_ordinal: SegmentOrdinal,
     context: AggContextParams,
 ) -> crate::Result<AggregationsSegmentCtx> {
@@ -489,7 +489,7 @@ pub(crate) fn build_aggregations_data_from_req(
 fn build_nodes(
     agg_name: &str,
     req: &Aggregation,
-    reader: &SegmentReader,
+    reader: &dyn SegmentReader,
     segment_ordinal: SegmentOrdinal,
     data: &mut AggregationsSegmentCtx,
     is_top_level: bool,
@@ -728,7 +728,7 @@ fn build_nodes(
             let idx_in_req_data = data.push_filter_req_data(FilterAggReqData {
                 name: agg_name.to_string(),
                 req: filter_req.clone(),
-                segment_reader: reader.clone(),
+                segment_reader: reader.clone_arc(),
                 evaluator,
                 matching_docs_buffer,
                 is_top_level,
@@ -745,7 +745,7 @@ fn build_nodes(
 
 fn build_children(
     aggs: &Aggregations,
-    reader: &SegmentReader,
+    reader: &dyn SegmentReader,
     segment_ordinal: SegmentOrdinal,
     data: &mut AggregationsSegmentCtx,
 ) -> crate::Result<Vec<AggRefNode>> {
@@ -764,7 +764,7 @@ fn build_children(
 }
 
 fn get_term_agg_accessors(
-    reader: &SegmentReader,
+    reader: &dyn SegmentReader,
     field_name: &str,
     missing: &Option<Key>,
 ) -> crate::Result<Vec<(Column<u64>, ColumnType)>> {
@@ -817,7 +817,7 @@ fn build_terms_or_cardinality_nodes(
     agg_name: &str,
     field_name: &str,
     missing: &Option<Key>,
-    reader: &SegmentReader,
+    reader: &dyn SegmentReader,
     segment_ordinal: SegmentOrdinal,
     data: &mut AggregationsSegmentCtx,
     sub_aggs: &Aggregations,

src/aggregation/bucket/filter.rs

@@ -1,4 +1,5 @@
 use std::fmt::Debug;
+use std::sync::Arc;
 
 use common::BitSet;
 use serde::{Deserialize, Deserializer, Serialize, Serializer};
@@ -402,7 +403,7 @@ pub struct FilterAggReqData {
     /// The filter aggregation
     pub req: FilterAggregation,
     /// The segment reader
-    pub segment_reader: SegmentReader,
+    pub segment_reader: Arc<dyn SegmentReader>,
     /// Document evaluator for the filter query (precomputed BitSet)
     /// This is built once when the request data is created
     pub evaluator: DocumentQueryEvaluator,
@@ -416,7 +417,7 @@ impl FilterAggReqData {
     pub(crate) fn get_memory_consumption(&self) -> usize {
         // Estimate: name + segment reader reference + bitset + buffer capacity
         self.name.len()
-        + std::mem::size_of::<SegmentReader>()
+        + std::mem::size_of::<Arc<dyn SegmentReader>>()
         + self.evaluator.bitset.len() / 8 // BitSet memory (bits to bytes)
         + self.matching_docs_buffer.capacity() * std::mem::size_of::<DocId>()
         + std::mem::size_of::<bool>()
@@ -438,7 +439,7 @@ impl DocumentQueryEvaluator {
     pub(crate) fn new(
         query: Box<dyn Query>,
         schema: Schema,
-        segment_reader: &SegmentReader,
+        segment_reader: &dyn SegmentReader,
     ) -> crate::Result<Self> {
         let max_doc = segment_reader.max_doc();
 

src/aggregation/collector.rs

@@ -66,7 +66,7 @@ impl Collector for DistributedAggregationCollector {
     fn for_segment(
         &self,
         segment_local_id: crate::SegmentOrdinal,
-        reader: &crate::SegmentReader,
+        reader: &dyn SegmentReader,
     ) -> crate::Result<Self::Child> {
         AggregationSegmentCollector::from_agg_req_and_reader(
             &self.agg,
@@ -96,7 +96,7 @@ impl Collector for AggregationCollector {
     fn for_segment(
         &self,
         segment_local_id: crate::SegmentOrdinal,
-        reader: &crate::SegmentReader,
+        reader: &dyn SegmentReader,
     ) -> crate::Result<Self::Child> {
         AggregationSegmentCollector::from_agg_req_and_reader(
             &self.agg,
@@ -145,7 +145,7 @@ impl AggregationSegmentCollector {
     /// reader. Also includes validation, e.g. checking field types and existence.
     pub fn from_agg_req_and_reader(
         agg: &Aggregations,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
         segment_ordinal: SegmentOrdinal,
         context: &AggContextParams,
     ) -> crate::Result<Self> {

src/aggregation/intermediate_agg_result.rs

@@ -90,6 +90,19 @@ impl From<IntermediateKey> for Key {
 
 impl Eq for IntermediateKey {}
 
+impl std::fmt::Display for IntermediateKey {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        match self {
+            IntermediateKey::Str(val) => f.write_str(val),
+            IntermediateKey::F64(val) => f.write_str(&val.to_string()),
+            IntermediateKey::U64(val) => f.write_str(&val.to_string()),
+            IntermediateKey::I64(val) => f.write_str(&val.to_string()),
+            IntermediateKey::Bool(val) => f.write_str(&val.to_string()),
+            IntermediateKey::IpAddr(val) => f.write_str(&val.to_string()),
+        }
+    }
+}
+
 impl std::hash::Hash for IntermediateKey {
     fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
         core::mem::discriminant(self).hash(state);
@@ -105,6 +118,21 @@ impl std::hash::Hash for IntermediateKey {
 }
 
 impl IntermediateAggregationResults {
+    /// Returns a reference to the intermediate aggregation result for the given key.
+    pub fn get(&self, key: &str) -> Option<&IntermediateAggregationResult> {
+        self.aggs_res.get(key)
+    }
+
+    /// Removes and returns the intermediate aggregation result for the given key.
+    pub fn remove(&mut self, key: &str) -> Option<IntermediateAggregationResult> {
+        self.aggs_res.remove(key)
+    }
+
+    /// Returns an iterator over the keys in the intermediate aggregation results.
+    pub fn keys(&self) -> impl Iterator<Item = &String> {
+        self.aggs_res.keys()
+    }
+
     /// Add a result
     pub fn push(&mut self, key: String, value: IntermediateAggregationResult) -> crate::Result<()> {
         let entry = self.aggs_res.entry(key);
@@ -639,6 +667,21 @@ pub struct IntermediateTermBucketResult {
 }
 
 impl IntermediateTermBucketResult {
+    /// Returns a reference to the map of bucket entries keyed by [`IntermediateKey`].
+    pub fn entries(&self) -> &FxHashMap<IntermediateKey, IntermediateTermBucketEntry> {
+        &self.entries
+    }
+
+    /// Returns the count of documents not included in the returned buckets.
+    pub fn sum_other_doc_count(&self) -> u64 {
+        self.sum_other_doc_count
+    }
+
+    /// Returns the upper bound of the error on document counts in the returned buckets.
+    pub fn doc_count_error_upper_bound(&self) -> u64 {
+        self.doc_count_error_upper_bound
+    }
+
     pub(crate) fn into_final_result(
         self,
         req: &TermsAggregation,
@@ -820,7 +863,7 @@ impl IntermediateRangeBucketEntry {
         };
 
         // If we have a date type on the histogram buckets, we add the `key_as_string` field as
-        // rfc339
+        // rfc3339
         if column_type == Some(ColumnType::DateTime) {
             if let Some(val) = range_bucket_entry.to {
                 let key_as_string = format_date(val as i64)?;

src/aggregation/metric/average.rs

@@ -55,6 +55,12 @@ impl IntermediateAverage {
     pub(crate) fn from_stats(stats: IntermediateStats) -> Self {
         Self { stats }
     }
+
+    /// Returns a reference to the underlying [`IntermediateStats`].
+    pub fn stats(&self) -> &IntermediateStats {
+        &self.stats
+    }
+
     /// Merges the other intermediate result into self.
     pub fn merge_fruits(&mut self, other: IntermediateAverage) {
         self.stats.merge_fruits(other.stats);

src/aggregation/metric/mod.rs

@@ -107,8 +107,11 @@ pub enum PercentileValues {
 #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
 /// The entry when requesting percentiles with keyed: false
 pub struct PercentileValuesVecEntry {
-    key: f64,
-    value: f64,
+    /// Percentile
+    pub key: f64,
+
+    /// Value at the percentile
+    pub value: f64,
 }
 
 /// Single-metric aggregations use this common result structure.

src/aggregation/metric/stats.rs

@@ -110,6 +110,16 @@ impl Default for IntermediateStats {
 }
 
 impl IntermediateStats {
+    /// Returns the number of values collected.
+    pub fn count(&self) -> u64 {
+        self.count
+    }
+
+    /// Returns the sum of all values collected.
+    pub fn sum(&self) -> f64 {
+        self.sum
+    }
+
     /// Merges the other stats intermediate result into self.
     pub fn merge_fruits(&mut self, other: IntermediateStats) {
         self.count += other.count;

src/codec/mod.rs

@@ -4,18 +4,18 @@ pub mod postings;
 /// Standard tantivy codec. This is the codec you use by default.
 pub mod standard;
 
-use std::io;
+use std::sync::Arc;
 
 pub use standard::StandardCodec;
 
 use crate::codec::postings::PostingsCodec;
-use crate::fieldnorm::FieldNormReader;
-use crate::postings::{Postings, TermInfo};
+use crate::directory::Directory;
+use crate::fastfield::AliveBitSet;
 use crate::query::score_combiner::DoNothingCombiner;
 use crate::query::term_query::TermScorer;
-use crate::query::{box_scorer, Bm25Weight, BufferedUnionScorer, Scorer, SumCombiner};
-use crate::schema::IndexRecordOption;
-use crate::{DocId, InvertedIndexReader, Score};
+use crate::query::{box_scorer, BufferedUnionScorer, Scorer, SumCombiner};
+use crate::schema::Schema;
+use crate::{DocId, Score, SegmentMeta, SegmentReader, TantivySegmentReader};
 
 /// Codecs describes how data is layed out on disk.
 ///
@@ -36,58 +36,46 @@ pub trait Codec: Clone + std::fmt::Debug + Send + Sync + 'static {
 
     /// Returns the postings codec.
     fn postings_codec(&self) -> &Self::PostingsCodec;
+
+    /// Loads postings using the codec's concrete postings type.
+    fn load_postings_typed(
+        &self,
+        reader: &dyn crate::index::InvertedIndexReader,
+        term_info: &crate::postings::TermInfo,
+        option: crate::schema::IndexRecordOption,
+    ) -> std::io::Result<<Self::PostingsCodec as crate::codec::postings::PostingsCodec>::Postings>
+    {
+        let postings_data = reader.read_raw_postings_data(term_info, option)?;
+        self.postings_codec()
+            .load_postings(term_info.doc_freq, postings_data)
+    }
+
+    /// Opens a segment reader using this codec.
+    ///
+    /// Override this if your codec uses a custom segment reader implementation.
+    fn open_segment_reader(
+        &self,
+        directory: &dyn Directory,
+        segment_meta: &SegmentMeta,
+        schema: Schema,
+        custom_bitset: Option<AliveBitSet>,
+    ) -> crate::Result<Arc<dyn SegmentReader>> {
+        let codec: Arc<dyn ObjectSafeCodec> = Arc::new(self.clone());
+        let reader = TantivySegmentReader::open_with_custom_alive_set_from_directory(
+            directory,
+            segment_meta,
+            schema,
+            codec,
+            custom_bitset,
+        )?;
+        Ok(Arc::new(reader))
+    }
 }
 
 /// Object-safe codec is a Codec that can be used in a trait object.
 ///
 /// The point of it is to offer a way to use a codec without a proliferation of generics.
 pub trait ObjectSafeCodec: 'static + Send + Sync {
-    /// Loads a type-erased Postings object for the given term.
-    ///
-    /// If the schema used to build the index did not provide enough
-    /// information to match the requested `option`, a Postings is still
-    /// returned in a best-effort manner.
-    fn load_postings_type_erased(
-        &self,
-        term_info: &TermInfo,
-        option: IndexRecordOption,
-        inverted_index_reader: &InvertedIndexReader,
-    ) -> io::Result<Box<dyn Postings>>;
-
-    /// Loads a type-erased TermScorer object for the given term.
-    ///
-    /// If the schema used to build the index did not provide enough
-    /// information to match the requested `option`, a TermScorer is still
-    /// returned in a best-effort manner.
-    ///
-    /// The point of this contraption is that the return TermScorer is backed,
-    /// not by Box<dyn Postings> but by the codec's concrete Postings type.
-    fn load_term_scorer_type_erased(
-        &self,
-        term_info: &TermInfo,
-        option: IndexRecordOption,
-        inverted_index_reader: &InvertedIndexReader,
-        fieldnorm_reader: FieldNormReader,
-        similarity_weight: Bm25Weight,
-    ) -> io::Result<Box<dyn Scorer>>;
-
-    /// Loads a type-erased PhraseScorer object for the given term.
-    ///
-    /// If the schema used to build the index did not provide enough
-    /// information to match the requested `option`, a TermScorer is still
-    /// returned in a best-effort manner.
-    ///
-    /// The point of this contraption is that the return PhraseScorer is backed,
-    /// not by Box<dyn Postings> but by the codec's concrete Postings type.
-    fn new_phrase_scorer_type_erased(
-        &self,
-        term_infos: &[(usize, TermInfo)],
-        similarity_weight: Option<Bm25Weight>,
-        fieldnorm_reader: FieldNormReader,
-        slop: u32,
-        inverted_index_reader: &InvertedIndexReader,
-    ) -> io::Result<Box<dyn Scorer>>;
-
     /// Performs a for_each_pruning operation on the given scorer.
     ///
     /// The function will go through matching documents and call the callback
@@ -116,53 +104,6 @@ pub trait ObjectSafeCodec: 'static + Send + Sync {
 }
 
 impl<TCodec: Codec> ObjectSafeCodec for TCodec {
-    fn load_postings_type_erased(
-        &self,
-        term_info: &TermInfo,
-        option: IndexRecordOption,
-        inverted_index_reader: &InvertedIndexReader,
-    ) -> io::Result<Box<dyn Postings>> {
-        let postings = inverted_index_reader
-            .read_postings_from_terminfo_specialized(term_info, option, self)?;
-        Ok(Box::new(postings))
-    }
-
-    fn load_term_scorer_type_erased(
-        &self,
-        term_info: &TermInfo,
-        option: IndexRecordOption,
-        inverted_index_reader: &InvertedIndexReader,
-        fieldnorm_reader: FieldNormReader,
-        similarity_weight: Bm25Weight,
-    ) -> io::Result<Box<dyn Scorer>> {
-        let scorer = inverted_index_reader.new_term_scorer_specialized(
-            term_info,
-            option,
-            fieldnorm_reader,
-            similarity_weight,
-            self,
-        )?;
-        Ok(box_scorer(scorer))
-    }
-
-    fn new_phrase_scorer_type_erased(
-        &self,
-        term_infos: &[(usize, TermInfo)],
-        similarity_weight: Option<Bm25Weight>,
-        fieldnorm_reader: FieldNormReader,
-        slop: u32,
-        inverted_index_reader: &InvertedIndexReader,
-    ) -> io::Result<Box<dyn Scorer>> {
-        let scorer = inverted_index_reader.new_phrase_scorer_type_specialized(
-            term_infos,
-            similarity_weight,
-            fieldnorm_reader,
-            slop,
-            self,
-        )?;
-        Ok(box_scorer(scorer))
-    }
-
     fn build_union_scorer_with_sum_combiner(
         &self,
         scorers: Vec<Box<dyn Scorer>>,

src/codec/postings/mod.rs

@@ -1,7 +1,7 @@
-use std::io;
-
 /// Block-max WAND algorithm.
 pub mod block_wand;
+use std::io;
+
 use common::OwnedBytes;
 
 use crate::fieldnorm::FieldNormReader;
@@ -10,38 +10,16 @@ use crate::query::{Bm25Weight, Scorer};
 use crate::schema::IndexRecordOption;
 use crate::{DocId, Score};
 
-/// Postings codec.
+/// Postings codec (read path).
 pub trait PostingsCodec: Send + Sync + 'static {
-    /// Serializer type for the postings codec.
-    type PostingsSerializer: PostingsSerializer;
     /// Postings type for the postings codec.
     type Postings: Postings + Clone;
-    /// Creates a new postings serializer.
-    fn new_serializer(
-        &self,
-        avg_fieldnorm: Score,
-        mode: IndexRecordOption,
-        fieldnorm_reader: Option<FieldNormReader>,
-    ) -> Self::PostingsSerializer;
 
-    /// Loads postings
-    ///
-    /// Record option is the option that was passed at indexing time.
-    /// Requested option is the option that is requested.
-    ///
-    /// For instance, we may have term_freq in the posting list
-    /// but we can skip decompressing as we read the posting list.
-    ///
-    /// If record option does not support the requested option,
-    /// this method does NOT return an error and will in fact restrict
-    /// requested_option to what is available.
+    /// Load postings from raw bytes and metadata.
     fn load_postings(
         &self,
         doc_freq: u32,
-        postings_data: OwnedBytes,
-        record_option: IndexRecordOption,
-        requested_option: IndexRecordOption,
-        positions_data: Option<OwnedBytes>,
+        postings_data: RawPostingsData,
     ) -> io::Result<Self::Postings>;
 
     /// If your codec supports different ways to accelerate `for_each_pruning` that's
@@ -63,43 +41,17 @@ pub trait PostingsCodec: Send + Sync + 'static {
     }
 }
 
-/// A postings serializer is a listener that is in charge of serializing postings
-///
-/// IO is done only once per postings, once all of the data has been received.
-/// A serializer will therefore contain internal buffers.
-///
-/// A serializer is created once and recycled for all postings.
-///
-/// Clients should use PostingsSerializer as follows.
-/// ```rust,no_run
-/// // First postings list
-/// serializer.new_term(2, true);
-/// serializer.write_doc(2, 1);
-/// serializer.write_doc(6, 2);
-/// serializer.close_term(3);
-/// serializer.clear();
-/// // Second postings list
-/// serializer.new_term(1, true);
-/// serializer.write_doc(3, 1);
-/// serializer.close_term(3);
-/// ```
-pub trait PostingsSerializer {
-    /// The term_doc_freq here is the number of documents
-    /// in the postings lists.
-    ///
-    /// It can be used to compute the idf that will be used for the
-    /// blockmax parameters.
-    ///
-    /// If not available (e.g. if we do not collect `term_frequencies`
-    /// blockwand is disabled), the term_doc_freq passed will be set 0.
-    fn new_term(&mut self, term_doc_freq: u32, record_term_freq: bool);
-
-    /// Records a new document id for the current term.
-    /// The serializer may ignore it.
-    fn write_doc(&mut self, doc_id: DocId, term_freq: u32);
-
-    /// Closes the current term and writes the postings list associated.
-    fn close_term(&mut self, doc_freq: u32, wrt: &mut impl io::Write) -> io::Result<()>;
+/// Raw postings bytes and metadata read from storage.
+#[derive(Debug, Clone)]
+pub struct RawPostingsData {
+    /// Raw postings bytes for the term.
+    pub postings_data: OwnedBytes,
+    /// Raw positions bytes for the term, if positions are available.
+    pub positions_data: Option<OwnedBytes>,
+    /// Record option of the indexed field.
+    pub record_option: IndexRecordOption,
+    /// Effective record option after downgrading to the indexed field capability.
+    pub effective_option: IndexRecordOption,
 }
 
 /// A light complement interface to Postings to allow block-max wand acceleration.

src/codec/standard/postings/block.rs

@@ -1,50 +0,0 @@
-use crate::postings::compression::COMPRESSION_BLOCK_SIZE;
-use crate::DocId;
-
-pub struct Block {
-    doc_ids: [DocId; COMPRESSION_BLOCK_SIZE],
-    term_freqs: [u32; COMPRESSION_BLOCK_SIZE],
-    len: usize,
-}
-
-impl Block {
-    pub fn new() -> Self {
-        Block {
-            doc_ids: [0u32; COMPRESSION_BLOCK_SIZE],
-            term_freqs: [0u32; COMPRESSION_BLOCK_SIZE],
-            len: 0,
-        }
-    }
-
-    pub fn doc_ids(&self) -> &[DocId] {
-        &self.doc_ids[..self.len]
-    }
-
-    pub fn term_freqs(&self) -> &[u32] {
-        &self.term_freqs[..self.len]
-    }
-
-    pub fn clear(&mut self) {
-        self.len = 0;
-    }
-
-    pub fn append_doc(&mut self, doc: DocId, term_freq: u32) {
-        let len = self.len;
-        self.doc_ids[len] = doc;
-        self.term_freqs[len] = term_freq;
-        self.len = len + 1;
-    }
-
-    pub fn is_full(&self) -> bool {
-        self.len == COMPRESSION_BLOCK_SIZE
-    }
-
-    pub fn is_empty(&self) -> bool {
-        self.len == 0
-    }
-
-    pub fn last_doc(&self) -> DocId {
-        assert_eq!(self.len, COMPRESSION_BLOCK_SIZE);
-        self.doc_ids[COMPRESSION_BLOCK_SIZE - 1]
-    }
-}

src/codec/standard/postings/block_segment_postings.rs

@@ -2,10 +2,10 @@ use std::io;
 
 use common::{OwnedBytes, VInt};
 
-use crate::codec::standard::postings::skip::{BlockInfo, SkipReader};
 use crate::codec::standard::postings::FreqReadingOption;
 use crate::fieldnorm::FieldNormReader;
 use crate::postings::compression::{BlockDecoder, VIntDecoder as _, COMPRESSION_BLOCK_SIZE};
+use crate::postings::skip::{BlockInfo, SkipReader};
 use crate::query::Bm25Weight;
 use crate::schema::IndexRecordOption;
 use crate::{DocId, Score, TERMINATED};
@@ -337,18 +337,17 @@ mod tests {
     use common::OwnedBytes;
 
     use super::BlockSegmentPostings;
-    use crate::codec::postings::PostingsSerializer;
     use crate::codec::standard::postings::segment_postings::SegmentPostings;
-    use crate::codec::standard::postings::StandardPostingsSerializer;
     use crate::docset::{DocSet, TERMINATED};
     use crate::postings::compression::COMPRESSION_BLOCK_SIZE;
+    use crate::postings::serializer::PostingsSerializer;
     use crate::schema::IndexRecordOption;
 
     #[cfg(test)]
     fn build_block_postings(docs: &[u32]) -> BlockSegmentPostings {
         let doc_freq = docs.len() as u32;
         let mut postings_serializer =
-            StandardPostingsSerializer::new(1.0f32, IndexRecordOption::Basic, None);
+            PostingsSerializer::new(1.0f32, IndexRecordOption::Basic, None);
         postings_serializer.new_term(docs.len() as u32, false);
         for doc in docs {
             postings_serializer.write_doc(*doc, 1u32);

src/codec/standard/postings/mod.rs

@@ -1,24 +1,20 @@
 use std::io;
 
+use common::BitSet;
+
 use crate::codec::postings::block_wand::{block_wand, block_wand_single_scorer};
-use crate::codec::postings::PostingsCodec;
+use crate::codec::postings::{PostingsCodec, RawPostingsData};
 use crate::codec::standard::postings::block_segment_postings::BlockSegmentPostings;
 pub use crate::codec::standard::postings::segment_postings::SegmentPostings;
-use crate::fieldnorm::FieldNormReader;
 use crate::positions::PositionReader;
 use crate::query::term_query::TermScorer;
 use crate::query::{BufferedUnionScorer, Scorer, SumCombiner};
-use crate::schema::IndexRecordOption;
 use crate::{DocSet as _, Score, TERMINATED};
 
-mod block;
 mod block_segment_postings;
 mod segment_postings;
-mod skip;
-mod standard_postings_serializer;
 
 pub use segment_postings::SegmentPostings as StandardPostings;
-pub use standard_postings_serializer::StandardPostingsSerializer;
 
 /// The default postings codec for tantivy.
 pub struct StandardPostingsCodec;
@@ -32,35 +28,14 @@ pub(crate) enum FreqReadingOption {
 }
 
 impl PostingsCodec for StandardPostingsCodec {
-    type PostingsSerializer = StandardPostingsSerializer;
     type Postings = SegmentPostings;
 
-    fn new_serializer(
-        &self,
-        avg_fieldnorm: Score,
-        mode: IndexRecordOption,
-        fieldnorm_reader: Option<FieldNormReader>,
-    ) -> Self::PostingsSerializer {
-        StandardPostingsSerializer::new(avg_fieldnorm, mode, fieldnorm_reader)
-    }
-
     fn load_postings(
         &self,
         doc_freq: u32,
-        postings_data: common::OwnedBytes,
-        record_option: IndexRecordOption,
-        requested_option: IndexRecordOption,
-        positions_data_opt: Option<common::OwnedBytes>,
+        postings_data: RawPostingsData,
     ) -> io::Result<Self::Postings> {
-        // Rationalize record_option/requested_option.
-        let requested_option = requested_option.downgrade(record_option);
-        let block_segment_postings =
-            BlockSegmentPostings::open(doc_freq, postings_data, record_option, requested_option)?;
-        let position_reader = positions_data_opt.map(PositionReader::open).transpose()?;
-        Ok(SegmentPostings::from_block_postings(
-            block_segment_postings,
-            position_reader,
-        ))
+        load_postings_from_raw_data(doc_freq, postings_data)
     }
 
     fn try_accelerated_for_each_pruning(
@@ -76,14 +51,7 @@ impl PostingsCodec for StandardPostingsCodec {
             Err(scorer) => scorer,
         };
         let mut union_scorer =
-            scorer.downcast::<BufferedUnionScorer<Box<dyn Scorer>, SumCombiner>>()?;
-        if !union_scorer
-            .scorers()
-            .iter()
-            .all(|scorer| scorer.is::<TermScorer<Self::Postings>>())
-        {
-            return Err(union_scorer);
-        }
+            scorer.downcast::<BufferedUnionScorer<TermScorer<Self::Postings>, SumCombiner>>()?;
         let doc = union_scorer.doc();
         if doc == TERMINATED {
             return Ok(());
@@ -92,31 +60,69 @@ impl PostingsCodec for StandardPostingsCodec {
         if score > threshold {
             threshold = callback(doc, score);
         }
-        let boxed_scorers: Vec<Box<dyn Scorer>> = union_scorer.into_scorers();
-        let scorers: Vec<TermScorer<Self::Postings>> = boxed_scorers
-            .into_iter()
-            .map(|scorer| {
-                *scorer.downcast::<TermScorer<Self::Postings>>().ok().expect(
-                    "Downcast failed despite the fact we already checked the type was correct",
-                )
-            })
-            .collect();
+        let scorers: Vec<TermScorer<Self::Postings>> = union_scorer.into_scorers();
         block_wand(scorers, threshold, callback);
         Ok(())
     }
 }
+pub(crate) fn load_postings_from_raw_data(
+    doc_freq: u32,
+    postings_data: RawPostingsData,
+) -> io::Result<SegmentPostings> {
+    let RawPostingsData {
+        postings_data,
+        positions_data: positions_data_opt,
+        record_option,
+        effective_option,
+    } = postings_data;
+    let requested_option = effective_option;
+    let block_segment_postings =
+        BlockSegmentPostings::open(doc_freq, postings_data, record_option, requested_option)?;
+    let position_reader = positions_data_opt.map(PositionReader::open).transpose()?;
+    Ok(SegmentPostings::from_block_postings(
+        block_segment_postings,
+        position_reader,
+    ))
+}
+
+pub(crate) fn fill_bitset_from_raw_data(
+    doc_freq: u32,
+    postings_data: RawPostingsData,
+    doc_bitset: &mut BitSet,
+) -> io::Result<()> {
+    let RawPostingsData {
+        postings_data,
+        record_option,
+        effective_option,
+        ..
+    } = postings_data;
+    let mut block_postings =
+        BlockSegmentPostings::open(doc_freq, postings_data, record_option, effective_option)?;
+    loop {
+        let docs = block_postings.docs();
+        if docs.is_empty() {
+            break;
+        }
+        for &doc in docs {
+            doc_bitset.insert(doc);
+        }
+        block_postings.advance();
+    }
+    Ok(())
+}
 
 #[cfg(test)]
 mod tests {
     use common::OwnedBytes;
 
     use super::*;
-    use crate::codec::postings::PostingsSerializer as _;
+    use crate::postings::serializer::PostingsSerializer;
     use crate::postings::Postings as _;
+    use crate::schema::IndexRecordOption;
 
     fn test_segment_postings_tf_aux(num_docs: u32, include_term_freq: bool) -> SegmentPostings {
         let mut postings_serializer =
-            StandardPostingsCodec.new_serializer(1.0f32, IndexRecordOption::WithFreqs, None);
+            PostingsSerializer::new(1.0f32, IndexRecordOption::WithFreqs, None);
         let mut buffer = Vec::new();
         postings_serializer.new_term(num_docs, include_term_freq);
         for i in 0..num_docs {
@@ -125,15 +131,16 @@ mod tests {
         postings_serializer
             .close_term(num_docs, &mut buffer)
             .unwrap();
-        StandardPostingsCodec
-            .load_postings(
-                num_docs,
-                OwnedBytes::new(buffer),
-                IndexRecordOption::WithFreqs,
-                IndexRecordOption::WithFreqs,
-                None,
-            )
-            .unwrap()
+        load_postings_from_raw_data(
+            num_docs,
+            RawPostingsData {
+                postings_data: OwnedBytes::new(buffer),
+                positions_data: None,
+                record_option: IndexRecordOption::WithFreqs,
+                effective_option: IndexRecordOption::WithFreqs,
+            },
+        )
+        .unwrap()
     }
 
     #[test]

src/codec/standard/postings/segment_postings.rs

@@ -47,14 +47,10 @@ impl SegmentPostings {
         use crate::schema::IndexRecordOption;
         let mut buffer = Vec::new();
         {
-            use crate::codec::postings::PostingsSerializer;
+            use crate::postings::serializer::PostingsSerializer;
 
             let mut postings_serializer =
-                crate::codec::standard::postings::StandardPostingsSerializer::new(
-                    0.0,
-                    IndexRecordOption::Basic,
-                    None,
-                );
+                PostingsSerializer::new(0.0, IndexRecordOption::Basic, None);
             postings_serializer.new_term(docs.len() as u32, false);
             for &doc in docs {
                 postings_serializer.write_doc(doc, 1u32);
@@ -81,9 +77,8 @@ impl SegmentPostings {
     ) -> SegmentPostings {
         use common::OwnedBytes;
 
-        use crate::codec::postings::PostingsSerializer as _;
-        use crate::codec::standard::postings::StandardPostingsSerializer;
         use crate::fieldnorm::FieldNormReader;
+        use crate::postings::serializer::PostingsSerializer;
         use crate::schema::IndexRecordOption;
         use crate::Score;
         let mut buffer: Vec<u8> = Vec::new();
@@ -100,7 +95,7 @@ impl SegmentPostings {
                 total_num_tokens as Score / fieldnorms.len() as Score
             })
             .unwrap_or(0.0);
-        let mut postings_serializer = StandardPostingsSerializer::new(
+        let mut postings_serializer = PostingsSerializer::new(
             average_field_norm,
             IndexRecordOption::WithFreqs,
             fieldnorm_reader,
@@ -269,6 +264,7 @@ impl Postings for SegmentPostings {
 }
 
 impl PostingsWithBlockMax for SegmentPostings {
+    #[inline]
     fn seek_block_max(
         &mut self,
         target_doc: crate::DocId,
@@ -280,6 +276,7 @@ impl PostingsWithBlockMax for SegmentPostings {
             .block_max_score(fieldnorm_reader, similarity_weight)
     }
 
+    #[inline]
     fn last_doc_in_block(&self) -> crate::DocId {
         self.block_cursor.skip_reader().last_doc_in_block()
     }

src/codec/standard/postings/standard_postings_serializer.rs

@@ -1,184 +0,0 @@
-use std::cmp::Ordering;
-use std::io::{self, Write as _};
-
-use common::{BinarySerializable as _, VInt};
-
-use crate::codec::postings::PostingsSerializer;
-use crate::codec::standard::postings::block::Block;
-use crate::codec::standard::postings::skip::SkipSerializer;
-use crate::fieldnorm::FieldNormReader;
-use crate::postings::compression::{BlockEncoder, VIntEncoder as _, COMPRESSION_BLOCK_SIZE};
-use crate::query::Bm25Weight;
-use crate::schema::IndexRecordOption;
-use crate::{DocId, Score};
-
-/// Serializer object for tantivy's default postings format.
-pub struct StandardPostingsSerializer {
-    last_doc_id_encoded: u32,
-
-    block_encoder: BlockEncoder,
-    block: Box<Block>,
-
-    postings_write: Vec<u8>,
-    skip_write: SkipSerializer,
-
-    mode: IndexRecordOption,
-    fieldnorm_reader: Option<FieldNormReader>,
-
-    bm25_weight: Option<Bm25Weight>,
-    avg_fieldnorm: Score, /* Average number of term in the field for that segment.
-                           * this value is used to compute the block wand information. */
-    term_has_freq: bool,
-}
-
-impl StandardPostingsSerializer {
-    pub(crate) fn new(
-        avg_fieldnorm: Score,
-        mode: IndexRecordOption,
-        fieldnorm_reader: Option<FieldNormReader>,
-    ) -> StandardPostingsSerializer {
-        Self {
-            last_doc_id_encoded: 0,
-            block_encoder: BlockEncoder::new(),
-            block: Box::new(Block::new()),
-            postings_write: Vec::new(),
-            skip_write: SkipSerializer::new(),
-            mode,
-            fieldnorm_reader,
-            bm25_weight: None,
-            avg_fieldnorm,
-            term_has_freq: false,
-        }
-    }
-}
-
-impl PostingsSerializer for StandardPostingsSerializer {
-    fn new_term(&mut self, term_doc_freq: u32, record_term_freq: bool) {
-        self.clear();
-
-        self.term_has_freq = self.mode.has_freq() && record_term_freq;
-        if !self.term_has_freq {
-            return;
-        }
-
-        let num_docs_in_segment: u64 =
-            if let Some(fieldnorm_reader) = self.fieldnorm_reader.as_ref() {
-                fieldnorm_reader.num_docs() as u64
-            } else {
-                return;
-            };
-
-        if num_docs_in_segment == 0 {
-            return;
-        }
-
-        self.bm25_weight = Some(Bm25Weight::for_one_term_without_explain(
-            term_doc_freq as u64,
-            num_docs_in_segment,
-            self.avg_fieldnorm,
-        ));
-    }
-
-    fn write_doc(&mut self, doc_id: DocId, term_freq: u32) {
-        self.block.append_doc(doc_id, term_freq);
-        if self.block.is_full() {
-            self.write_block();
-        }
-    }
-
-    fn close_term(&mut self, doc_freq: u32, output_write: &mut impl io::Write) -> io::Result<()> {
-        if !self.block.is_empty() {
-            // we have doc ids waiting to be written
-            // this happens when the number of doc ids is
-            // not a perfect multiple of our block size.
-            //
-            // In that case, the remaining part is encoded
-            // using variable int encoding.
-            {
-                let block_encoded = self
-                    .block_encoder
-                    .compress_vint_sorted(self.block.doc_ids(), self.last_doc_id_encoded);
-                self.postings_write.write_all(block_encoded)?;
-            }
-            // ... Idem for term frequencies
-            if self.term_has_freq {
-                let block_encoded = self
-                    .block_encoder
-                    .compress_vint_unsorted(self.block.term_freqs());
-                self.postings_write.write_all(block_encoded)?;
-            }
-            self.block.clear();
-        }
-        if doc_freq >= COMPRESSION_BLOCK_SIZE as u32 {
-            let skip_data = self.skip_write.data();
-            VInt(skip_data.len() as u64).serialize(output_write)?;
-            output_write.write_all(skip_data)?;
-        }
-        output_write.write_all(&self.postings_write[..])?;
-        self.skip_write.clear();
-        self.postings_write.clear();
-        self.bm25_weight = None;
-        Ok(())
-    }
-}
-
-impl StandardPostingsSerializer {
-    fn clear(&mut self) {
-        self.bm25_weight = None;
-        self.block.clear();
-        self.last_doc_id_encoded = 0;
-    }
-
-    fn write_block(&mut self) {
-        {
-            // encode the doc ids
-            let (num_bits, block_encoded): (u8, &[u8]) = self
-                .block_encoder
-                .compress_block_sorted(self.block.doc_ids(), self.last_doc_id_encoded);
-            self.last_doc_id_encoded = self.block.last_doc();
-            self.skip_write
-                .write_doc(self.last_doc_id_encoded, num_bits);
-            // last el block 0, offset block 1,
-            self.postings_write.extend(block_encoded);
-        }
-        if self.term_has_freq {
-            let (num_bits, block_encoded): (u8, &[u8]) = self
-                .block_encoder
-                .compress_block_unsorted(self.block.term_freqs(), true);
-            self.postings_write.extend(block_encoded);
-            self.skip_write.write_term_freq(num_bits);
-            if self.mode.has_positions() {
-                // We serialize the sum of term freqs within the skip information
-                // in order to navigate through positions.
-                let sum_freq = self.block.term_freqs().iter().cloned().sum();
-                self.skip_write.write_total_term_freq(sum_freq);
-            }
-            let mut blockwand_params = (0u8, 0u32);
-            if let Some(bm25_weight) = self.bm25_weight.as_ref() {
-                if let Some(fieldnorm_reader) = self.fieldnorm_reader.as_ref() {
-                    let docs = self.block.doc_ids().iter().cloned();
-                    let term_freqs = self.block.term_freqs().iter().cloned();
-                    let fieldnorms = docs.map(|doc| fieldnorm_reader.fieldnorm_id(doc));
-                    blockwand_params = fieldnorms
-                        .zip(term_freqs)
-                        .max_by(
-                            |(left_fieldnorm_id, left_term_freq),
-                             (right_fieldnorm_id, right_term_freq)| {
-                                let left_score =
-                                    bm25_weight.tf_factor(*left_fieldnorm_id, *left_term_freq);
-                                let right_score =
-                                    bm25_weight.tf_factor(*right_fieldnorm_id, *right_term_freq);
-                                left_score
-                                    .partial_cmp(&right_score)
-                                    .unwrap_or(Ordering::Equal)
-                            },
-                        )
-                        .unwrap();
-                }
-            }
-            let (fieldnorm_id, term_freq) = blockwand_params;
-            self.skip_write.write_blockwand_max(fieldnorm_id, term_freq);
-        }
-        self.block.clear();
-    }
-}

src/collector/count_collector.rs

@@ -43,7 +43,7 @@ impl Collector for Count {
     fn for_segment(
         &self,
         _: SegmentOrdinal,
-        _: &SegmentReader,
+        _: &dyn SegmentReader,
     ) -> crate::Result<SegmentCountCollector> {
         Ok(SegmentCountCollector::default())
     }

src/collector/docset_collector.rs

@@ -1,7 +1,7 @@
 use std::collections::HashSet;
 
 use super::{Collector, SegmentCollector};
-use crate::{DocAddress, DocId, Score};
+use crate::{DocAddress, DocId, Score, SegmentReader};
 
 /// Collectors that returns the set of DocAddress that matches the query.
 ///
@@ -15,7 +15,7 @@ impl Collector for DocSetCollector {
     fn for_segment(
         &self,
         segment_local_id: crate::SegmentOrdinal,
-        _segment: &crate::SegmentReader,
+        _segment: &dyn SegmentReader,
     ) -> crate::Result<Self::Child> {
         Ok(DocSetChildCollector {
             segment_local_id,

src/collector/facet_collector.rs

@@ -265,7 +265,7 @@ impl Collector for FacetCollector {
     fn for_segment(
         &self,
         _: SegmentOrdinal,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
     ) -> crate::Result<FacetSegmentCollector> {
         let facet_reader = reader.facet_reader(&self.field_name)?;
         let facet_dict = facet_reader.facet_dict();

src/collector/filter_collector_wrapper.rs

@@ -113,7 +113,7 @@ where
     fn for_segment(
         &self,
         segment_local_id: u32,
-        segment_reader: &SegmentReader,
+        segment_reader: &dyn SegmentReader,
     ) -> crate::Result<Self::Child> {
         let column_opt = segment_reader.fast_fields().column_opt(&self.field)?;
 
@@ -287,7 +287,7 @@ where
     fn for_segment(
         &self,
         segment_local_id: u32,
-        segment_reader: &SegmentReader,
+        segment_reader: &dyn SegmentReader,
     ) -> crate::Result<Self::Child> {
         let column_opt = segment_reader.fast_fields().bytes(&self.field)?;
 

src/collector/histogram_collector.rs

@@ -6,7 +6,7 @@ use fastdivide::DividerU64;
 use crate::collector::{Collector, SegmentCollector};
 use crate::fastfield::{FastFieldNotAvailableError, FastValue};
 use crate::schema::Type;
-use crate::{DocId, Score};
+use crate::{DocId, Score, SegmentReader};
 
 /// Histogram builds an histogram of the values of a fastfield for the
 /// collected DocSet.
@@ -110,7 +110,7 @@ impl Collector for HistogramCollector {
     fn for_segment(
         &self,
         _segment_local_id: crate::SegmentOrdinal,
-        segment: &crate::SegmentReader,
+        segment: &dyn SegmentReader,
     ) -> crate::Result<Self::Child> {
         let column_opt = segment.fast_fields().u64_lenient(&self.field)?;
         let (column, _column_type) = column_opt.ok_or_else(|| FastFieldNotAvailableError {

src/collector/mod.rs

@@ -156,7 +156,7 @@ pub trait Collector: Sync + Send {
     fn for_segment(
         &self,
         segment_local_id: SegmentOrdinal,
-        segment: &SegmentReader,
+        segment: &dyn SegmentReader,
     ) -> crate::Result<Self::Child>;
 
     /// Returns true iff the collector requires to compute scores for documents.
@@ -174,7 +174,7 @@ pub trait Collector: Sync + Send {
         &self,
         weight: &dyn Weight,
         segment_ord: u32,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
     ) -> crate::Result<<Self::Child as SegmentCollector>::Fruit> {
         let with_scoring = self.requires_scoring();
         let mut segment_collector = self.for_segment(segment_ord, reader)?;
@@ -186,7 +186,7 @@ pub trait Collector: Sync + Send {
 pub(crate) fn default_collect_segment_impl<TSegmentCollector: SegmentCollector>(
     segment_collector: &mut TSegmentCollector,
     weight: &dyn Weight,
-    reader: &SegmentReader,
+    reader: &dyn SegmentReader,
     with_scoring: bool,
 ) -> crate::Result<()> {
     match (reader.alive_bitset(), with_scoring) {
@@ -255,7 +255,7 @@ impl<TCollector: Collector> Collector for Option<TCollector> {
     fn for_segment(
         &self,
         segment_local_id: SegmentOrdinal,
-        segment: &SegmentReader,
+        segment: &dyn SegmentReader,
     ) -> crate::Result<Self::Child> {
         Ok(if let Some(inner) = self {
             let inner_segment_collector = inner.for_segment(segment_local_id, segment)?;
@@ -336,7 +336,7 @@ where
     fn for_segment(
         &self,
         segment_local_id: u32,
-        segment: &SegmentReader,
+        segment: &dyn SegmentReader,
     ) -> crate::Result<Self::Child> {
         let left = self.0.for_segment(segment_local_id, segment)?;
         let right = self.1.for_segment(segment_local_id, segment)?;
@@ -407,7 +407,7 @@ where
     fn for_segment(
         &self,
         segment_local_id: u32,
-        segment: &SegmentReader,
+        segment: &dyn SegmentReader,
     ) -> crate::Result<Self::Child> {
         let one = self.0.for_segment(segment_local_id, segment)?;
         let two = self.1.for_segment(segment_local_id, segment)?;
@@ -487,7 +487,7 @@ where
     fn for_segment(
         &self,
         segment_local_id: u32,
-        segment: &SegmentReader,
+        segment: &dyn SegmentReader,
     ) -> crate::Result<Self::Child> {
         let one = self.0.for_segment(segment_local_id, segment)?;
         let two = self.1.for_segment(segment_local_id, segment)?;

src/collector/multi_collector.rs

@@ -24,7 +24,7 @@ impl<TCollector: Collector> Collector for CollectorWrapper<TCollector> {
     fn for_segment(
         &self,
         segment_local_id: u32,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
     ) -> crate::Result<Box<dyn BoxableSegmentCollector>> {
         let child = self.0.for_segment(segment_local_id, reader)?;
         Ok(Box::new(SegmentCollectorWrapper(child)))
@@ -209,7 +209,7 @@ impl Collector for MultiCollector<'_> {
     fn for_segment(
         &self,
         segment_local_id: SegmentOrdinal,
-        segment: &SegmentReader,
+        segment: &dyn SegmentReader,
     ) -> crate::Result<MultiCollectorChild> {
         let children = self
             .collector_wrappers

src/collector/sort_key/mod.rs

@@ -1,4 +1,5 @@
 mod order;
+mod sort_by_bytes;
 mod sort_by_erased_type;
 mod sort_by_score;
 mod sort_by_static_fast_value;
@@ -6,6 +7,7 @@ mod sort_by_string;
 mod sort_key_computer;
 
 pub use order::*;
+pub use sort_by_bytes::SortByBytes;
 pub use sort_by_erased_type::SortByErasedType;
 pub use sort_by_score::SortBySimilarityScore;
 pub use sort_by_static_fast_value::SortByStaticFastValue;

src/collector/sort_key/order.rs

@@ -5,7 +5,7 @@ use serde::{Deserialize, Serialize};
 
 use crate::collector::{SegmentSortKeyComputer, SortKeyComputer};
 use crate::schema::{OwnedValue, Schema};
-use crate::{DocId, Order, Score};
+use crate::{DocId, Order, Score, SegmentReader};
 
 fn compare_owned_value<const NULLS_FIRST: bool>(lhs: &OwnedValue, rhs: &OwnedValue) -> Ordering {
     match (lhs, rhs) {
@@ -430,7 +430,7 @@ where
 
     fn segment_sort_key_computer(
         &self,
-        segment_reader: &crate::SegmentReader,
+        segment_reader: &dyn SegmentReader,
     ) -> crate::Result<Self::Child> {
         let child = self.0.segment_sort_key_computer(segment_reader)?;
         Ok(SegmentSortKeyComputerWithComparator {
@@ -468,7 +468,7 @@ where
 
     fn segment_sort_key_computer(
         &self,
-        segment_reader: &crate::SegmentReader,
+        segment_reader: &dyn SegmentReader,
     ) -> crate::Result<Self::Child> {
         let child = self.0.segment_sort_key_computer(segment_reader)?;
         Ok(SegmentSortKeyComputerWithComparator {

src/collector/sort_key/sort_by_bytes.rs

@@ -0,0 +1,168 @@
+use columnar::BytesColumn;
+
+use crate::collector::sort_key::NaturalComparator;
+use crate::collector::{SegmentSortKeyComputer, SortKeyComputer};
+use crate::termdict::TermOrdinal;
+use crate::{DocId, Score};
+
+/// Sort by the first value of a bytes column.
+///
+/// If the field is multivalued, only the first value is considered.
+///
+/// Documents that do not have this value are still considered.
+/// Their sort key will simply be `None`.
+#[derive(Debug, Clone)]
+pub struct SortByBytes {
+    column_name: String,
+}
+
+impl SortByBytes {
+    /// Creates a new sort by bytes sort key computer.
+    pub fn for_field(column_name: impl ToString) -> Self {
+        SortByBytes {
+            column_name: column_name.to_string(),
+        }
+    }
+}
+
+impl SortKeyComputer for SortByBytes {
+    type SortKey = Option<Vec<u8>>;
+    type Child = ByBytesColumnSegmentSortKeyComputer;
+    type Comparator = NaturalComparator;
+
+    fn segment_sort_key_computer(
+        &self,
+        segment_reader: &dyn crate::SegmentReader,
+    ) -> crate::Result<Self::Child> {
+        let bytes_column_opt = segment_reader.fast_fields().bytes(&self.column_name)?;
+        Ok(ByBytesColumnSegmentSortKeyComputer { bytes_column_opt })
+    }
+}
+
+/// Segment-level sort key computer for bytes columns.
+pub struct ByBytesColumnSegmentSortKeyComputer {
+    bytes_column_opt: Option<BytesColumn>,
+}
+
+impl SegmentSortKeyComputer for ByBytesColumnSegmentSortKeyComputer {
+    type SortKey = Option<Vec<u8>>;
+    type SegmentSortKey = Option<TermOrdinal>;
+    type SegmentComparator = NaturalComparator;
+
+    #[inline(always)]
+    fn segment_sort_key(&mut self, doc: DocId, _score: Score) -> Option<TermOrdinal> {
+        let bytes_column = self.bytes_column_opt.as_ref()?;
+        bytes_column.ords().first(doc)
+    }
+
+    fn convert_segment_sort_key(&self, term_ord_opt: Option<TermOrdinal>) -> Option<Vec<u8>> {
+        // TODO: Individual lookups to the dictionary like this are very likely to repeatedly
+        // decompress the same blocks. See https://github.com/quickwit-oss/tantivy/issues/2776
+        let term_ord = term_ord_opt?;
+        let bytes_column = self.bytes_column_opt.as_ref()?;
+        let mut bytes = Vec::new();
+        bytes_column
+            .dictionary()
+            .ord_to_term(term_ord, &mut bytes)
+            .ok()?;
+        Some(bytes)
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::SortByBytes;
+    use crate::collector::TopDocs;
+    use crate::query::AllQuery;
+    use crate::schema::{BytesOptions, Schema, FAST, INDEXED};
+    use crate::{Index, IndexWriter, Order, TantivyDocument};
+
+    #[test]
+    fn test_sort_by_bytes_asc() -> crate::Result<()> {
+        let mut schema_builder = Schema::builder();
+        let bytes_field = schema_builder
+            .add_bytes_field("data", BytesOptions::default().set_fast().set_indexed());
+        let id_field = schema_builder.add_u64_field("id", FAST | INDEXED);
+        let schema = schema_builder.build();
+        let index = Index::create_in_ram(schema);
+        let mut index_writer: IndexWriter = index.writer_for_tests()?;
+
+        // Insert documents with byte values in non-sorted order
+        let test_data: Vec<(u64, Vec<u8>)> = vec![
+            (1, vec![0x02, 0x00]),
+            (2, vec![0x00, 0x10]),
+            (3, vec![0x01, 0x00]),
+            (4, vec![0x00, 0x20]),
+        ];
+
+        for (id, bytes) in &test_data {
+            let mut doc = TantivyDocument::new();
+            doc.add_u64(id_field, *id);
+            doc.add_bytes(bytes_field, bytes);
+            index_writer.add_document(doc)?;
+        }
+        index_writer.commit()?;
+
+        let reader = index.reader()?;
+        let searcher = reader.searcher();
+
+        // Sort ascending by bytes
+        let top_docs =
+            TopDocs::with_limit(10).order_by((SortByBytes::for_field("data"), Order::Asc));
+        let results: Vec<(Option<Vec<u8>>, _)> = searcher.search(&AllQuery, &top_docs)?;
+
+        // Expected order: [0x00,0x10], [0x00,0x20], [0x01,0x00], [0x02,0x00]
+        let sorted_bytes: Vec<Option<Vec<u8>>> = results.into_iter().map(|(b, _)| b).collect();
+        assert_eq!(
+            sorted_bytes,
+            vec![
+                Some(vec![0x00, 0x10]),
+                Some(vec![0x00, 0x20]),
+                Some(vec![0x01, 0x00]),
+                Some(vec![0x02, 0x00]),
+            ]
+        );
+
+        Ok(())
+    }
+
+    #[test]
+    fn test_sort_by_bytes_desc() -> crate::Result<()> {
+        let mut schema_builder = Schema::builder();
+        let bytes_field = schema_builder
+            .add_bytes_field("data", BytesOptions::default().set_fast().set_indexed());
+        let schema = schema_builder.build();
+        let index = Index::create_in_ram(schema);
+        let mut index_writer: IndexWriter = index.writer_for_tests()?;
+
+        let test_data: Vec<Vec<u8>> = vec![vec![0x00, 0x10], vec![0x02, 0x00], vec![0x01, 0x00]];
+
+        for bytes in &test_data {
+            let mut doc = TantivyDocument::new();
+            doc.add_bytes(bytes_field, bytes);
+            index_writer.add_document(doc)?;
+        }
+        index_writer.commit()?;
+
+        let reader = index.reader()?;
+        let searcher = reader.searcher();
+
+        // Sort descending by bytes
+        let top_docs =
+            TopDocs::with_limit(10).order_by((SortByBytes::for_field("data"), Order::Desc));
+        let results: Vec<(Option<Vec<u8>>, _)> = searcher.search(&AllQuery, &top_docs)?;
+
+        // Expected order (descending): [0x02,0x00], [0x01,0x00], [0x00,0x10]
+        let sorted_bytes: Vec<Option<Vec<u8>>> = results.into_iter().map(|(b, _)| b).collect();
+        assert_eq!(
+            sorted_bytes,
+            vec![
+                Some(vec![0x02, 0x00]),
+                Some(vec![0x01, 0x00]),
+                Some(vec![0x00, 0x10]),
+            ]
+        );
+
+        Ok(())
+    }
+}

src/collector/sort_key/sort_by_erased_type.rs

@@ -1,12 +1,12 @@
 use columnar::{ColumnType, MonotonicallyMappableToU64};
 
 use crate::collector::sort_key::{
-    NaturalComparator, SortBySimilarityScore, SortByStaticFastValue, SortByString,
+    NaturalComparator, SortByBytes, SortBySimilarityScore, SortByStaticFastValue, SortByString,
 };
 use crate::collector::{SegmentSortKeyComputer, SortKeyComputer};
 use crate::fastfield::FastFieldNotAvailableError;
 use crate::schema::OwnedValue;
-use crate::{DateTime, DocId, Score};
+use crate::{DateTime, DocId, Score, SegmentReader};
 
 /// Sort by the boxed / OwnedValue representation of either a fast field, or of the score.
 ///
@@ -86,7 +86,7 @@ impl SortKeyComputer for SortByErasedType {
 
     fn segment_sort_key_computer(
         &self,
-        segment_reader: &crate::SegmentReader,
+        segment_reader: &dyn SegmentReader,
     ) -> crate::Result<Self::Child> {
         let inner: Box<dyn ErasedSegmentSortKeyComputer> = match self {
             Self::Field(column_name) => {
@@ -114,6 +114,16 @@ impl SortKeyComputer for SortByErasedType {
                             },
                         })
                     }
+                    ColumnType::Bytes => {
+                        let computer = SortByBytes::for_field(column_name);
+                        let inner = computer.segment_sort_key_computer(segment_reader)?;
+                        Box::new(ErasedSegmentSortKeyComputerWrapper {
+                            inner,
+                            converter: |val: Option<Vec<u8>>| {
+                                val.map(OwnedValue::Bytes).unwrap_or(OwnedValue::Null)
+                            },
+                        })
+                    }
                     ColumnType::U64 => {
                         let computer = SortByStaticFastValue::<u64>::for_field(column_name);
                         let inner = computer.segment_sort_key_computer(segment_reader)?;
@@ -281,6 +291,65 @@ mod tests {
         );
     }
 
+    #[test]
+    fn test_sort_by_owned_bytes() {
+        let mut schema_builder = Schema::builder();
+        let data_field = schema_builder.add_bytes_field("data", FAST);
+        let schema = schema_builder.build();
+        let index = Index::create_in_ram(schema);
+        let mut writer = index.writer_for_tests().unwrap();
+        writer
+            .add_document(doc!(data_field => vec![0x03u8, 0x00]))
+            .unwrap();
+        writer
+            .add_document(doc!(data_field => vec![0x01u8, 0x00]))
+            .unwrap();
+        writer
+            .add_document(doc!(data_field => vec![0x02u8, 0x00]))
+            .unwrap();
+        writer.add_document(doc!()).unwrap();
+        writer.commit().unwrap();
+
+        let reader = index.reader().unwrap();
+        let searcher = reader.searcher();
+
+        // Sort descending (Natural - highest first)
+        let collector = TopDocs::with_limit(10)
+            .order_by((SortByErasedType::for_field("data"), ComparatorEnum::Natural));
+        let top_docs = searcher.search(&AllQuery, &collector).unwrap();
+
+        let values: Vec<OwnedValue> = top_docs.into_iter().map(|(key, _)| key).collect();
+
+        assert_eq!(
+            values,
+            vec![
+                OwnedValue::Bytes(vec![0x03, 0x00]),
+                OwnedValue::Bytes(vec![0x02, 0x00]),
+                OwnedValue::Bytes(vec![0x01, 0x00]),
+                OwnedValue::Null
+            ]
+        );
+
+        // Sort ascending (ReverseNoneLower - lowest first, nulls last)
+        let collector = TopDocs::with_limit(10).order_by((
+            SortByErasedType::for_field("data"),
+            ComparatorEnum::ReverseNoneLower,
+        ));
+        let top_docs = searcher.search(&AllQuery, &collector).unwrap();
+
+        let values: Vec<OwnedValue> = top_docs.into_iter().map(|(key, _)| key).collect();
+
+        assert_eq!(
+            values,
+            vec![
+                OwnedValue::Bytes(vec![0x01, 0x00]),
+                OwnedValue::Bytes(vec![0x02, 0x00]),
+                OwnedValue::Bytes(vec![0x03, 0x00]),
+                OwnedValue::Null
+            ]
+        );
+    }
+
     #[test]
     fn test_sort_by_owned_reverse() {
         let mut schema_builder = Schema::builder();

src/collector/sort_key/sort_by_score.rs

@@ -1,6 +1,6 @@
 use crate::collector::sort_key::NaturalComparator;
 use crate::collector::{SegmentSortKeyComputer, SortKeyComputer, TopNComputer};
-use crate::{DocAddress, DocId, Score};
+use crate::{DocAddress, DocId, Score, SegmentReader};
 
 /// Sort by similarity score.
 #[derive(Clone, Debug, Copy)]
@@ -19,7 +19,7 @@ impl SortKeyComputer for SortBySimilarityScore {
 
     fn segment_sort_key_computer(
         &self,
-        _segment_reader: &crate::SegmentReader,
+        _segment_reader: &dyn SegmentReader,
     ) -> crate::Result<Self::Child> {
         Ok(SortBySimilarityScore)
     }
@@ -29,7 +29,7 @@ impl SortKeyComputer for SortBySimilarityScore {
         &self,
         k: usize,
         weight: &dyn crate::query::Weight,
-        reader: &crate::SegmentReader,
+        reader: &dyn SegmentReader,
         segment_ord: u32,
     ) -> crate::Result<Vec<(Self::SortKey, DocAddress)>> {
         let mut top_n: TopNComputer<Score, DocId, Self::Comparator> =

src/collector/sort_key/sort_by_static_fast_value.rs

@@ -61,7 +61,7 @@ impl<T: FastValue> SortKeyComputer for SortByStaticFastValue<T> {
 
     fn segment_sort_key_computer(
         &self,
-        segment_reader: &SegmentReader,
+        segment_reader: &dyn SegmentReader,
     ) -> crate::Result<Self::Child> {
         let sort_column_opt = segment_reader.fast_fields().u64_lenient(&self.field)?;
         let (sort_column, _sort_column_type) =

src/collector/sort_key/sort_by_string.rs

@@ -3,7 +3,7 @@ use columnar::StrColumn;
 use crate::collector::sort_key::NaturalComparator;
 use crate::collector::{SegmentSortKeyComputer, SortKeyComputer};
 use crate::termdict::TermOrdinal;
-use crate::{DocId, Score};
+use crate::{DocId, Score, SegmentReader};
 
 /// Sort by the first value of a string column.
 ///
@@ -35,7 +35,7 @@ impl SortKeyComputer for SortByString {
 
     fn segment_sort_key_computer(
         &self,
-        segment_reader: &crate::SegmentReader,
+        segment_reader: &dyn SegmentReader,
     ) -> crate::Result<Self::Child> {
         let str_column_opt = segment_reader.fast_fields().str(&self.column_name)?;
         Ok(ByStringColumnSegmentSortKeyComputer { str_column_opt })

src/collector/sort_key/sort_key_computer.rs

@@ -119,7 +119,7 @@ pub trait SortKeyComputer: Sync {
         &self,
         k: usize,
         weight: &dyn crate::query::Weight,
-        reader: &crate::SegmentReader,
+        reader: &dyn SegmentReader,
         segment_ord: u32,
     ) -> crate::Result<Vec<(Self::SortKey, DocAddress)>> {
         let with_scoring = self.requires_scoring();
@@ -135,7 +135,7 @@ pub trait SortKeyComputer: Sync {
     }
 
     /// Builds a child sort key computer for a specific segment.
-    fn segment_sort_key_computer(&self, segment_reader: &SegmentReader) -> Result<Self::Child>;
+    fn segment_sort_key_computer(&self, segment_reader: &dyn SegmentReader) -> Result<Self::Child>;
 }
 
 impl<HeadSortKeyComputer, TailSortKeyComputer> SortKeyComputer
@@ -156,7 +156,7 @@ where
         (self.0.comparator(), self.1.comparator())
     }
 
-    fn segment_sort_key_computer(&self, segment_reader: &SegmentReader) -> Result<Self::Child> {
+    fn segment_sort_key_computer(&self, segment_reader: &dyn SegmentReader) -> Result<Self::Child> {
         Ok((
             self.0.segment_sort_key_computer(segment_reader)?,
             self.1.segment_sort_key_computer(segment_reader)?,
@@ -357,7 +357,7 @@ where
         )
     }
 
-    fn segment_sort_key_computer(&self, segment_reader: &SegmentReader) -> Result<Self::Child> {
+    fn segment_sort_key_computer(&self, segment_reader: &dyn SegmentReader) -> Result<Self::Child> {
         let sort_key_computer1 = self.0.segment_sort_key_computer(segment_reader)?;
         let sort_key_computer2 = self.1.segment_sort_key_computer(segment_reader)?;
         let sort_key_computer3 = self.2.segment_sort_key_computer(segment_reader)?;
@@ -420,7 +420,7 @@ where
         SortKeyComputer4::Comparator,
     );
 
-    fn segment_sort_key_computer(&self, segment_reader: &SegmentReader) -> Result<Self::Child> {
+    fn segment_sort_key_computer(&self, segment_reader: &dyn SegmentReader) -> Result<Self::Child> {
         let sort_key_computer1 = self.0.segment_sort_key_computer(segment_reader)?;
         let sort_key_computer2 = self.1.segment_sort_key_computer(segment_reader)?;
         let sort_key_computer3 = self.2.segment_sort_key_computer(segment_reader)?;
@@ -454,7 +454,7 @@ where
 
 impl<F, SegmentF, TSortKey> SortKeyComputer for F
 where
-    F: 'static + Send + Sync + Fn(&SegmentReader) -> SegmentF,
+    F: 'static + Send + Sync + Fn(&dyn SegmentReader) -> SegmentF,
     SegmentF: 'static + FnMut(DocId) -> TSortKey,
     TSortKey: 'static + PartialOrd + Clone + Send + Sync + std::fmt::Debug,
 {
@@ -462,7 +462,7 @@ where
     type Child = SegmentF;
     type Comparator = NaturalComparator;
 
-    fn segment_sort_key_computer(&self, segment_reader: &SegmentReader) -> Result<Self::Child> {
+    fn segment_sort_key_computer(&self, segment_reader: &dyn SegmentReader) -> Result<Self::Child> {
         Ok((self)(segment_reader))
     }
 }
@@ -509,10 +509,10 @@ mod tests {
 
     #[test]
     fn test_lazy_score_computer() {
-        let score_computer_primary = |_segment_reader: &SegmentReader| |_doc: DocId| 200u32;
+        let score_computer_primary = |_segment_reader: &dyn SegmentReader| |_doc: DocId| 200u32;
         let call_count = Arc::new(AtomicUsize::new(0));
         let call_count_clone = call_count.clone();
-        let score_computer_secondary = move |_segment_reader: &SegmentReader| {
+        let score_computer_secondary = move |_segment_reader: &dyn SegmentReader| {
             let call_count_new_clone = call_count_clone.clone();
             move |_doc: DocId| {
                 call_count_new_clone.fetch_add(1, AtomicOrdering::SeqCst);
@@ -572,10 +572,10 @@ mod tests {
 
     #[test]
     fn test_lazy_score_computer_dynamic_ordering() {
-        let score_computer_primary = |_segment_reader: &SegmentReader| |_doc: DocId| 200u32;
+        let score_computer_primary = |_segment_reader: &dyn SegmentReader| |_doc: DocId| 200u32;
         let call_count = Arc::new(AtomicUsize::new(0));
         let call_count_clone = call_count.clone();
-        let score_computer_secondary = move |_segment_reader: &SegmentReader| {
+        let score_computer_secondary = move |_segment_reader: &dyn SegmentReader| {
             let call_count_new_clone = call_count_clone.clone();
             move |_doc: DocId| {
                 call_count_new_clone.fetch_add(1, AtomicOrdering::SeqCst);

src/collector/sort_key_top_collector.rs

@@ -32,7 +32,11 @@ where TSortKeyComputer: SortKeyComputer + Send + Sync + 'static
         self.sort_key_computer.check_schema(schema)
     }
 
-    fn for_segment(&self, segment_ord: u32, segment_reader: &SegmentReader) -> Result<Self::Child> {
+    fn for_segment(
+        &self,
+        segment_ord: u32,
+        segment_reader: &dyn SegmentReader,
+    ) -> Result<Self::Child> {
         let segment_sort_key_computer = self
             .sort_key_computer
             .segment_sort_key_computer(segment_reader)?;
@@ -63,7 +67,7 @@ where TSortKeyComputer: SortKeyComputer + Send + Sync + 'static
         &self,
         weight: &dyn Weight,
         segment_ord: u32,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
     ) -> crate::Result<Vec<(TSortKeyComputer::SortKey, DocAddress)>> {
         let k = self.doc_range.end;
         let docs = self

src/collector/tests.rs

@@ -5,7 +5,7 @@ use crate::query::{AllQuery, QueryParser};
 use crate::schema::{Schema, FAST, TEXT};
 use crate::time::format_description::well_known::Rfc3339;
 use crate::time::OffsetDateTime;
-use crate::{DateTime, DocAddress, Index, Searcher, TantivyDocument};
+use crate::{DateTime, DocAddress, Index, Searcher, SegmentReader, TantivyDocument};
 
 pub const TEST_COLLECTOR_WITH_SCORE: TestCollector = TestCollector {
     compute_score: true,
@@ -109,7 +109,7 @@ impl Collector for TestCollector {
     fn for_segment(
         &self,
         segment_id: SegmentOrdinal,
-        _reader: &SegmentReader,
+        _reader: &dyn SegmentReader,
     ) -> crate::Result<TestSegmentCollector> {
         Ok(TestSegmentCollector {
             segment_id,
@@ -180,7 +180,7 @@ impl Collector for FastFieldTestCollector {
     fn for_segment(
         &self,
         _: SegmentOrdinal,
-        segment_reader: &SegmentReader,
+        segment_reader: &dyn SegmentReader,
     ) -> crate::Result<FastFieldSegmentCollector> {
         let reader = segment_reader
             .fast_fields()
@@ -243,7 +243,7 @@ impl Collector for BytesFastFieldTestCollector {
     fn for_segment(
         &self,
         _segment_local_id: u32,
-        segment_reader: &SegmentReader,
+        segment_reader: &dyn SegmentReader,
     ) -> crate::Result<BytesFastFieldSegmentCollector> {
         let column_opt = segment_reader.fast_fields().bytes(&self.field)?;
         Ok(BytesFastFieldSegmentCollector {

src/collector/top_score_collector.rs

@@ -393,7 +393,7 @@ impl TopDocs {
     /// // This is where we build our collector with our custom score.
     /// let top_docs_by_custom_score = TopDocs
     ///         ::with_limit(10)
-    ///          .tweak_score(move |segment_reader: &SegmentReader| {
+    ///          .tweak_score(move |segment_reader: &dyn SegmentReader| {
     ///             // The argument is a function that returns our scoring
     ///             // function.
     ///             //
@@ -442,7 +442,7 @@ pub struct TweakScoreFn<F>(F);
 
 impl<F, TTweakScoreSortKeyFn, TSortKey> SortKeyComputer for TweakScoreFn<F>
 where
-    F: 'static + Send + Sync + Fn(&SegmentReader) -> TTweakScoreSortKeyFn,
+    F: 'static + Send + Sync + Fn(&dyn SegmentReader) -> TTweakScoreSortKeyFn,
     TTweakScoreSortKeyFn: 'static + Fn(DocId, Score) -> TSortKey,
     TweakScoreSegmentSortKeyComputer<TTweakScoreSortKeyFn>:
         SegmentSortKeyComputer<SortKey = TSortKey, SegmentSortKey = TSortKey>,
@@ -458,7 +458,7 @@ where
 
     fn segment_sort_key_computer(
         &self,
-        segment_reader: &SegmentReader,
+        segment_reader: &dyn SegmentReader,
     ) -> crate::Result<Self::Child> {
         Ok({
             TweakScoreSegmentSortKeyComputer {
@@ -1525,7 +1525,7 @@ mod tests {
         let text_query = query_parser.parse_query("droopy tax")?;
         let collector = TopDocs::with_limit(2)
             .and_offset(1)
-            .order_by(move |_segment_reader: &SegmentReader| move |doc: DocId| doc);
+            .order_by(move |_segment_reader: &dyn SegmentReader| move |doc: DocId| doc);
         let score_docs: Vec<(u32, DocAddress)> =
             index.reader()?.searcher().search(&text_query, &collector)?;
         assert_eq!(
@@ -1543,7 +1543,7 @@ mod tests {
         let text_query = query_parser.parse_query("droopy tax").unwrap();
         let collector = TopDocs::with_limit(2)
             .and_offset(1)
-            .order_by(move |_segment_reader: &SegmentReader| move |doc: DocId| doc);
+            .order_by(move |_segment_reader: &dyn SegmentReader| move |doc: DocId| doc);
         let score_docs: Vec<(u32, DocAddress)> = index
             .reader()
             .unwrap()

src/core/mod.rs

@@ -8,7 +8,7 @@ use std::path::Path;
 use once_cell::sync::Lazy;
 
 pub use self::executor::Executor;
-pub use self::searcher::{Searcher, SearcherGeneration};
+pub use self::searcher::{Searcher, SearcherContext, SearcherGeneration};
 
 /// The meta file contains all the information about the list of segments and the schema
 /// of the index.

src/core/searcher.rs

@@ -4,13 +4,13 @@ use std::{fmt, io};
 
 use crate::collector::Collector;
 use crate::core::Executor;
-use crate::index::{SegmentId, SegmentReader};
+use crate::index::{Index, SegmentId, SegmentReader};
 use crate::query::{Bm25StatisticsProvider, EnableScoring, Query};
-use crate::schema::document::DocumentDeserialize;
-use crate::schema::{Schema, Term};
+use crate::schema::{Field, FieldType, Schema, TantivyDocument, Term};
 use crate::space_usage::SearcherSpaceUsage;
-use crate::store::{CacheStats, StoreReader};
-use crate::{DocAddress, Index, Opstamp, TrackedObject};
+use crate::store::{CacheStats, StoreReader, DOCSTORE_CACHE_CAPACITY};
+use crate::tokenizer::{TextAnalyzer, TokenizerManager};
+use crate::{DocAddress, Inventory, Opstamp, TantivyError, TrackedObject};
 
 /// Identifies the searcher generation accessed by a [`Searcher`].
 ///
@@ -36,7 +36,7 @@ pub struct SearcherGeneration {
 
 impl SearcherGeneration {
     pub(crate) fn from_segment_readers(
-        segment_readers: &[SegmentReader],
+        segment_readers: &[Arc<dyn SegmentReader>],
         generation_id: u64,
     ) -> Self {
         let mut segment_id_to_del_opstamp = BTreeMap::new();
@@ -61,6 +61,103 @@ impl SearcherGeneration {
     }
 }
 
+/// Search-time context required by a [`Searcher`].
+#[derive(Clone)]
+pub struct SearcherContext {
+    schema: Schema,
+    executor: Executor,
+    tokenizers: TokenizerManager,
+    fast_field_tokenizers: TokenizerManager,
+}
+
+impl SearcherContext {
+    /// Creates a context from explicit search-time components.
+    pub fn new(
+        schema: Schema,
+        executor: Executor,
+        tokenizers: TokenizerManager,
+        fast_field_tokenizers: TokenizerManager,
+    ) -> SearcherContext {
+        SearcherContext {
+            schema,
+            executor,
+            tokenizers,
+            fast_field_tokenizers,
+        }
+    }
+
+    /// Creates a context from an index.
+    pub fn from_index<C: crate::codec::Codec>(index: &Index<C>) -> SearcherContext {
+        SearcherContext::new(
+            index.schema(),
+            index.search_executor().clone(),
+            index.tokenizers().clone(),
+            index.fast_field_tokenizer().clone(),
+        )
+    }
+
+    /// Access the schema associated with this context.
+    pub fn schema(&self) -> &Schema {
+        &self.schema
+    }
+
+    /// Access the executor associated with this context.
+    pub fn search_executor(&self) -> &Executor {
+        &self.executor
+    }
+
+    /// Access the tokenizer manager associated with this context.
+    pub fn tokenizers(&self) -> &TokenizerManager {
+        &self.tokenizers
+    }
+
+    /// Access the fast field tokenizer manager associated with this context.
+    pub fn fast_field_tokenizer(&self) -> &TokenizerManager {
+        &self.fast_field_tokenizers
+    }
+
+    /// Get the tokenizer associated with a specific field.
+    pub fn tokenizer_for_field(&self, field: Field) -> crate::Result<TextAnalyzer> {
+        let field_entry = self.schema.get_field_entry(field);
+        let field_type = field_entry.field_type();
+        let indexing_options_opt = match field_type {
+            FieldType::JsonObject(options) => options.get_text_indexing_options(),
+            FieldType::Str(options) => options.get_indexing_options(),
+            _ => {
+                return Err(TantivyError::SchemaError(format!(
+                    "{:?} is not a text field.",
+                    field_entry.name()
+                )))
+            }
+        };
+        let indexing_options = indexing_options_opt.ok_or_else(|| {
+            TantivyError::InvalidArgument(format!(
+                "No indexing options set for field {field_entry:?}"
+            ))
+        })?;
+
+        self.tokenizers
+            .get(indexing_options.tokenizer())
+            .ok_or_else(|| {
+                TantivyError::InvalidArgument(format!(
+                    "No Tokenizer found for field {field_entry:?}"
+                ))
+            })
+    }
+}
+
+impl<C: crate::codec::Codec> From<&Index<C>> for SearcherContext {
+    fn from(index: &Index<C>) -> Self {
+        SearcherContext::from_index(index)
+    }
+}
+
+impl<C: crate::codec::Codec> From<Index<C>> for SearcherContext {
+    fn from(index: Index<C>) -> Self {
+        SearcherContext::from(&index)
+    }
+}
+
 /// Holds a list of `SegmentReader`s ready for search.
 ///
 /// It guarantees that the `Segment` will not be removed before
@@ -71,9 +168,66 @@ pub struct Searcher {
 }
 
 impl Searcher {
-    /// Returns the `Index` associated with the `Searcher`
-    pub fn index(&self) -> &Index {
-        &self.inner.index
+    /// Creates a `Searcher` from an arbitrary list of segment readers.
+    ///
+    /// This is useful when segment readers are not opened from
+    /// `IndexReader` / `meta.json` (e.g. external segment sources).
+    /// The generated [`SearcherGeneration`] uses `generation_id = 0`.
+    pub fn from_segment_readers<Ctx: Into<SearcherContext>>(
+        context: Ctx,
+        segment_readers: Vec<Arc<dyn SegmentReader>>,
+    ) -> crate::Result<Searcher> {
+        Self::from_segment_readers_with_generation_id(context, segment_readers, 0)
+    }
+
+    /// Same as [`Searcher::from_segment_readers`] but allows setting
+    /// a custom generation id.
+    pub fn from_segment_readers_with_generation_id<Ctx: Into<SearcherContext>>(
+        context: Ctx,
+        segment_readers: Vec<Arc<dyn SegmentReader>>,
+        generation_id: u64,
+    ) -> crate::Result<Searcher> {
+        let context = context.into();
+        let generation = SearcherGeneration::from_segment_readers(&segment_readers, generation_id);
+        let tracked_generation = Inventory::default().track(generation);
+        let inner = SearcherInner::new(
+            context,
+            segment_readers,
+            tracked_generation,
+            DOCSTORE_CACHE_CAPACITY,
+        )?;
+        Ok(Arc::new(inner).into())
+    }
+
+    /// Returns the search context associated with the `Searcher`.
+    pub fn context(&self) -> &SearcherContext {
+        &self.inner.context
+    }
+
+    /// Deprecated alias for [`Searcher::context`].
+    #[deprecated(note = "use Searcher::context()")]
+    pub fn index(&self) -> &SearcherContext {
+        self.context()
+    }
+
+    /// Access the search executor associated with this searcher.
+    pub fn search_executor(&self) -> &Executor {
+        self.context().search_executor()
+    }
+
+    /// Access the tokenizer manager associated with this searcher.
+    pub fn tokenizers(&self) -> &TokenizerManager {
+        self.context().tokenizers()
+    }
+
+    /// Access the fast field tokenizer manager associated with this searcher.
+    pub fn fast_field_tokenizer(&self) -> &TokenizerManager {
+        self.context().fast_field_tokenizer()
+    }
+
+    /// Get the tokenizer associated with a specific field.
+    pub fn tokenizer_for_field(&self, field: Field) -> crate::Result<TextAnalyzer> {
+        self.context().tokenizer_for_field(field)
     }
 
     /// [`SearcherGeneration`] which identifies the version of the snapshot held by this `Searcher`.
@@ -85,7 +239,7 @@ impl Searcher {
     ///
     /// The searcher uses the segment ordinal to route the
     /// request to the right `Segment`.
-    pub fn doc<D: DocumentDeserialize>(&self, doc_address: DocAddress) -> crate::Result<D> {
+    pub fn doc(&self, doc_address: DocAddress) -> crate::Result<TantivyDocument> {
         let store_reader = &self.inner.store_readers[doc_address.segment_ord as usize];
         store_reader.get(doc_address.doc_id)
     }
@@ -105,18 +259,15 @@ impl Searcher {
 
     /// Fetches a document in an asynchronous manner.
     #[cfg(feature = "quickwit")]
-    pub async fn doc_async<D: DocumentDeserialize>(
-        &self,
-        doc_address: DocAddress,
-    ) -> crate::Result<D> {
-        let executor = self.inner.index.search_executor();
+    pub async fn doc_async(&self, doc_address: DocAddress) -> crate::Result<TantivyDocument> {
+        let executor = self.search_executor();
         let store_reader = &self.inner.store_readers[doc_address.segment_ord as usize];
         store_reader.get_async(doc_address.doc_id, executor).await
     }
 
     /// Access the schema associated with the index of this searcher.
     pub fn schema(&self) -> &Schema {
-        &self.inner.schema
+        self.context().schema()
     }
 
     /// Returns the overall number of documents in the index.
@@ -154,13 +305,13 @@ impl Searcher {
     }
 
     /// Return the list of segment readers
-    pub fn segment_readers(&self) -> &[SegmentReader] {
+    pub fn segment_readers(&self) -> &[Arc<dyn SegmentReader>] {
         &self.inner.segment_readers
     }
 
     /// Returns the segment_reader associated with the given segment_ord
-    pub fn segment_reader(&self, segment_ord: u32) -> &SegmentReader {
-        &self.inner.segment_readers[segment_ord as usize]
+    pub fn segment_reader(&self, segment_ord: u32) -> &dyn SegmentReader {
+        self.inner.segment_readers[segment_ord as usize].as_ref()
     }
 
     /// Runs a query on the segment readers wrapped by the searcher.
@@ -201,7 +352,7 @@ impl Searcher {
         } else {
             EnableScoring::disabled_from_searcher(self)
         };
-        let executor = self.inner.index.search_executor();
+        let executor = self.search_executor();
         self.search_with_executor(query, collector, executor, enabled_scoring)
     }
 
@@ -229,7 +380,11 @@ impl Searcher {
         let segment_readers = self.segment_readers();
         let fruits = executor.map(
             |(segment_ord, segment_reader)| {
-                collector.collect_segment(weight.as_ref(), segment_ord as u32, segment_reader)
+                collector.collect_segment(
+                    weight.as_ref(),
+                    segment_ord as u32,
+                    segment_reader.as_ref(),
+                )
             },
             segment_readers.iter().enumerate(),
         )?;
@@ -257,19 +412,17 @@ impl From<Arc<SearcherInner>> for Searcher {
 /// It guarantees that the `Segment` will not be removed before
 /// the destruction of the `Searcher`.
 pub(crate) struct SearcherInner {
-    schema: Schema,
-    index: Index,
-    segment_readers: Vec<SegmentReader>,
-    store_readers: Vec<StoreReader>,
+    context: SearcherContext,
+    segment_readers: Vec<Arc<dyn SegmentReader>>,
+    store_readers: Vec<Box<dyn StoreReader>>,
     generation: TrackedObject<SearcherGeneration>,
 }
 
 impl SearcherInner {
     /// Creates a new `Searcher`
     pub(crate) fn new(
-        schema: Schema,
-        index: Index,
-        segment_readers: Vec<SegmentReader>,
+        context: SearcherContext,
+        segment_readers: Vec<Arc<dyn SegmentReader>>,
         generation: TrackedObject<SearcherGeneration>,
         doc_store_cache_num_blocks: usize,
     ) -> io::Result<SearcherInner> {
@@ -281,14 +434,13 @@ impl SearcherInner {
             generation.segments(),
             "Set of segments referenced by this Searcher and its SearcherGeneration must match"
         );
-        let store_readers: Vec<StoreReader> = segment_readers
+        let store_readers: Vec<Box<dyn StoreReader>> = segment_readers
             .iter()
             .map(|segment_reader| segment_reader.get_store_reader(doc_store_cache_num_blocks))
             .collect::<io::Result<Vec<_>>>()?;
 
         Ok(SearcherInner {
-            schema,
-            index,
+            context,
             segment_readers,
             store_readers,
             generation,
@@ -301,7 +453,7 @@ impl fmt::Debug for Searcher {
         let segment_ids = self
             .segment_readers()
             .iter()
-            .map(SegmentReader::segment_id)
+            .map(|segment_reader| segment_reader.segment_id())
             .collect::<Vec<_>>();
         write!(f, "Searcher({segment_ids:?})")
     }

src/core/tests.rs

@@ -7,8 +7,8 @@ use crate::query::TermQuery;
 use crate::schema::{Field, IndexRecordOption, Schema, INDEXED, STRING, TEXT};
 use crate::tokenizer::TokenizerManager;
 use crate::{
-    Directory, DocSet, Index, IndexBuilder, IndexReader, IndexSettings, IndexWriter, ReloadPolicy,
-    TantivyDocument, Term,
+    Directory, DocSet, Executor, Index, IndexBuilder, IndexReader, IndexSettings, IndexWriter,
+    ReloadPolicy, Searcher, SearcherContext, TantivyDocument, Term,
 };
 
 #[test]
@@ -300,6 +300,40 @@ fn test_single_segment_index_writer() -> crate::Result<()> {
     Ok(())
 }
 
+#[test]
+fn test_searcher_from_external_segment_readers() -> crate::Result<()> {
+    let mut schema_builder = Schema::builder();
+    let text_field = schema_builder.add_text_field("text", TEXT);
+    let schema = schema_builder.build();
+    let index = Index::create_in_ram(schema.clone());
+    let mut writer: IndexWriter = index.writer_for_tests()?;
+    writer.add_document(doc!(text_field => "hello"))?;
+    writer.add_document(doc!(text_field => "hello"))?;
+    writer.commit()?;
+
+    let reader = index.reader()?;
+    let searcher = reader.searcher();
+    let segment_readers = searcher.segment_readers().to_vec();
+    let context = SearcherContext::new(
+        schema,
+        Executor::single_thread(),
+        TokenizerManager::default(),
+        TokenizerManager::default(),
+    );
+    let custom_searcher =
+        Searcher::from_segment_readers_with_generation_id(context, segment_readers, 42)?;
+
+    let term_query = TermQuery::new(
+        Term::from_field_text(text_field, "hello"),
+        IndexRecordOption::Basic,
+    );
+    let count = custom_searcher.search(&term_query, &Count)?;
+    assert_eq!(count, 2);
+    assert_eq!(custom_searcher.generation().generation_id(), 42);
+    assert_eq!(custom_searcher.segment_readers().len(), 1);
+    Ok(())
+}
+
 #[test]
 fn test_merging_segment_update_docfreq() {
     let mut schema_builder = Schema::builder();

src/directory/composite_file.rs

@@ -167,6 +167,9 @@ impl CompositeFile {
             .map(|byte_range| self.data.slice(byte_range.clone()))
     }
 
+    /// Returns per-field byte usage for all slices stored in this composite file.
+    ///
+    /// The provided `schema` is used to resolve field ids into field names.
     pub fn space_usage(&self, schema: &Schema) -> PerFieldSpaceUsage {
         let mut fields = Vec::new();
         for (&field_addr, byte_range) in &self.offsets_index {

src/directory/mod.rs

@@ -21,7 +21,7 @@ use std::path::PathBuf;
 pub use common::file_slice::{FileHandle, FileSlice};
 pub use common::{AntiCallToken, OwnedBytes, TerminatingWrite};
 
-pub(crate) use self::composite_file::{CompositeFile, CompositeWrite};
+pub use self::composite_file::{CompositeFile, CompositeWrite};
 pub use self::directory::{Directory, DirectoryClone, DirectoryLock};
 pub use self::directory_lock::{Lock, INDEX_WRITER_LOCK, META_LOCK};
 pub use self::ram_directory::RamDirectory;
@@ -52,7 +52,7 @@ pub use self::mmap_directory::MmapDirectory;
 ///
 /// `WritePtr` are required to implement both Write
 /// and Seek.
-pub type WritePtr = BufWriter<Box<dyn TerminatingWrite>>;
+pub type WritePtr = BufWriter<Box<dyn TerminatingWrite + Send + Sync>>;
 
 #[cfg(test)]
 mod tests;

src/docset.rs

@@ -1,3 +1,4 @@
+use std::borrow::BorrowMut;
 use std::ops::{Deref as _, DerefMut as _};
 
 use common::BitSet;
@@ -265,8 +266,10 @@ impl<TDocSet: DocSet + ?Sized> DocSet for Box<TDocSet> {
         self.deref_mut().seek(target)
     }
 
+    #[inline]
     fn seek_danger(&mut self, target: DocId) -> SeekDangerResult {
-        self.deref_mut().seek_danger(target)
+        let unboxed: &mut TDocSet = self.borrow_mut();
+        unboxed.seek_danger(target)
     }
 
     #[inline]

src/fastfield/facet_reader.rs

@@ -84,9 +84,7 @@ mod tests {
         let mut facet = Facet::default();
         facet_reader.facet_from_ord(0, &mut facet).unwrap();
         assert_eq!(facet.to_path_string(), "/a/b");
-        let doc = searcher
-            .doc::<TantivyDocument>(DocAddress::new(0u32, 0u32))
-            .unwrap();
+        let doc = searcher.doc(DocAddress::new(0u32, 0u32)).unwrap();
         let value = doc
             .get_first(facet_field)
             .and_then(|v| v.as_value().as_facet());
@@ -145,7 +143,7 @@ mod tests {
         let mut facet_ords = Vec::new();
         facet_ords.extend(facet_reader.facet_ords(0u32));
         assert_eq!(&facet_ords, &[0u64]);
-        let doc = searcher.doc::<TantivyDocument>(DocAddress::new(0u32, 0u32))?;
+        let doc = searcher.doc(DocAddress::new(0u32, 0u32))?;
         let value: Option<Facet> = doc
             .get_first(facet_field)
             .and_then(|v| v.as_facet())

src/fastfield/mod.rs

@@ -96,7 +96,7 @@ mod tests {
     };
     use crate::time::OffsetDateTime;
     use crate::tokenizer::{LowerCaser, RawTokenizer, TextAnalyzer, TokenizerManager};
-    use crate::{Index, IndexWriter, SegmentReader};
+    use crate::{Index, IndexWriter};
 
     pub static SCHEMA: Lazy<Schema> = Lazy::new(|| {
         let mut schema_builder = Schema::builder();
@@ -430,7 +430,7 @@ mod tests {
             .searcher()
             .segment_readers()
             .iter()
-            .map(SegmentReader::segment_id)
+            .map(|segment_reader| segment_reader.segment_id())
             .collect();
         assert_eq!(segment_ids.len(), 2);
         index_writer.merge(&segment_ids[..]).wait().unwrap();

src/fastfield/readers.rs

@@ -25,7 +25,8 @@ pub struct FastFieldReaders {
 }
 
 impl FastFieldReaders {
-    pub(crate) fn open(fast_field_file: FileSlice, schema: Schema) -> io::Result<FastFieldReaders> {
+    /// Opens the segment fast-field container and binds it to a schema.
+    pub fn open(fast_field_file: FileSlice, schema: Schema) -> io::Result<FastFieldReaders> {
         let columnar = Arc::new(ColumnarReader::open(fast_field_file)?);
         Ok(FastFieldReaders { columnar, schema })
     }
@@ -39,7 +40,8 @@ impl FastFieldReaders {
         self.resolve_column_name_given_default_field(column_name, default_field_opt)
     }
 
-    pub(crate) fn space_usage(&self) -> io::Result<PerFieldSpaceUsage> {
+    /// Returns per-field space usage for all loaded fast-field columns.
+    pub fn space_usage(&self) -> io::Result<PerFieldSpaceUsage> {
         let mut per_field_usages: Vec<FieldUsage> = Default::default();
         for (mut field_name, column_handle) in self.columnar.iter_columns()? {
             json_path_sep_to_dot(&mut field_name);
@@ -51,7 +53,8 @@ impl FastFieldReaders {
         Ok(PerFieldSpaceUsage::new(per_field_usages))
     }
 
-    pub(crate) fn columnar(&self) -> &ColumnarReader {
+    /// Returns the underlying `ColumnarReader`.
+    pub fn columnar(&self) -> &ColumnarReader {
         self.columnar.as_ref()
     }
 

src/index/index.rs

@@ -26,7 +26,6 @@ use crate::reader::{IndexReader, IndexReaderBuilder};
 use crate::schema::document::Document;
 use crate::schema::{Field, FieldType, Schema};
 use crate::tokenizer::{TextAnalyzer, TokenizerManager};
-use crate::SegmentReader;
 
 fn load_metas(
     directory: &dyn Directory,
@@ -408,22 +407,6 @@ impl Index {
 }
 
 impl<Codec: crate::codec::Codec> Index<Codec> {
-    /// Returns a version of this index with the standard codec.
-    /// This is useful when you need to pass the index to APIs that
-    /// don't care about the codec (e.g., for reading).
-    pub(crate) fn with_standard_codec(&self) -> Index<StandardCodec> {
-        Index {
-            directory: self.directory.clone(),
-            schema: self.schema.clone(),
-            settings: self.settings.clone(),
-            executor: self.executor.clone(),
-            tokenizers: self.tokenizers.clone(),
-            fast_field_tokenizers: self.fast_field_tokenizers.clone(),
-            inventory: self.inventory.clone(),
-            codec: StandardCodec,
-        }
-    }
-
     /// Open the index using the provided directory
     #[inline(never)]
     pub fn open_with_codec(directory: Box<dyn Directory>) -> crate::Result<Index<Codec>> {
@@ -579,7 +562,15 @@ impl<Codec: crate::codec::Codec> Index<Codec> {
         let segments = self.searchable_segments()?;
         let fields_metadata: Vec<Vec<FieldMetadata>> = segments
             .into_iter()
-            .map(|segment| SegmentReader::open(&segment)?.fields_metadata())
+            .map(|segment| {
+                let segment_reader = segment.index().codec().open_segment_reader(
+                    segment.index().directory(),
+                    segment.meta(),
+                    segment.schema(),
+                    None,
+                )?;
+                segment_reader.fields_metadata()
+            })
             .collect::<Result<_, _>>()?;
         Ok(merge_field_meta_data(fields_metadata))
     }

src/index/inverted_index_reader.rs

@@ -1,8 +1,11 @@
+#[cfg(feature = "quickwit")]
+use std::future::Future;
 use std::io;
-use std::sync::Arc;
+#[cfg(feature = "quickwit")]
+use std::pin::Pin;
 
 use common::json_path_writer::JSON_END_OF_PATH;
-use common::{BinarySerializable, ByteCount, OwnedBytes};
+use common::{BinarySerializable, BitSet, ByteCount, OwnedBytes};
 #[cfg(feature = "quickwit")]
 use futures_util::{FutureExt, StreamExt, TryStreamExt};
 #[cfg(feature = "quickwit")]
@@ -10,43 +13,213 @@ use itertools::Itertools;
 #[cfg(feature = "quickwit")]
 use tantivy_fst::automaton::{AlwaysMatch, Automaton};
 
-use crate::codec::postings::PostingsCodec;
-use crate::codec::{Codec, ObjectSafeCodec, StandardCodec};
+use crate::codec::postings::RawPostingsData;
+use crate::codec::standard::postings::{
+    fill_bitset_from_raw_data, load_postings_from_raw_data, SegmentPostings,
+};
 use crate::directory::FileSlice;
 use crate::fieldnorm::FieldNormReader;
 use crate::postings::{Postings, TermInfo};
 use crate::query::term_query::TermScorer;
-use crate::query::{Bm25Weight, PhraseScorer, Scorer};
+use crate::query::{box_scorer, Bm25Weight, PhraseScorer, Scorer};
 use crate::schema::{IndexRecordOption, Term, Type};
 use crate::termdict::TermDictionary;
 
+#[cfg(feature = "quickwit")]
+pub type TermRangeBounds = (std::ops::Bound<Term>, std::ops::Bound<Term>);
+
+/// Type-erased term scorer guaranteed to wrap a Tantivy [`TermScorer`].
+pub struct BoxedTermScorer(Box<dyn Scorer>);
+
+impl BoxedTermScorer {
+    /// Creates a boxed term scorer from a concrete Tantivy [`TermScorer`].
+    pub fn new<TPostings: Postings>(term_scorer: TermScorer<TPostings>) -> BoxedTermScorer {
+        BoxedTermScorer(box_scorer(term_scorer))
+    }
+
+    /// Converts this boxed term scorer into a generic boxed scorer.
+    pub fn into_boxed_scorer(self) -> Box<dyn Scorer> {
+        self.0
+    }
+}
+
+/// Trait defining the contract for inverted index readers.
+pub trait InvertedIndexReader: Send + Sync {
+    /// Returns the term info associated with the term.
+    fn get_term_info(&self, term: &Term) -> io::Result<Option<TermInfo>> {
+        self.terms().get(term.serialized_value_bytes())
+    }
+
+    /// Return the term dictionary datastructure.
+    fn terms(&self) -> &TermDictionary;
+
+    /// Return the fields and types encoded in the dictionary in lexicographic order.
+    /// Only valid on JSON fields.
+    ///
+    /// Notice: This requires a full scan and therefore **very expensive**.
+    fn list_encoded_json_fields(&self) -> io::Result<Vec<InvertedIndexFieldSpace>>;
+
+    /// Build a new term scorer.
+    fn new_term_scorer(
+        &self,
+        term_info: &TermInfo,
+        option: IndexRecordOption,
+        fieldnorm_reader: FieldNormReader,
+        similarity_weight: Bm25Weight,
+    ) -> io::Result<BoxedTermScorer>;
+
+    /// Returns a posting object given a `term_info`.
+    /// This method is for an advanced usage only.
+    ///
+    /// Most users should prefer using [`Self::read_postings()`] instead.
+    fn read_postings_from_terminfo(
+        &self,
+        term_info: &TermInfo,
+        option: IndexRecordOption,
+    ) -> io::Result<Box<dyn Postings>>;
+
+    /// Returns the raw postings bytes and metadata for a term.
+    fn read_raw_postings_data(
+        &self,
+        term_info: &TermInfo,
+        option: IndexRecordOption,
+    ) -> io::Result<RawPostingsData>;
+
+    /// Fills a bitset with documents containing the term.
+    ///
+    /// Implementers can override this to avoid boxing postings.
+    fn fill_bitset_for_term(
+        &self,
+        term_info: &TermInfo,
+        option: IndexRecordOption,
+        doc_bitset: &mut BitSet,
+    ) -> io::Result<()> {
+        let mut postings = self.read_postings_from_terminfo(term_info, option)?;
+        postings.fill_bitset(doc_bitset);
+        Ok(())
+    }
+
+    /// Builds a phrase scorer for the given term infos.
+    fn new_phrase_scorer(
+        &self,
+        term_infos: &[(usize, TermInfo)],
+        similarity_weight: Option<Bm25Weight>,
+        fieldnorm_reader: FieldNormReader,
+        slop: u32,
+    ) -> io::Result<Box<dyn Scorer>>;
+
+    /// Returns the total number of tokens recorded for all documents
+    /// (including deleted documents).
+    fn total_num_tokens(&self) -> u64;
+
+    /// Returns the segment postings associated with the term, and with the given option,
+    /// or `None` if the term has never been encountered and indexed.
+    fn read_postings(
+        &self,
+        term: &Term,
+        option: IndexRecordOption,
+    ) -> io::Result<Option<Box<dyn Postings>>> {
+        self.get_term_info(term)?
+            .map(move |term_info| self.read_postings_from_terminfo(&term_info, option))
+            .transpose()
+    }
+
+    /// Returns the number of documents containing the term.
+    fn doc_freq(&self, term: &Term) -> io::Result<u32>;
+
+    /// Returns the number of documents containing the term asynchronously.
+    #[cfg(feature = "quickwit")]
+    fn doc_freq_async<'a>(
+        &'a self,
+        term: &'a Term,
+    ) -> Pin<Box<dyn Future<Output = io::Result<u32>> + Send + 'a>>;
+
+    /// Warmup fieldnorm readers for this inverted index field.
+    #[cfg(feature = "quickwit")]
+    fn warm_fieldnorms_readers<'a>(
+        &'a self,
+    ) -> Pin<Box<dyn Future<Output = io::Result<()>> + Send + 'a>>;
+
+    /// Warmup the block postings for all terms.
+    ///
+    /// Default implementation is a no-op.
+    #[cfg(feature = "quickwit")]
+    fn warm_postings_full<'a>(
+        &'a self,
+        _with_positions: bool,
+    ) -> Pin<Box<dyn Future<Output = io::Result<()>> + Send + 'a>> {
+        Box::pin(async { Ok(()) })
+    }
+
+    /// Warmup a block postings given a `Term`.
+    ///
+    /// Returns whether the term was found in the dictionary.
+    #[cfg(feature = "quickwit")]
+    fn warm_postings<'a>(
+        &'a self,
+        term: &'a Term,
+        with_positions: bool,
+    ) -> Pin<Box<dyn Future<Output = io::Result<bool>> + Send + 'a>>;
+
+    /// Warmup block postings for terms in a range.
+    ///
+    /// Returns whether at least one matching term was found.
+    #[cfg(feature = "quickwit")]
+    fn warm_postings_range<'a>(
+        &'a self,
+        terms: TermRangeBounds,
+        limit: Option<u64>,
+        with_positions: bool,
+    ) -> Pin<Box<dyn Future<Output = io::Result<bool>> + Send + 'a>>;
+
+    /// Warmup block postings for terms matching an automaton.
+    ///
+    /// Returns whether at least one matching term was found.
+    #[cfg(feature = "quickwit")]
+    fn warm_postings_automaton<'a, A: Automaton + Clone + Send + Sync + 'static>(
+        &'a self,
+        automaton: A,
+    ) -> Pin<Box<dyn Future<Output = io::Result<bool>> + Send + 'a>>
+    where
+        A::State: Clone + Send,
+        Self: Sized;
+}
+
+/// Tantivy's default inverted index reader implementation.
+///
 /// The inverted index reader is in charge of accessing
 /// the inverted index associated with a specific field.
 ///
 /// # Note
 ///
 /// It is safe to delete the segment associated with
-/// an `InvertedIndexReader`. As long as it is open,
+/// an `InvertedIndexReader` implementation. As long as it is open,
 /// the [`FileSlice`] it is relying on should
 /// stay available.
 ///
-/// `InvertedIndexReader` are created by calling
+/// `TantivyInvertedIndexReader` instances are created by calling
 /// [`SegmentReader::inverted_index()`](crate::SegmentReader::inverted_index).
-pub struct InvertedIndexReader {
+pub struct TantivyInvertedIndexReader {
     termdict: TermDictionary,
     postings_file_slice: FileSlice,
     positions_file_slice: FileSlice,
+    #[cfg_attr(not(feature = "quickwit"), allow(dead_code))]
+    fieldnorms_file_slice: FileSlice,
     record_option: IndexRecordOption,
     total_num_tokens: u64,
-    codec: Arc<dyn ObjectSafeCodec>,
 }
 
 /// Object that records the amount of space used by a field in an inverted index.
-pub(crate) struct InvertedIndexFieldSpace {
+pub struct InvertedIndexFieldSpace {
+    /// Field name as encoded in the term dictionary.
     pub field_name: String,
+    /// Value type for the encoded field.
     pub field_type: Type,
+    /// Total bytes used by postings for this field.
     pub postings_size: ByteCount,
+    /// Total bytes used by positions for this field.
     pub positions_size: ByteCount,
+    /// Number of terms in the field.
     pub num_terms: u64,
 }
 
@@ -68,55 +241,86 @@ impl InvertedIndexFieldSpace {
     }
 }
 
-impl InvertedIndexReader {
-    pub(crate) fn new(
-        termdict: TermDictionary,
-        postings_file_slice: FileSlice,
-        positions_file_slice: FileSlice,
-        record_option: IndexRecordOption,
-        codec: Arc<dyn ObjectSafeCodec>,
-    ) -> io::Result<InvertedIndexReader> {
-        let (total_num_tokens_slice, postings_body) = postings_file_slice.split(8);
-        let total_num_tokens = u64::deserialize(&mut total_num_tokens_slice.read_bytes()?)?;
-        Ok(InvertedIndexReader {
-            termdict,
-            postings_file_slice: postings_body,
-            positions_file_slice,
-            record_option,
-            total_num_tokens,
-            codec,
+impl TantivyInvertedIndexReader {
+    pub(crate) fn read_raw_postings_data_inner(
+        &self,
+        term_info: &TermInfo,
+        option: IndexRecordOption,
+    ) -> io::Result<RawPostingsData> {
+        let effective_option = option.downgrade(self.record_option);
+        let postings_data = self
+            .postings_file_slice
+            .slice(term_info.postings_range.clone())
+            .read_bytes()?;
+        let positions_data: Option<OwnedBytes> = if effective_option.has_positions() {
+            let positions_data = self
+                .positions_file_slice
+                .slice(term_info.positions_range.clone())
+                .read_bytes()?;
+            Some(positions_data)
+        } else {
+            None
+        };
+        Ok(RawPostingsData {
+            postings_data,
+            positions_data,
+            record_option: self.record_option,
+            effective_option,
         })
     }
 
-    /// Creates an empty `InvertedIndexReader` object, which
+    /// Opens an inverted index reader from already-loaded term/postings/positions slices.
+    ///
+    /// The first 8 bytes of `postings_file_slice` are expected to contain
+    /// the serialized total token count.
+    pub fn new(
+        termdict: TermDictionary,
+        postings_file_slice: FileSlice,
+        positions_file_slice: FileSlice,
+        fieldnorms_file_slice: FileSlice,
+        record_option: IndexRecordOption,
+    ) -> io::Result<TantivyInvertedIndexReader> {
+        let (total_num_tokens_slice, postings_body) = postings_file_slice.split(8);
+        let total_num_tokens = u64::deserialize(&mut total_num_tokens_slice.read_bytes()?)?;
+        Ok(TantivyInvertedIndexReader {
+            termdict,
+            postings_file_slice: postings_body,
+            positions_file_slice,
+            fieldnorms_file_slice,
+            record_option,
+            total_num_tokens,
+        })
+    }
+
+    /// Creates an empty `TantivyInvertedIndexReader` object, which
     /// contains no terms at all.
-    pub fn empty(record_option: IndexRecordOption) -> InvertedIndexReader {
-        InvertedIndexReader {
+    pub fn empty(record_option: IndexRecordOption) -> TantivyInvertedIndexReader {
+        TantivyInvertedIndexReader {
             termdict: TermDictionary::empty(),
             postings_file_slice: FileSlice::empty(),
             positions_file_slice: FileSlice::empty(),
+            fieldnorms_file_slice: FileSlice::empty(),
             record_option,
             total_num_tokens: 0u64,
-            codec: Arc::new(StandardCodec),
         }
     }
 
-    /// Returns the term info associated with the term.
-    pub fn get_term_info(&self, term: &Term) -> io::Result<Option<TermInfo>> {
-        self.termdict.get(term.serialized_value_bytes())
+    fn load_segment_postings(
+        &self,
+        term_info: &TermInfo,
+        option: IndexRecordOption,
+    ) -> io::Result<SegmentPostings> {
+        let postings_data = self.read_raw_postings_data_inner(term_info, option)?;
+        load_postings_from_raw_data(term_info.doc_freq, postings_data)
     }
+}
 
-    /// Return the term dictionary datastructure.
-    pub fn terms(&self) -> &TermDictionary {
+impl InvertedIndexReader for TantivyInvertedIndexReader {
+    fn terms(&self) -> &TermDictionary {
         &self.termdict
     }
 
-    /// Return the fields and types encoded in the dictionary in lexicographic order.
-    /// Only valid on JSON fields.
-    ///
-    /// Notice: This requires a full scan and therefore **very expensive**.
-    /// TODO: Move to sstable to use the index.
-    pub(crate) fn list_encoded_json_fields(&self) -> io::Result<Vec<InvertedIndexFieldSpace>> {
+    fn list_encoded_json_fields(&self) -> io::Result<Vec<InvertedIndexFieldSpace>> {
         let mut stream = self.termdict.stream()?;
         let mut fields: Vec<InvertedIndexFieldSpace> = Vec::new();
 
@@ -169,130 +373,73 @@ impl InvertedIndexReader {
         Ok(fields)
     }
 
-    pub(crate) fn new_term_scorer_specialized<C: Codec>(
+    fn new_term_scorer(
         &self,
         term_info: &TermInfo,
         option: IndexRecordOption,
         fieldnorm_reader: FieldNormReader,
         similarity_weight: Bm25Weight,
-        codec: &C,
-    ) -> io::Result<TermScorer<<<C as Codec>::PostingsCodec as PostingsCodec>::Postings>> {
-        let postings = self.read_postings_from_terminfo_specialized(term_info, option, codec)?;
+    ) -> io::Result<BoxedTermScorer> {
+        let postings = self.load_segment_postings(term_info, option)?;
         let term_scorer = TermScorer::new(postings, fieldnorm_reader, similarity_weight);
-        Ok(term_scorer)
+        Ok(BoxedTermScorer::new(term_scorer))
     }
 
-    pub(crate) fn new_phrase_scorer_type_specialized<C: Codec>(
-        &self,
-        term_infos: &[(usize, TermInfo)],
-        similarity_weight_opt: Option<Bm25Weight>,
-        fieldnorm_reader: FieldNormReader,
-        slop: u32,
-        codec: &C,
-    ) -> io::Result<PhraseScorer<<<C as Codec>::PostingsCodec as PostingsCodec>::Postings>> {
-        let mut offset_and_term_postings: Vec<(
-            usize,
-            <<C as Codec>::PostingsCodec as PostingsCodec>::Postings,
-        )> = Vec::with_capacity(term_infos.len());
-        for (offset, term_info) in term_infos {
-            let postings = self.read_postings_from_terminfo_specialized(
-                term_info,
-                IndexRecordOption::WithFreqsAndPositions,
-                codec,
-            )?;
-            offset_and_term_postings.push((*offset, postings));
-        }
-        let phrase_scorer = PhraseScorer::new(
-            offset_and_term_postings,
-            similarity_weight_opt,
-            fieldnorm_reader,
-            slop,
-        );
-        Ok(phrase_scorer)
-    }
-
-    /// Build a new term scorer.
-    pub fn new_term_scorer(
-        &self,
-        term_info: &TermInfo,
-        option: IndexRecordOption,
-        fieldnorm_reader: FieldNormReader,
-        similarity_weight: Bm25Weight,
-    ) -> io::Result<Box<dyn Scorer>> {
-        let term_scorer = self.codec.load_term_scorer_type_erased(
-            term_info,
-            option,
-            self,
-            fieldnorm_reader,
-            similarity_weight,
-        )?;
-        Ok(term_scorer)
-    }
-
-    /// Returns a postings object specific with a concrete type.
-    ///
-    /// This requires you to provied the actual codec.
-    pub fn read_postings_from_terminfo_specialized<C: Codec>(
-        &self,
-        term_info: &TermInfo,
-        option: IndexRecordOption,
-        codec: &C,
-    ) -> io::Result<<<C as Codec>::PostingsCodec as PostingsCodec>::Postings> {
-        let option = option.downgrade(self.record_option);
-        let postings_data = self
-            .postings_file_slice
-            .slice(term_info.postings_range.clone())
-            .read_bytes()?;
-        let positions_data: Option<OwnedBytes> = if option.has_positions() {
-            let positions_data = self
-                .positions_file_slice
-                .slice(term_info.positions_range.clone())
-                .read_bytes()?;
-            Some(positions_data)
-        } else {
-            None
-        };
-        let postings: <<C as Codec>::PostingsCodec as PostingsCodec>::Postings =
-            codec.postings_codec().load_postings(
-                term_info.doc_freq,
-                postings_data,
-                self.record_option,
-                option,
-                positions_data,
-            )?;
-        Ok(postings)
-    }
-
-    /// Returns a posting object given a `term_info`.
-    /// This method is for an advanced usage only.
-    ///
-    /// Most users should prefer using [`Self::read_postings()`] instead.
-    pub fn read_postings_from_terminfo(
+    fn read_postings_from_terminfo(
         &self,
         term_info: &TermInfo,
         option: IndexRecordOption,
     ) -> io::Result<Box<dyn Postings>> {
-        self.codec
-            .load_postings_type_erased(term_info, option, self)
+        let postings = self.load_segment_postings(term_info, option)?;
+        Ok(Box::new(postings))
     }
 
-    /// Returns the total number of tokens recorded for all documents
-    /// (including deleted documents).
-    pub fn total_num_tokens(&self) -> u64 {
+    fn read_raw_postings_data(
+        &self,
+        term_info: &TermInfo,
+        option: IndexRecordOption,
+    ) -> io::Result<RawPostingsData> {
+        self.read_raw_postings_data_inner(term_info, option)
+    }
+
+    fn fill_bitset_for_term(
+        &self,
+        term_info: &TermInfo,
+        option: IndexRecordOption,
+        doc_bitset: &mut BitSet,
+    ) -> io::Result<()> {
+        let postings_data = self.read_raw_postings_data_inner(term_info, option)?;
+        fill_bitset_from_raw_data(term_info.doc_freq, postings_data, doc_bitset)
+    }
+
+    fn new_phrase_scorer(
+        &self,
+        term_infos: &[(usize, TermInfo)],
+        similarity_weight: Option<Bm25Weight>,
+        fieldnorm_reader: FieldNormReader,
+        slop: u32,
+    ) -> io::Result<Box<dyn Scorer>> {
+        let mut offset_and_term_postings: Vec<(usize, SegmentPostings)> =
+            Vec::with_capacity(term_infos.len());
+        for (offset, term_info) in term_infos {
+            let postings =
+                self.load_segment_postings(term_info, IndexRecordOption::WithFreqsAndPositions)?;
+            offset_and_term_postings.push((*offset, postings));
+        }
+        let scorer = PhraseScorer::new(
+            offset_and_term_postings,
+            similarity_weight,
+            fieldnorm_reader,
+            slop,
+        );
+        Ok(box_scorer(scorer))
+    }
+
+    fn total_num_tokens(&self) -> u64 {
         self.total_num_tokens
     }
 
-    /// Returns the segment postings associated with the term, and with the given option,
-    /// or `None` if the term has never been encountered and indexed.
-    ///
-    /// If the field was not indexed with the indexing options that cover
-    /// the requested options, the returned [`SegmentPostings`] the method does not fail
-    /// and returns a `SegmentPostings` with as much information as possible.
-    ///
-    /// For instance, requesting [`IndexRecordOption::WithFreqs`] for a
-    /// [`TextOptions`](crate::schema::TextOptions) that does not index position
-    /// will return a [`SegmentPostings`] with `DocId`s and frequencies.
-    pub fn read_postings(
+    fn read_postings(
         &self,
         term: &Term,
         option: IndexRecordOption,
@@ -302,24 +449,184 @@ impl InvertedIndexReader {
             .transpose()
     }
 
-    /// Returns the number of documents containing the term.
-    pub fn doc_freq(&self, term: &Term) -> io::Result<u32> {
+    fn doc_freq(&self, term: &Term) -> io::Result<u32> {
         Ok(self
             .get_term_info(term)?
             .map(|term_info| term_info.doc_freq)
             .unwrap_or(0u32))
     }
+
+    #[cfg(feature = "quickwit")]
+    fn doc_freq_async<'a>(
+        &'a self,
+        term: &'a Term,
+    ) -> Pin<Box<dyn Future<Output = io::Result<u32>> + Send + 'a>> {
+        Box::pin(async move {
+            Ok(self
+                .get_term_info_async(term)
+                .await?
+                .map(|term_info| term_info.doc_freq)
+                .unwrap_or(0u32))
+        })
+    }
+
+    #[cfg(feature = "quickwit")]
+    fn warm_fieldnorms_readers<'a>(
+        &'a self,
+    ) -> Pin<Box<dyn Future<Output = io::Result<()>> + Send + 'a>> {
+        Box::pin(async move {
+            self.fieldnorms_file_slice.read_bytes_async().await?;
+            Ok(())
+        })
+    }
+
+    #[cfg(feature = "quickwit")]
+    fn warm_postings_full<'a>(
+        &'a self,
+        with_positions: bool,
+    ) -> Pin<Box<dyn Future<Output = io::Result<()>> + Send + 'a>> {
+        Box::pin(async move {
+            self.postings_file_slice.read_bytes_async().await?;
+            if with_positions {
+                self.positions_file_slice.read_bytes_async().await?;
+            }
+            Ok(())
+        })
+    }
+
+    #[cfg(feature = "quickwit")]
+    fn warm_postings<'a>(
+        &'a self,
+        term: &'a Term,
+        with_positions: bool,
+    ) -> Pin<Box<dyn Future<Output = io::Result<bool>> + Send + 'a>> {
+        Box::pin(async move {
+            let term_info_opt: Option<TermInfo> = self.get_term_info_async(term).await?;
+            if let Some(term_info) = term_info_opt {
+                let postings = self
+                    .postings_file_slice
+                    .read_bytes_slice_async(term_info.postings_range.clone());
+                if with_positions {
+                    let positions = self
+                        .positions_file_slice
+                        .read_bytes_slice_async(term_info.positions_range.clone());
+                    futures_util::future::try_join(postings, positions).await?;
+                } else {
+                    postings.await?;
+                }
+                Ok(true)
+            } else {
+                Ok(false)
+            }
+        })
+    }
+
+    #[cfg(feature = "quickwit")]
+    fn warm_postings_range<'a>(
+        &'a self,
+        terms: TermRangeBounds,
+        limit: Option<u64>,
+        with_positions: bool,
+    ) -> Pin<Box<dyn Future<Output = io::Result<bool>> + Send + 'a>> {
+        Box::pin(async move {
+            let mut term_info = self
+                .get_term_range_async(terms, AlwaysMatch, limit, 0)
+                .await?;
+
+            let Some(first_terminfo) = term_info.next() else {
+                // no key matches, nothing more to load
+                return Ok(false);
+            };
+
+            let last_terminfo = term_info.last().unwrap_or_else(|| first_terminfo.clone());
+
+            let postings_range =
+                first_terminfo.postings_range.start..last_terminfo.postings_range.end;
+            let positions_range =
+                first_terminfo.positions_range.start..last_terminfo.positions_range.end;
+
+            let postings = self
+                .postings_file_slice
+                .read_bytes_slice_async(postings_range);
+            if with_positions {
+                let positions = self
+                    .positions_file_slice
+                    .read_bytes_slice_async(positions_range);
+                futures_util::future::try_join(postings, positions).await?;
+            } else {
+                postings.await?;
+            }
+            Ok(true)
+        })
+    }
+
+    #[cfg(feature = "quickwit")]
+    fn warm_postings_automaton<'a, A: Automaton + Clone + Send + Sync + 'static>(
+        &'a self,
+        automaton: A,
+    ) -> Pin<Box<dyn Future<Output = io::Result<bool>> + Send + 'a>>
+    where
+        A::State: Clone + Send,
+        Self: Sized,
+    {
+        Box::pin(async move {
+            // merge holes under 4MiB, that's how many bytes we can hope to receive during a TTFB
+            // from S3 (~80MiB/s, and 50ms latency)
+            const MERGE_HOLES_UNDER_BYTES: usize = (80 * 1024 * 1024 * 50) / 1000;
+            // Trigger async prefetch of relevant termdict blocks.
+            let _term_info_iter = self
+                .get_term_range_async(
+                    (std::ops::Bound::Unbounded, std::ops::Bound::Unbounded),
+                    automaton.clone(),
+                    None,
+                    MERGE_HOLES_UNDER_BYTES,
+                )
+                .await?;
+            drop(_term_info_iter);
+
+            // Build a 2nd stream without merged holes so we only scan matching blocks.
+            // This assumes the storage layer caches data fetched by the first pass.
+            let mut stream = self.termdict.search(automaton).into_stream()?;
+            let posting_ranges_iter =
+                std::iter::from_fn(move || stream.next().map(|(_k, v)| v.postings_range.clone()));
+            let merged_posting_ranges: Vec<std::ops::Range<usize>> = posting_ranges_iter
+                .coalesce(|range1, range2| {
+                    if range1.end + MERGE_HOLES_UNDER_BYTES >= range2.start {
+                        Ok(range1.start..range2.end)
+                    } else {
+                        Err((range1, range2))
+                    }
+                })
+                .collect();
+
+            if merged_posting_ranges.is_empty() {
+                return Ok(false);
+            }
+
+            let slices_downloaded = futures_util::stream::iter(merged_posting_ranges.into_iter())
+                .map(|posting_slice| {
+                    self.postings_file_slice
+                        .read_bytes_slice_async(posting_slice)
+                        .map(|result| result.map(|_slice| ()))
+                })
+                .buffer_unordered(5)
+                .try_collect::<Vec<()>>()
+                .await?;
+
+            Ok(!slices_downloaded.is_empty())
+        })
+    }
 }
 
 #[cfg(feature = "quickwit")]
-impl InvertedIndexReader {
+impl TantivyInvertedIndexReader {
     pub(crate) async fn get_term_info_async(&self, term: &Term) -> io::Result<Option<TermInfo>> {
         self.termdict.get_async(term.serialized_value_bytes()).await
     }
 
     async fn get_term_range_async<'a, A: Automaton + 'a>(
         &'a self,
-        terms: impl std::ops::RangeBounds<Term>,
+        terms: TermRangeBounds,
         automaton: A,
         limit: Option<u64>,
         merge_holes_under_bytes: usize,
@@ -327,17 +634,17 @@ impl InvertedIndexReader {
     where
         A::State: Clone,
     {
-        use std::ops::Bound;
         let range_builder = self.termdict.search(automaton);
-        let range_builder = match terms.start_bound() {
-            Bound::Included(bound) => range_builder.ge(bound.serialized_value_bytes()),
-            Bound::Excluded(bound) => range_builder.gt(bound.serialized_value_bytes()),
-            Bound::Unbounded => range_builder,
+        let (start_bound, end_bound) = terms;
+        let range_builder = match start_bound {
+            std::ops::Bound::Included(bound) => range_builder.ge(bound.serialized_value_bytes()),
+            std::ops::Bound::Excluded(bound) => range_builder.gt(bound.serialized_value_bytes()),
+            std::ops::Bound::Unbounded => range_builder,
         };
-        let range_builder = match terms.end_bound() {
-            Bound::Included(bound) => range_builder.le(bound.serialized_value_bytes()),
-            Bound::Excluded(bound) => range_builder.lt(bound.serialized_value_bytes()),
-            Bound::Unbounded => range_builder,
+        let range_builder = match end_bound {
+            std::ops::Bound::Included(bound) => range_builder.le(bound.serialized_value_bytes()),
+            std::ops::Bound::Excluded(bound) => range_builder.lt(bound.serialized_value_bytes()),
+            std::ops::Bound::Unbounded => range_builder,
         };
         let range_builder = if let Some(limit) = limit {
             range_builder.limit(limit)
@@ -358,167 +665,4 @@ impl InvertedIndexReader {
 
         Ok(iter)
     }
-
-    /// Warmup a block postings given a `Term`.
-    /// This method is for an advanced usage only.
-    ///
-    /// returns a boolean, whether the term was found in the dictionary
-    pub async fn warm_postings(&self, term: &Term, with_positions: bool) -> io::Result<bool> {
-        let term_info_opt: Option<TermInfo> = self.get_term_info_async(term).await?;
-        if let Some(term_info) = term_info_opt {
-            let postings = self
-                .postings_file_slice
-                .read_bytes_slice_async(term_info.postings_range.clone());
-            if with_positions {
-                let positions = self
-                    .positions_file_slice
-                    .read_bytes_slice_async(term_info.positions_range.clone());
-                futures_util::future::try_join(postings, positions).await?;
-            } else {
-                postings.await?;
-            }
-            Ok(true)
-        } else {
-            Ok(false)
-        }
-    }
-
-    /// Warmup a block postings given a range of `Term`s.
-    /// This method is for an advanced usage only.
-    ///
-    /// returns a boolean, whether a term matching the range was found in the dictionary
-    pub async fn warm_postings_range(
-        &self,
-        terms: impl std::ops::RangeBounds<Term>,
-        limit: Option<u64>,
-        with_positions: bool,
-    ) -> io::Result<bool> {
-        let mut term_info = self
-            .get_term_range_async(terms, AlwaysMatch, limit, 0)
-            .await?;
-
-        let Some(first_terminfo) = term_info.next() else {
-            // no key matches, nothing more to load
-            return Ok(false);
-        };
-
-        let last_terminfo = term_info.last().unwrap_or_else(|| first_terminfo.clone());
-
-        let postings_range = first_terminfo.postings_range.start..last_terminfo.postings_range.end;
-        let positions_range =
-            first_terminfo.positions_range.start..last_terminfo.positions_range.end;
-
-        let postings = self
-            .postings_file_slice
-            .read_bytes_slice_async(postings_range);
-        if with_positions {
-            let positions = self
-                .positions_file_slice
-                .read_bytes_slice_async(positions_range);
-            futures_util::future::try_join(postings, positions).await?;
-        } else {
-            postings.await?;
-        }
-        Ok(true)
-    }
-
-    /// Warmup a block postings given a range of `Term`s.
-    /// This method is for an advanced usage only.
-    ///
-    /// returns a boolean, whether a term matching the range was found in the dictionary
-    pub async fn warm_postings_automaton<
-        A: Automaton + Clone + Send + 'static,
-        E: FnOnce(Box<dyn FnOnce() -> io::Result<()> + Send>) -> F,
-        F: std::future::Future<Output = io::Result<()>>,
-    >(
-        &self,
-        automaton: A,
-        // with_positions: bool, at the moment we have no use for it, and supporting it would add
-        // complexity to the coalesce
-        executor: E,
-    ) -> io::Result<bool>
-    where
-        A::State: Clone,
-    {
-        // merge holes under 4MiB, that's how many bytes we can hope to receive during a TTFB from
-        // S3 (~80MiB/s, and 50ms latency)
-        const MERGE_HOLES_UNDER_BYTES: usize = (80 * 1024 * 1024 * 50) / 1000;
-        // we build a first iterator to download everything. Simply calling the function already
-        // download everything we need from the sstable, but doesn't start iterating over it.
-        let _term_info_iter = self
-            .get_term_range_async(.., automaton.clone(), None, MERGE_HOLES_UNDER_BYTES)
-            .await?;
-
-        let (sender, posting_ranges_to_load_stream) = futures_channel::mpsc::unbounded();
-        let termdict = self.termdict.clone();
-        let cpu_bound_task = move || {
-            // then we build a 2nd iterator, this one with no holes, so we don't go through blocks
-            // we can't match.
-            // This makes the assumption there is a caching layer below us, which gives sync read
-            // for free after the initial async access. This might not always be true, but is in
-            // Quickwit.
-            // We build things from this closure otherwise we get into lifetime issues that can only
-            // be solved with self referential strucs. Returning an io::Result from here is a bit
-            // more leaky abstraction-wise, but a lot better than the alternative
-            let mut stream = termdict.search(automaton).into_stream()?;
-
-            // we could do without an iterator, but this allows us access to coalesce which simplify
-            // things
-            let posting_ranges_iter =
-                std::iter::from_fn(move || stream.next().map(|(_k, v)| v.postings_range.clone()));
-
-            let merged_posting_ranges_iter = posting_ranges_iter.coalesce(|range1, range2| {
-                if range1.end + MERGE_HOLES_UNDER_BYTES >= range2.start {
-                    Ok(range1.start..range2.end)
-                } else {
-                    Err((range1, range2))
-                }
-            });
-
-            for posting_range in merged_posting_ranges_iter {
-                if let Err(_) = sender.unbounded_send(posting_range) {
-                    // this should happen only when search is cancelled
-                    return Err(io::Error::other("failed to send posting range back"));
-                }
-            }
-            Ok(())
-        };
-        let task_handle = executor(Box::new(cpu_bound_task));
-
-        let posting_downloader = posting_ranges_to_load_stream
-            .map(|posting_slice| {
-                self.postings_file_slice
-                    .read_bytes_slice_async(posting_slice)
-                    .map(|result| result.map(|_slice| ()))
-            })
-            .buffer_unordered(5)
-            .try_collect::<Vec<()>>();
-
-        let (_, slices_downloaded) =
-            futures_util::future::try_join(task_handle, posting_downloader).await?;
-
-        Ok(!slices_downloaded.is_empty())
-    }
-
-    /// Warmup the block postings for all terms.
-    /// This method is for an advanced usage only.
-    ///
-    /// If you know which terms to pre-load, prefer using [`Self::warm_postings`] or
-    /// [`Self::warm_postings`] instead.
-    pub async fn warm_postings_full(&self, with_positions: bool) -> io::Result<()> {
-        self.postings_file_slice.read_bytes_async().await?;
-        if with_positions {
-            self.positions_file_slice.read_bytes_async().await?;
-        }
-        Ok(())
-    }
-
-    /// Returns the number of documents containing the term asynchronously.
-    pub async fn doc_freq_async(&self, term: &Term) -> io::Result<u32> {
-        Ok(self
-            .get_term_info_async(term)
-            .await?
-            .map(|term_info| term_info.doc_freq)
-            .unwrap_or(0u32))
-    }
 }

src/index/mod.rs

@@ -15,8 +15,10 @@ pub use self::codec_configuration::CodecConfiguration;
 pub use self::index::{Index, IndexBuilder};
 pub(crate) use self::index_meta::SegmentMetaInventory;
 pub use self::index_meta::{IndexMeta, IndexSettings, Order, SegmentMeta};
-pub use self::inverted_index_reader::InvertedIndexReader;
+pub use self::inverted_index_reader::{
+    BoxedTermScorer, InvertedIndexFieldSpace, InvertedIndexReader, TantivyInvertedIndexReader,
+};
 pub use self::segment::Segment;
 pub use self::segment_component::SegmentComponent;
 pub use self::segment_id::SegmentId;
-pub use self::segment_reader::{FieldMetadata, SegmentReader};
+pub use self::segment_reader::{FieldMetadata, SegmentReader, TantivySegmentReader};

src/index/segment_id.rs

@@ -44,7 +44,7 @@ fn create_uuid() -> Uuid {
 }
 
 impl SegmentId {
-    #[doc(hidden)]
+    /// Generates a new random `SegmentId`.
     pub fn generate_random() -> SegmentId {
         SegmentId(create_uuid())
     }

src/index/segment_reader.rs

@@ -6,18 +6,107 @@ use common::{ByteCount, HasLen};
 use fnv::FnvHashMap;
 use itertools::Itertools;
 
-use crate::codec::ObjectSafeCodec;
-use crate::directory::{CompositeFile, FileSlice};
+use crate::codec::{ObjectSafeCodec, SumOrDoNothingCombiner};
+use crate::directory::{CompositeFile, Directory, FileSlice};
 use crate::error::DataCorruption;
 use crate::fastfield::{intersect_alive_bitsets, AliveBitSet, FacetReader, FastFieldReaders};
 use crate::fieldnorm::{FieldNormReader, FieldNormReaders};
-use crate::index::{InvertedIndexReader, Segment, SegmentComponent, SegmentId};
+use crate::index::{
+    InvertedIndexReader, Segment, SegmentComponent, SegmentId, SegmentMeta,
+    TantivyInvertedIndexReader,
+};
 use crate::json_utils::json_path_sep_to_dot;
+use crate::query::Scorer;
 use crate::schema::{Field, IndexRecordOption, Schema, Type};
 use crate::space_usage::SegmentSpaceUsage;
-use crate::store::StoreReader;
+use crate::store::{StoreReader, TantivyStoreReader};
 use crate::termdict::TermDictionary;
-use crate::{DocId, Opstamp};
+use crate::{DocId, Opstamp, Score};
+
+/// Trait defining the contract for a segment reader.
+pub trait SegmentReader: Send + Sync {
+    /// Returns the highest document id ever attributed in this segment + 1.
+    fn max_doc(&self) -> DocId;
+
+    /// Returns the number of alive documents. Deleted documents are not counted.
+    fn num_docs(&self) -> DocId;
+
+    /// Returns the schema of the index this segment belongs to.
+    fn schema(&self) -> &Schema;
+
+    /// Performs a for_each_pruning operation on the given scorer.
+    fn for_each_pruning(
+        &self,
+        threshold: Score,
+        scorer: Box<dyn Scorer>,
+        callback: &mut dyn FnMut(DocId, Score) -> Score,
+    );
+
+    /// Builds a union scorer possibly specialized if all scorers are term scorers.
+    fn build_union_scorer_with_sum_combiner(
+        &self,
+        scorers: Vec<Box<dyn Scorer>>,
+        num_docs: DocId,
+        score_combiner_type: SumOrDoNothingCombiner,
+    ) -> Box<dyn Scorer>;
+
+    /// Return the number of documents that have been deleted in the segment.
+    fn num_deleted_docs(&self) -> DocId;
+
+    /// Returns true if some of the documents of the segment have been deleted.
+    fn has_deletes(&self) -> bool;
+
+    /// Accessor to a segment's fast field reader given a field.
+    fn fast_fields(&self) -> &FastFieldReaders;
+
+    /// Accessor to the `FacetReader` associated with a given `Field`.
+    fn facet_reader(&self, field_name: &str) -> crate::Result<FacetReader> {
+        let field = self.schema().get_field(field_name)?;
+        let field_entry = self.schema().get_field_entry(field);
+        if field_entry.field_type().value_type() != Type::Facet {
+            return Err(crate::TantivyError::SchemaError(format!(
+                "`{field_name}` is not a facet field.`"
+            )));
+        }
+        let Some(facet_column) = self.fast_fields().str(field_name)? else {
+            panic!("Facet Field `{field_name}` is missing. This should not happen");
+        };
+        Ok(FacetReader::new(facet_column))
+    }
+
+    /// Accessor to the segment's `Field norms`'s reader.
+    fn get_fieldnorms_reader(&self, field: Field) -> crate::Result<FieldNormReader>;
+
+    /// Accessor to the segment's [`StoreReader`](crate::store::StoreReader).
+    fn get_store_reader(&self, cache_num_blocks: usize) -> io::Result<Box<dyn StoreReader>>;
+
+    /// Returns a field reader associated with the field given in argument.
+    fn inverted_index(&self, field: Field) -> crate::Result<Arc<dyn InvertedIndexReader>>;
+
+    /// Returns the list of fields that have been indexed in the segment.
+    fn fields_metadata(&self) -> crate::Result<Vec<FieldMetadata>>;
+
+    /// Returns the segment id.
+    fn segment_id(&self) -> SegmentId;
+
+    /// Returns the delete opstamp.
+    fn delete_opstamp(&self) -> Option<Opstamp>;
+
+    /// Returns the bitset representing the alive `DocId`s.
+    fn alive_bitset(&self) -> Option<&AliveBitSet>;
+
+    /// Returns true if the `doc` is marked as deleted.
+    fn is_deleted(&self, doc: DocId) -> bool;
+
+    /// Returns an iterator that will iterate over the alive document ids.
+    fn doc_ids_alive(&self) -> Box<dyn Iterator<Item = DocId> + Send + '_>;
+
+    /// Summarize total space usage of this segment.
+    fn space_usage(&self) -> io::Result<SegmentSpaceUsage>;
+
+    /// Clones this reader into a shared trait object.
+    fn clone_arc(&self) -> Arc<dyn SegmentReader>;
+}
 
 /// Entry point to access all of the datastructures of the `Segment`
 ///
@@ -30,8 +119,8 @@ use crate::{DocId, Opstamp};
 /// The segment reader has a very low memory footprint,
 /// as close to all of the memory data is mmapped.
 #[derive(Clone)]
-pub struct SegmentReader {
-    inv_idx_reader_cache: Arc<RwLock<HashMap<Field, Arc<InvertedIndexReader>>>>,
+pub struct TantivySegmentReader {
+    inv_idx_reader_cache: Arc<RwLock<HashMap<Field, Arc<dyn InvertedIndexReader>>>>,
 
     segment_id: SegmentId,
     delete_opstamp: Option<Opstamp>,
@@ -51,78 +140,145 @@ pub struct SegmentReader {
     codec: Arc<dyn ObjectSafeCodec>,
 }
 
-impl SegmentReader {
-    /// Returns the highest document id ever attributed in
-    /// this segment + 1.
-    pub fn max_doc(&self) -> DocId {
+impl TantivySegmentReader {
+    /// Open a new segment for reading.
+    pub fn open<C: crate::codec::Codec>(
+        segment: &Segment<C>,
+    ) -> crate::Result<Arc<dyn SegmentReader>> {
+        Self::open_with_custom_alive_set(segment, None)
+    }
+
+    /// Open a new segment for reading.
+    pub fn open_with_custom_alive_set<C: crate::codec::Codec>(
+        segment: &Segment<C>,
+        custom_bitset: Option<AliveBitSet>,
+    ) -> crate::Result<Arc<dyn SegmentReader>> {
+        segment.index().codec().open_segment_reader(
+            segment.index().directory(),
+            segment.meta(),
+            segment.schema(),
+            custom_bitset,
+        )
+    }
+
+    pub(crate) fn open_with_custom_alive_set_from_directory(
+        directory: &dyn Directory,
+        segment_meta: &SegmentMeta,
+        schema: Schema,
+        codec: Arc<dyn ObjectSafeCodec>,
+        custom_bitset: Option<AliveBitSet>,
+    ) -> crate::Result<TantivySegmentReader> {
+        let termdict_file =
+            directory.open_read(&segment_meta.relative_path(SegmentComponent::Terms))?;
+        let termdict_composite = CompositeFile::open(&termdict_file)?;
+
+        let store_file =
+            directory.open_read(&segment_meta.relative_path(SegmentComponent::Store))?;
+
+        crate::fail_point!("SegmentReader::open#middle");
+
+        let postings_file =
+            directory.open_read(&segment_meta.relative_path(SegmentComponent::Postings))?;
+        let postings_composite = CompositeFile::open(&postings_file)?;
+
+        let positions_composite = {
+            if let Ok(positions_file) =
+                directory.open_read(&segment_meta.relative_path(SegmentComponent::Positions))
+            {
+                CompositeFile::open(&positions_file)?
+            } else {
+                CompositeFile::empty()
+            }
+        };
+
+        let fast_fields_data =
+            directory.open_read(&segment_meta.relative_path(SegmentComponent::FastFields))?;
+        let fast_fields_readers = FastFieldReaders::open(fast_fields_data, schema.clone())?;
+        let fieldnorm_data =
+            directory.open_read(&segment_meta.relative_path(SegmentComponent::FieldNorms))?;
+        let fieldnorm_readers = FieldNormReaders::open(fieldnorm_data)?;
+
+        let original_bitset = if segment_meta.has_deletes() {
+            let alive_doc_file_slice =
+                directory.open_read(&segment_meta.relative_path(SegmentComponent::Delete))?;
+            let alive_doc_data = alive_doc_file_slice.read_bytes()?;
+            Some(AliveBitSet::open(alive_doc_data))
+        } else {
+            None
+        };
+
+        let alive_bitset_opt = intersect_alive_bitset(original_bitset, custom_bitset);
+
+        let max_doc = segment_meta.max_doc();
+        let num_docs = alive_bitset_opt
+            .as_ref()
+            .map(|alive_bitset| alive_bitset.num_alive_docs() as u32)
+            .unwrap_or(max_doc);
+
+        Ok(TantivySegmentReader {
+            inv_idx_reader_cache: Default::default(),
+            num_docs,
+            max_doc,
+            termdict_composite,
+            postings_composite,
+            fast_fields_readers,
+            fieldnorm_readers,
+            segment_id: segment_meta.id(),
+            delete_opstamp: segment_meta.delete_opstamp(),
+            store_file,
+            alive_bitset_opt,
+            positions_composite,
+            schema,
+            codec,
+        })
+    }
+}
+
+impl SegmentReader for TantivySegmentReader {
+    fn max_doc(&self) -> DocId {
         self.max_doc
     }
 
-    /// Returns the number of alive documents.
-    /// Deleted documents are not counted.
-    pub fn num_docs(&self) -> DocId {
+    fn num_docs(&self) -> DocId {
         self.num_docs
     }
 
-    /// Returns the schema of the index this segment belongs to.
-    pub fn schema(&self) -> &Schema {
+    fn schema(&self) -> &Schema {
         &self.schema
     }
 
-    /// Returns the index codec.
-    pub fn codec(&self) -> &dyn ObjectSafeCodec {
-        &*self.codec
+    fn for_each_pruning(
+        &self,
+        threshold: Score,
+        scorer: Box<dyn Scorer>,
+        callback: &mut dyn FnMut(DocId, Score) -> Score,
+    ) {
+        self.codec.for_each_pruning(threshold, scorer, callback);
     }
 
-    /// Return the number of documents that have been
-    /// deleted in the segment.
-    pub fn num_deleted_docs(&self) -> DocId {
+    fn build_union_scorer_with_sum_combiner(
+        &self,
+        scorers: Vec<Box<dyn Scorer>>,
+        num_docs: DocId,
+        score_combiner_type: SumOrDoNothingCombiner,
+    ) -> Box<dyn Scorer> {
+        self.codec
+            .build_union_scorer_with_sum_combiner(scorers, num_docs, score_combiner_type)
+    }
+
+    fn num_deleted_docs(&self) -> DocId {
         self.max_doc - self.num_docs
     }
 
-    /// Returns true if some of the documents of the segment have been deleted.
-    pub fn has_deletes(&self) -> bool {
-        self.num_deleted_docs() > 0
+    fn has_deletes(&self) -> bool {
+        self.num_docs != self.max_doc
     }
 
-    /// Accessor to a segment's fast field reader given a field.
-    ///
-    /// Returns the u64 fast value reader if the field
-    /// is a u64 field indexed as "fast".
-    ///
-    /// Return a FastFieldNotAvailableError if the field is not
-    /// declared as a fast field in the schema.
-    ///
-    /// # Panics
-    /// May panic if the index is corrupted.
-    pub fn fast_fields(&self) -> &FastFieldReaders {
+    fn fast_fields(&self) -> &FastFieldReaders {
         &self.fast_fields_readers
     }
 
-    /// Accessor to the `FacetReader` associated with a given `Field`.
-    pub fn facet_reader(&self, field_name: &str) -> crate::Result<FacetReader> {
-        let schema = self.schema();
-        let field = schema.get_field(field_name)?;
-        let field_entry = schema.get_field_entry(field);
-        if field_entry.field_type().value_type() != Type::Facet {
-            return Err(crate::TantivyError::SchemaError(format!(
-                "`{field_name}` is not a facet field.`"
-            )));
-        }
-        let Some(facet_column) = self.fast_fields().str(field_name)? else {
-            panic!("Facet Field `{field_name}` is missing. This should not happen");
-        };
-        Ok(FacetReader::new(facet_column))
-    }
-
-    /// Accessor to the segment's `Field norms`'s reader.
-    ///
-    /// Field norms are the length (in tokens) of the fields.
-    /// It is used in the computation of the [TfIdf](https://fulmicoton.gitbooks.io/tantivy-doc/content/tfidf.html).
-    ///
-    /// They are simply stored as a fast field, serialized in
-    /// the `.fieldnorm` file of the segment.
-    pub fn get_fieldnorms_reader(&self, field: Field) -> crate::Result<FieldNormReader> {
+    fn get_fieldnorms_reader(&self, field: Field) -> crate::Result<FieldNormReader> {
         self.fieldnorm_readers.get_field(field)?.ok_or_else(|| {
             let field_name = self.schema.get_field_name(field);
             let err_msg = format!(
@@ -133,102 +289,14 @@ impl SegmentReader {
         })
     }
 
-    #[doc(hidden)]
-    pub fn fieldnorms_readers(&self) -> &FieldNormReaders {
-        &self.fieldnorm_readers
+    fn get_store_reader(&self, cache_num_blocks: usize) -> io::Result<Box<dyn StoreReader>> {
+        Ok(Box::new(TantivyStoreReader::open(
+            self.store_file.clone(),
+            cache_num_blocks,
+        )?))
     }
 
-    /// Accessor to the segment's [`StoreReader`](crate::store::StoreReader).
-    ///
-    /// `cache_num_blocks` sets the number of decompressed blocks to be cached in an LRU.
-    /// The size of blocks is configurable, this should be reflexted in the
-    pub fn get_store_reader(&self, cache_num_blocks: usize) -> io::Result<StoreReader> {
-        StoreReader::open(self.store_file.clone(), cache_num_blocks)
-    }
-
-    /// Open a new segment for reading.
-    pub fn open<C: crate::codec::Codec>(segment: &Segment<C>) -> crate::Result<SegmentReader> {
-        Self::open_with_custom_alive_set(segment, None)
-    }
-
-    /// Open a new segment for reading.
-    pub fn open_with_custom_alive_set<C: crate::codec::Codec>(
-        segment: &Segment<C>,
-        custom_bitset: Option<AliveBitSet>,
-    ) -> crate::Result<SegmentReader> {
-        let codec: Arc<dyn ObjectSafeCodec> = Arc::new(segment.index().codec().clone());
-        let termdict_file = segment.open_read(SegmentComponent::Terms)?;
-        let termdict_composite = CompositeFile::open(&termdict_file)?;
-
-        let store_file = segment.open_read(SegmentComponent::Store)?;
-
-        crate::fail_point!("SegmentReader::open#middle");
-
-        let postings_file = segment.open_read(SegmentComponent::Postings)?;
-        let postings_composite = CompositeFile::open(&postings_file)?;
-
-        let positions_composite = {
-            if let Ok(positions_file) = segment.open_read(SegmentComponent::Positions) {
-                CompositeFile::open(&positions_file)?
-            } else {
-                CompositeFile::empty()
-            }
-        };
-
-        let schema = segment.schema();
-
-        let fast_fields_data = segment.open_read(SegmentComponent::FastFields)?;
-        let fast_fields_readers = FastFieldReaders::open(fast_fields_data, schema.clone())?;
-        let fieldnorm_data = segment.open_read(SegmentComponent::FieldNorms)?;
-        let fieldnorm_readers = FieldNormReaders::open(fieldnorm_data)?;
-
-        let original_bitset = if segment.meta().has_deletes() {
-            let alive_doc_file_slice = segment.open_read(SegmentComponent::Delete)?;
-            let alive_doc_data = alive_doc_file_slice.read_bytes()?;
-            Some(AliveBitSet::open(alive_doc_data))
-        } else {
-            None
-        };
-
-        let alive_bitset_opt = intersect_alive_bitset(original_bitset, custom_bitset);
-
-        let max_doc = segment.meta().max_doc();
-        let num_docs = alive_bitset_opt
-            .as_ref()
-            .map(|alive_bitset| alive_bitset.num_alive_docs() as u32)
-            .unwrap_or(max_doc);
-
-        Ok(SegmentReader {
-            inv_idx_reader_cache: Default::default(),
-            num_docs,
-            max_doc,
-            termdict_composite,
-            postings_composite,
-            fast_fields_readers,
-            fieldnorm_readers,
-            segment_id: segment.id(),
-            delete_opstamp: segment.meta().delete_opstamp(),
-            store_file,
-            alive_bitset_opt,
-            positions_composite,
-            schema,
-            codec,
-        })
-    }
-
-    /// Returns a field reader associated with the field given in argument.
-    /// If the field was not present in the index during indexing time,
-    /// the InvertedIndexReader is empty.
-    ///
-    /// The field reader is in charge of iterating through the
-    /// term dictionary associated with a specific field,
-    /// and opening the posting list associated with any term.
-    ///
-    /// If the field is not marked as index, a warning is logged and an empty `InvertedIndexReader`
-    /// is returned.
-    /// Similarly, if the field is marked as indexed but no term has been indexed for the given
-    /// index, an empty `InvertedIndexReader` is returned (but no warning is logged).
-    pub fn inverted_index(&self, field: Field) -> crate::Result<Arc<InvertedIndexReader>> {
+    fn inverted_index(&self, field: Field) -> crate::Result<Arc<dyn InvertedIndexReader>> {
         if let Some(inv_idx_reader) = self
             .inv_idx_reader_cache
             .read()
@@ -253,7 +321,9 @@ impl SegmentReader {
             //
             // Returns an empty inverted index.
             let record_option = record_option_opt.unwrap_or(IndexRecordOption::Basic);
-            return Ok(Arc::new(InvertedIndexReader::empty(record_option)));
+            let inv_idx_reader: Arc<dyn InvertedIndexReader> =
+                Arc::new(TantivyInvertedIndexReader::empty(record_option));
+            return Ok(inv_idx_reader);
         }
 
         let record_option = record_option_opt.unwrap();
@@ -276,14 +346,20 @@ impl SegmentReader {
             );
             DataCorruption::comment_only(error_msg)
         })?;
+        let fieldnorms_file = self
+            .fieldnorm_readers
+            .get_inner_file()
+            .open_read(field)
+            .unwrap_or_else(FileSlice::empty);
 
-        let inv_idx_reader = Arc::new(InvertedIndexReader::new(
-            TermDictionary::open(termdict_file)?,
-            postings_file,
-            positions_file,
-            record_option,
-            self.codec.clone(),
-        )?);
+        let inv_idx_reader: Arc<dyn InvertedIndexReader> =
+            Arc::new(TantivyInvertedIndexReader::new(
+                TermDictionary::open(termdict_file)?,
+                postings_file,
+                positions_file,
+                fieldnorms_file,
+                record_option,
+            )?);
 
         // by releasing the lock in between, we may end up opening the inverting index
         // twice, but this is fine.
@@ -295,23 +371,10 @@ impl SegmentReader {
         Ok(inv_idx_reader)
     }
 
-    /// Returns the list of fields that have been indexed in the segment.
-    /// The field list includes the field defined in the schema as well as the fields
-    /// that have been indexed as a part of a JSON field.
-    /// The returned field name is the full field name, including the name of the JSON field.
-    ///
-    /// The returned field names can be used in queries.
-    ///
-    /// Notice: If your data contains JSON fields this is **very expensive**, as it requires
-    /// browsing through the inverted index term dictionary and the columnar field dictionary.
-    ///
-    /// Disclaimer: Some fields may not be listed here. For instance, if the schema contains a json
-    /// field that is not indexed nor a fast field but is stored, it is possible for the field
-    /// to not be listed.
-    pub fn fields_metadata(&self) -> crate::Result<Vec<FieldMetadata>> {
+    fn fields_metadata(&self) -> crate::Result<Vec<FieldMetadata>> {
         let mut indexed_fields: Vec<FieldMetadata> = Vec::new();
         let mut map_to_canonical = FnvHashMap::default();
-        for (field, field_entry) in self.schema().fields() {
+        for (field, field_entry) in self.schema.fields() {
             let field_name = field_entry.name().to_string();
             let is_indexed = field_entry.is_indexed();
             if is_indexed {
@@ -401,7 +464,7 @@ impl SegmentReader {
             }
         }
         let fast_fields: Vec<FieldMetadata> = self
-            .fast_fields()
+            .fast_fields_readers
             .columnar()
             .iter_columns()?
             .map(|(mut field_name, handle)| {
@@ -429,31 +492,26 @@ impl SegmentReader {
         Ok(merged_field_metadatas)
     }
 
-    /// Returns the segment id
-    pub fn segment_id(&self) -> SegmentId {
+    fn segment_id(&self) -> SegmentId {
         self.segment_id
     }
 
-    /// Returns the delete opstamp
-    pub fn delete_opstamp(&self) -> Option<Opstamp> {
+    fn delete_opstamp(&self) -> Option<Opstamp> {
         self.delete_opstamp
     }
 
-    /// Returns the bitset representing the alive `DocId`s.
-    pub fn alive_bitset(&self) -> Option<&AliveBitSet> {
+    fn alive_bitset(&self) -> Option<&AliveBitSet> {
         self.alive_bitset_opt.as_ref()
     }
 
-    /// Returns true if the `doc` is marked
-    /// as deleted.
-    pub fn is_deleted(&self, doc: DocId) -> bool {
-        self.alive_bitset()
+    fn is_deleted(&self, doc: DocId) -> bool {
+        self.alive_bitset_opt
+            .as_ref()
             .map(|alive_bitset| alive_bitset.is_deleted(doc))
             .unwrap_or(false)
     }
 
-    /// Returns an iterator that will iterate over the alive document ids
-    pub fn doc_ids_alive(&self) -> Box<dyn Iterator<Item = DocId> + Send + '_> {
+    fn doc_ids_alive(&self) -> Box<dyn Iterator<Item = DocId> + Send + '_> {
         if let Some(alive_bitset) = &self.alive_bitset_opt {
             Box::new(alive_bitset.iter_alive())
         } else {
@@ -461,22 +519,25 @@ impl SegmentReader {
         }
     }
 
-    /// Summarize total space usage of this segment.
-    pub fn space_usage(&self) -> io::Result<SegmentSpaceUsage> {
+    fn space_usage(&self) -> io::Result<SegmentSpaceUsage> {
         Ok(SegmentSpaceUsage::new(
-            self.num_docs(),
-            self.termdict_composite.space_usage(self.schema()),
-            self.postings_composite.space_usage(self.schema()),
-            self.positions_composite.space_usage(self.schema()),
+            self.num_docs,
+            self.termdict_composite.space_usage(&self.schema),
+            self.postings_composite.space_usage(&self.schema),
+            self.positions_composite.space_usage(&self.schema),
             self.fast_fields_readers.space_usage()?,
-            self.fieldnorm_readers.space_usage(self.schema()),
-            self.get_store_reader(0)?.space_usage(),
+            self.fieldnorm_readers.space_usage(&self.schema),
+            TantivyStoreReader::open(self.store_file.clone(), 0)?.space_usage(),
             self.alive_bitset_opt
                 .as_ref()
                 .map(AliveBitSet::space_usage)
                 .unwrap_or_default(),
         ))
     }
+
+    fn clone_arc(&self) -> Arc<dyn SegmentReader> {
+        Arc::new(self.clone())
+    }
 }
 
 #[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)]
@@ -586,7 +647,7 @@ fn intersect_alive_bitset(
     }
 }
 
-impl fmt::Debug for SegmentReader {
+impl fmt::Debug for TantivySegmentReader {
     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
         write!(f, "SegmentReader({:?})", self.segment_id)
     }

src/indexer/delete_queue.rs

@@ -250,11 +250,15 @@ mod tests {
 
     struct DummyWeight;
     impl Weight for DummyWeight {
-        fn scorer(&self, _reader: &SegmentReader, _boost: Score) -> crate::Result<Box<dyn Scorer>> {
+        fn scorer(
+            &self,
+            _reader: &dyn SegmentReader,
+            _boost: Score,
+        ) -> crate::Result<Box<dyn Scorer>> {
             Err(crate::TantivyError::InternalError("dummy impl".to_owned()))
         }
 
-        fn explain(&self, _reader: &SegmentReader, _doc: DocId) -> crate::Result<Explanation> {
+        fn explain(&self, _reader: &dyn SegmentReader, _doc: DocId) -> crate::Result<Explanation> {
             Err(crate::TantivyError::InternalError("dummy impl".to_owned()))
         }
     }

src/indexer/index_writer.rs

@@ -95,7 +95,7 @@ pub struct IndexWriter<C: Codec = StandardCodec, D: Document = TantivyDocument>
 
 fn compute_deleted_bitset(
     alive_bitset: &mut BitSet,
-    segment_reader: &SegmentReader,
+    segment_reader: &dyn SegmentReader,
     delete_cursor: &mut DeleteCursor,
     doc_opstamps: &DocToOpstampMapping,
     target_opstamp: Opstamp,
@@ -144,7 +144,12 @@ pub fn advance_deletes<C: Codec>(
         return Ok(());
     }
 
-    let segment_reader = SegmentReader::open(&segment)?;
+    let segment_reader = segment.index().codec().open_segment_reader(
+        segment.index().directory(),
+        segment.meta(),
+        segment.schema(),
+        None,
+    )?;
 
     let max_doc = segment_reader.max_doc();
     let mut alive_bitset: BitSet = match segment_entry.alive_bitset() {
@@ -156,7 +161,7 @@ pub fn advance_deletes<C: Codec>(
 
     compute_deleted_bitset(
         &mut alive_bitset,
-        &segment_reader,
+        segment_reader.as_ref(),
         segment_entry.delete_cursor(),
         &DocToOpstampMapping::None,
         target_opstamp,
@@ -244,14 +249,19 @@ fn apply_deletes<C: crate::codec::Codec>(
         .max()
         .expect("Empty DocOpstamp is forbidden");
 
-    let segment_reader = SegmentReader::open(segment)?;
+    let segment_reader = segment.index().codec().open_segment_reader(
+        segment.index().directory(),
+        segment.meta(),
+        segment.schema(),
+        None,
+    )?;
     let doc_to_opstamps = DocToOpstampMapping::WithMap(doc_opstamps);
 
     let max_doc = segment.meta().max_doc();
     let mut deleted_bitset = BitSet::with_max_value_and_full(max_doc);
     let may_have_deletes = compute_deleted_bitset(
         &mut deleted_bitset,
-        &segment_reader,
+        segment_reader.as_ref(),
         delete_cursor,
         &doc_to_opstamps,
         max_doc_opstamp,
@@ -1966,9 +1976,9 @@ mod tests {
                 .get_store_reader(DOCSTORE_CACHE_CAPACITY)
                 .unwrap();
             // test store iterator
-            for doc in store_reader.iter::<TantivyDocument>(segment_reader.alive_bitset()) {
+            for doc_id in segment_reader.doc_ids_alive() {
+                let doc = store_reader.get(doc_id).unwrap();
                 let id = doc
-                    .unwrap()
                     .get_first(id_field)
                     .unwrap()
                     .as_value()
@@ -1979,7 +1989,7 @@ mod tests {
             // test store random access
             for doc_id in segment_reader.doc_ids_alive() {
                 let id = store_reader
-                    .get::<TantivyDocument>(doc_id)
+                    .get(doc_id)
                     .unwrap()
                     .get_first(id_field)
                     .unwrap()
@@ -1988,7 +1998,7 @@ mod tests {
                 assert!(expected_ids_and_num_occurrences.contains_key(&id));
                 if id_is_full_doc(id) {
                     let id2 = store_reader
-                        .get::<TantivyDocument>(doc_id)
+                        .get(doc_id)
                         .unwrap()
                         .get_first(multi_numbers)
                         .unwrap()
@@ -1996,13 +2006,13 @@ mod tests {
                         .unwrap();
                     assert_eq!(id, id2);
                     let bool = store_reader
-                        .get::<TantivyDocument>(doc_id)
+                        .get(doc_id)
                         .unwrap()
                         .get_first(bool_field)
                         .unwrap()
                         .as_bool()
                         .unwrap();
-                    let doc = store_reader.get::<TantivyDocument>(doc_id).unwrap();
+                    let doc = store_reader.get(doc_id).unwrap();
                     let mut bool2 = doc.get_all(multi_bools);
                     assert_eq!(bool, bool2.next().unwrap().as_bool().unwrap());
                     assert_ne!(bool, bool2.next().unwrap().as_bool().unwrap());

src/indexer/merge_index_test.rs

@@ -1,6 +1,5 @@
 #[cfg(test)]
 mod tests {
-    use crate::codec::StandardCodec;
     use crate::collector::TopDocs;
     use crate::fastfield::AliveBitSet;
     use crate::index::Index;
@@ -123,22 +122,28 @@ mod tests {
             let term_a = Term::from_field_text(my_text_field, "text");
             let inverted_index = segment_reader.inverted_index(my_text_field).unwrap();
             let term_info = inverted_index.get_term_info(&term_a).unwrap().unwrap();
-            let mut postings = inverted_index
-                .read_postings_from_terminfo_specialized(
-                    &term_info,
-                    IndexRecordOption::WithFreqsAndPositions,
-                    &StandardCodec,
-                )
-                .unwrap();
-            assert_eq!(postings.doc_freq(), DocFreq::Exact(2));
+            let typed_postings = crate::codec::Codec::load_postings_typed(
+                index.codec(),
+                inverted_index.as_ref(),
+                &term_info,
+                IndexRecordOption::WithFreqsAndPositions,
+            )
+            .unwrap();
             let fallback_bitset = AliveBitSet::for_test_from_deleted_docs(&[0], 100);
             assert_eq!(
                 crate::indexer::merger::doc_freq_given_deletes(
-                    &postings,
+                    &typed_postings,
                     segment_reader.alive_bitset().unwrap_or(&fallback_bitset)
                 ),
                 2
             );
+            let mut postings = inverted_index
+                .read_postings_from_terminfo(&term_info, IndexRecordOption::WithFreqsAndPositions)
+                .unwrap();
+            assert_eq!(postings.doc_freq(), DocFreq::Exact(2));
+            let mut postings = inverted_index
+                .read_postings_from_terminfo(&term_info, IndexRecordOption::WithFreqsAndPositions)
+                .unwrap();
 
             assert_eq!(postings.term_freq(), 1);
             let mut output = Vec::new();

src/indexer/merger.rs

@@ -1,3 +1,5 @@
+use std::io;
+use std::marker::PhantomData;
 use std::sync::Arc;
 
 use columnar::{
@@ -17,8 +19,8 @@ use crate::fieldnorm::{FieldNormReader, FieldNormReaders, FieldNormsSerializer,
 use crate::index::{Segment, SegmentComponent, SegmentReader};
 use crate::indexer::doc_id_mapping::{MappingType, SegmentDocIdMapping};
 use crate::indexer::SegmentSerializer;
-use crate::postings::{InvertedIndexSerializer, Postings};
-use crate::schema::{value_type_to_column_type, Field, FieldType, Schema};
+use crate::postings::{InvertedIndexSerializer, Postings, TermInfo};
+use crate::schema::{value_type_to_column_type, Field, FieldType, IndexRecordOption, Schema};
 use crate::store::StoreWriter;
 use crate::termdict::{TermMerger, TermOrdinal};
 use crate::{DocAddress, DocId, InvertedIndexReader};
@@ -29,7 +31,7 @@ use crate::{DocAddress, DocId, InvertedIndexReader};
 pub const MAX_DOC_LIMIT: u32 = 1 << 31;
 
 fn estimate_total_num_tokens_in_single_segment(
-    reader: &SegmentReader,
+    reader: &dyn SegmentReader,
     field: Field,
 ) -> crate::Result<u64> {
     // There are no deletes. We can simply use the exact value saved into the posting list.
@@ -41,7 +43,7 @@ fn estimate_total_num_tokens_in_single_segment(
 
     // When there are deletes, we use an approximation either
     // by using the fieldnorm.
-    if let Some(fieldnorm_reader) = reader.fieldnorms_readers().get_field(field)? {
+    if let Ok(fieldnorm_reader) = reader.get_fieldnorms_reader(field) {
         let mut count: [usize; 256] = [0; 256];
         for doc in reader.doc_ids_alive() {
             let fieldnorm_id = fieldnorm_reader.fieldnorm_id(doc);
@@ -70,19 +72,23 @@ fn estimate_total_num_tokens_in_single_segment(
     Ok((segment_num_tokens as f64 * ratio) as u64)
 }
 
-fn estimate_total_num_tokens(readers: &[SegmentReader], field: Field) -> crate::Result<u64> {
+fn estimate_total_num_tokens(
+    readers: &[Arc<dyn SegmentReader>],
+    field: Field,
+) -> crate::Result<u64> {
     let mut total_num_tokens: u64 = 0;
     for reader in readers {
-        total_num_tokens += estimate_total_num_tokens_in_single_segment(reader, field)?;
+        total_num_tokens += estimate_total_num_tokens_in_single_segment(reader.as_ref(), field)?;
     }
     Ok(total_num_tokens)
 }
 
 pub struct IndexMerger<C: Codec = StandardCodec> {
     schema: Schema,
-    pub(crate) readers: Vec<SegmentReader>,
+    pub(crate) readers: Vec<Arc<dyn SegmentReader>>,
     max_doc: u32,
     codec: C,
+    phantom: PhantomData<C>,
 }
 
 struct DeltaComputer {
@@ -177,8 +183,12 @@ impl<C: Codec> IndexMerger<C> {
         let mut readers = vec![];
         for (segment, new_alive_bitset_opt) in segments.iter().zip(alive_bitset_opt) {
             if segment.meta().num_docs() > 0 {
-                let reader =
-                    SegmentReader::open_with_custom_alive_set(segment, new_alive_bitset_opt)?;
+                let reader = segment.index().codec().open_segment_reader(
+                    segment.index().directory(),
+                    segment.meta(),
+                    segment.schema(),
+                    new_alive_bitset_opt,
+                )?;
                 readers.push(reader);
             }
         }
@@ -197,6 +207,7 @@ impl<C: Codec> IndexMerger<C> {
             readers,
             max_doc,
             codec,
+            phantom: PhantomData,
         })
     }
 
@@ -270,7 +281,7 @@ impl<C: Codec> IndexMerger<C> {
                 }),
         );
 
-        let has_deletes: bool = self.readers.iter().any(SegmentReader::has_deletes);
+        let has_deletes: bool = self.readers.iter().any(|reader| reader.has_deletes());
         let mapping_type = if has_deletes {
             MappingType::StackedWithDeletes
         } else {
@@ -295,7 +306,7 @@ impl<C: Codec> IndexMerger<C> {
         &self,
         indexed_field: Field,
         _field_type: &FieldType,
-        serializer: &mut InvertedIndexSerializer<C>,
+        serializer: &mut InvertedIndexSerializer,
         fieldnorm_reader: Option<FieldNormReader>,
         doc_id_mapping: &SegmentDocIdMapping,
     ) -> crate::Result<()> {
@@ -305,7 +316,7 @@ impl<C: Codec> IndexMerger<C> {
 
         let mut max_term_ords: Vec<TermOrdinal> = Vec::new();
 
-        let field_readers: Vec<Arc<InvertedIndexReader>> = self
+        let field_readers: Vec<Arc<dyn InvertedIndexReader>> = self
             .readers
             .iter()
             .map(|reader| reader.inverted_index(indexed_field))
@@ -377,23 +388,14 @@ impl<C: Codec> IndexMerger<C> {
             // Let's compute the list of non-empty posting lists
             for (segment_ord, term_info) in merged_terms.current_segment_ords_and_term_infos() {
                 let segment_reader = &self.readers[segment_ord];
-                let inverted_index: &InvertedIndexReader = &field_readers[segment_ord];
-                let postings = inverted_index.read_postings_from_terminfo_specialized(
+                let inverted_index = &field_readers[segment_ord];
+                if let Some((doc_freq, postings)) = postings_for_merge::<C>(
+                    inverted_index.as_ref(),
+                    &self.codec,
                     &term_info,
                     segment_postings_option,
-                    &self.codec,
-                )?;
-                let alive_bitset_opt = segment_reader.alive_bitset();
-                let doc_freq = if let Some(alive_bitset) = alive_bitset_opt {
-                    doc_freq_given_deletes(&postings, alive_bitset)
-                } else {
-                    // We do not an exact document frequency here.
-                    match postings.doc_freq() {
-                        crate::postings::DocFreq::Approximate(_) => exact_doc_freq(&postings),
-                        crate::postings::DocFreq::Exact(doc_freq) => doc_freq,
-                    }
-                };
-                if doc_freq > 0u32 {
+                    segment_reader.alive_bitset(),
+                )? {
                     total_doc_freq += doc_freq;
                     segment_postings_containing_the_term.push((segment_ord, postings));
                 }
@@ -481,7 +483,7 @@ impl<C: Codec> IndexMerger<C> {
 
     fn write_postings(
         &self,
-        serializer: &mut InvertedIndexSerializer<C>,
+        serializer: &mut InvertedIndexSerializer,
         fieldnorm_readers: FieldNormReaders,
         doc_id_mapping: &SegmentDocIdMapping,
     ) -> crate::Result<()> {
@@ -504,33 +506,7 @@ impl<C: Codec> IndexMerger<C> {
         debug_time!("write-storable-fields");
         debug!("write-storable-field");
 
-        for reader in &self.readers {
-            let store_reader = reader.get_store_reader(1)?;
-            if reader.has_deletes()
-                    // If there is not enough data in the store, we avoid stacking in order to
-                    // avoid creating many small blocks in the doc store. Once we have 5 full blocks,
-                    // we start stacking. In the worst case 2/7 of the blocks would be very small.
-                    // [segment 1 - {1 doc}][segment 2 - {fullblock * 5}{1doc}]
-                    // => 5 * full blocks, 2 * 1 document blocks
-                    //
-                    // In a more realistic scenario the segments are of the same size, so 1/6 of
-                    // the doc stores would be on average half full, given total randomness (which
-                    // is not the case here, but not sure how it behaves exactly).
-                    //
-                    // https://github.com/quickwit-oss/tantivy/issues/1053
-                    //
-                    // take 7 in order to not walk over all checkpoints.
-                    || store_reader.block_checkpoints().take(7).count() < 6
-                    || store_reader.decompressor() != store_writer.compressor().into()
-            {
-                for doc_bytes_res in store_reader.iter_raw(reader.alive_bitset()) {
-                    let doc_bytes = doc_bytes_res?;
-                    store_writer.store_bytes(&doc_bytes)?;
-                }
-            } else {
-                store_writer.stack(store_reader)?;
-            }
-        }
+        store_writer.merge_segment_readers(&self.readers)?;
         Ok(())
     }
 
@@ -575,32 +551,66 @@ pub(crate) fn doc_freq_given_deletes<P: Postings + Clone>(
     postings: &P,
     alive_bitset: &AliveBitSet,
 ) -> u32 {
-    let mut docset = postings.clone();
+    let mut postings = postings.clone();
     let mut doc_freq = 0;
     loop {
-        let doc = docset.doc();
+        let doc = postings.doc();
         if doc == TERMINATED {
             return doc_freq;
         }
         if alive_bitset.is_alive(doc) {
             doc_freq += 1u32;
         }
-        docset.advance();
+        postings.advance();
     }
 }
 
+fn read_postings_for_merge<C: Codec>(
+    inverted_index: &dyn InvertedIndexReader,
+    codec: &C,
+    term_info: &TermInfo,
+    option: IndexRecordOption,
+) -> io::Result<<C::PostingsCodec as PostingsCodec>::Postings> {
+    codec.load_postings_typed(inverted_index, term_info, option)
+}
+
+fn postings_for_merge<C: Codec>(
+    inverted_index: &dyn InvertedIndexReader,
+    codec: &C,
+    term_info: &TermInfo,
+    option: IndexRecordOption,
+    alive_bitset_opt: Option<&AliveBitSet>,
+) -> io::Result<Option<(u32, <C::PostingsCodec as PostingsCodec>::Postings)>> {
+    let postings = read_postings_for_merge(inverted_index, codec, term_info, option)?;
+    let doc_freq = if let Some(alive_bitset) = alive_bitset_opt {
+        doc_freq_given_deletes(&postings, alive_bitset)
+    } else {
+        // We do not need an exact document frequency here.
+        match postings.doc_freq() {
+            crate::postings::DocFreq::Exact(doc_freq) => doc_freq,
+            crate::postings::DocFreq::Approximate(_) => exact_doc_freq(&postings),
+        }
+    };
+
+    if doc_freq == 0u32 {
+        return Ok(None);
+    }
+
+    Ok(Some((doc_freq, postings)))
+}
+
 /// If the postings is not able to inform us of the document frequency,
 /// we just scan through it.
 pub(crate) fn exact_doc_freq<P: Postings + Clone>(postings: &P) -> u32 {
-    let mut docset = postings.clone();
+    let mut postings = postings.clone();
     let mut doc_freq = 0;
     loop {
-        let doc = docset.doc();
+        let doc = postings.doc();
         if doc == TERMINATED {
             return doc_freq;
         }
         doc_freq += 1u32;
-        docset.advance();
+        postings.advance();
     }
 }
 
@@ -736,32 +746,32 @@ mod tests {
                 );
             }
             {
-                let doc = searcher.doc::<TantivyDocument>(DocAddress::new(0, 0))?;
+                let doc = searcher.doc(DocAddress::new(0, 0))?;
                 assert_eq!(
                     doc.get_first(text_field).unwrap().as_value().as_str(),
                     Some("af b")
                 );
             }
             {
-                let doc = searcher.doc::<TantivyDocument>(DocAddress::new(0, 1))?;
+                let doc = searcher.doc(DocAddress::new(0, 1))?;
                 assert_eq!(
                     doc.get_first(text_field).unwrap().as_value().as_str(),
                     Some("a b c")
                 );
             }
             {
-                let doc = searcher.doc::<TantivyDocument>(DocAddress::new(0, 2))?;
+                let doc = searcher.doc(DocAddress::new(0, 2))?;
                 assert_eq!(
                     doc.get_first(text_field).unwrap().as_value().as_str(),
                     Some("a b c d")
                 );
             }
             {
-                let doc = searcher.doc::<TantivyDocument>(DocAddress::new(0, 3))?;
+                let doc = searcher.doc(DocAddress::new(0, 3))?;
                 assert_eq!(doc.get_first(text_field).unwrap().as_str(), Some("af b"));
             }
             {
-                let doc = searcher.doc::<TantivyDocument>(DocAddress::new(0, 4))?;
+                let doc = searcher.doc(DocAddress::new(0, 4))?;
                 assert_eq!(doc.get_first(text_field).unwrap().as_str(), Some("a b c g"));
             }
 
@@ -1589,7 +1599,7 @@ mod tests {
         for segment_reader in searcher.segment_readers() {
             let mut term_scorer = term_query
                 .specialized_weight(EnableScoring::enabled_from_searcher(&searcher))?
-                .term_scorer_for_test(segment_reader, 1.0)
+                .term_scorer_for_test(segment_reader.as_ref(), 1.0)
                 .unwrap();
             // the difference compared to before is intrinsic to the bm25 formula. no worries
             // there.
@@ -1644,6 +1654,8 @@ mod tests {
         assert_eq!(super::doc_freq_given_deletes(&docs, &alive_bitset), 2);
         let all_deleted =
             AliveBitSet::for_test_from_deleted_docs(&[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], 12);
+        let docs =
+            <StandardPostingsCodec as PostingsCodec>::Postings::create_from_docs(&[0, 2, 10]);
         assert_eq!(super::doc_freq_given_deletes(&docs, &all_deleted), 0);
     }
 }

src/indexer/segment_serializer.rs

@@ -13,7 +13,7 @@ pub struct SegmentSerializer<C: crate::codec::Codec> {
     pub(crate) store_writer: StoreWriter,
     fast_field_write: WritePtr,
     fieldnorms_serializer: Option<FieldNormsSerializer>,
-    postings_serializer: InvertedIndexSerializer<C>,
+    postings_serializer: InvertedIndexSerializer,
 }
 
 impl<C: crate::codec::Codec> SegmentSerializer<C> {
@@ -55,7 +55,7 @@ impl<C: crate::codec::Codec> SegmentSerializer<C> {
     }
 
     /// Accessor to the `PostingsSerializer`.
-    pub fn get_postings_serializer(&mut self) -> &mut InvertedIndexSerializer<C> {
+    pub fn get_postings_serializer(&mut self) -> &mut InvertedIndexSerializer {
         &mut self.postings_serializer
     }
 

src/indexer/segment_writer.rs

@@ -438,7 +438,7 @@ mod tests {
         Document, IndexRecordOption, OwnedValue, Schema, TextFieldIndexing, TextOptions, Value,
         DATE_TIME_PRECISION_INDEXED, FAST, STORED, STRING, TEXT,
     };
-    use crate::store::{Compressor, StoreReader, StoreWriter};
+    use crate::store::{Compressor, StoreWriter, TantivyStoreReader};
     use crate::time::format_description::well_known::Rfc3339;
     use crate::time::OffsetDateTime;
     use crate::tokenizer::{PreTokenizedString, Token};
@@ -486,8 +486,8 @@ mod tests {
         store_writer.store(&doc, &schema).unwrap();
         store_writer.close().unwrap();
 
-        let reader = StoreReader::open(directory.open_read(path).unwrap(), 0).unwrap();
-        let doc = reader.get::<TantivyDocument>(0).unwrap();
+        let reader = TantivyStoreReader::open(directory.open_read(path).unwrap(), 0).unwrap();
+        let doc = reader.get(0).unwrap();
 
         assert_eq!(doc.field_values().count(), 2);
         assert_eq!(
@@ -604,16 +604,12 @@ mod tests {
         let reader = index.reader().unwrap();
         let searcher = reader.searcher();
         let doc = searcher
-            .doc::<TantivyDocument>(DocAddress {
+            .doc(DocAddress {
                 segment_ord: 0u32,
                 doc_id: 0u32,
             })
             .unwrap();
-        let serdeser_json_val = serde_json::from_str::<serde_json::Value>(&doc.to_json(&schema))
-            .unwrap()
-            .get("json")
-            .unwrap()[0]
-            .clone();
+        let serdeser_json_val = doc.to_json(&schema).get("json").unwrap().clone();
         assert_eq!(json_val, serdeser_json_val);
         let segment_reader = searcher.segment_reader(0u32);
         let inv_idx = segment_reader.inverted_index(json_field).unwrap();
@@ -875,7 +871,7 @@ mod tests {
         let searcher = reader.searcher();
         let segment_reader = searcher.segment_reader(0u32);
 
-        fn assert_type(reader: &SegmentReader, field: &str, typ: ColumnType) {
+        fn assert_type(reader: &dyn SegmentReader, field: &str, typ: ColumnType) {
             let cols = reader.fast_fields().dynamic_column_handles(field).unwrap();
             assert_eq!(cols.len(), 1, "{field}");
             assert_eq!(cols[0].column_type(), typ, "{field}");
@@ -894,7 +890,7 @@ mod tests {
         assert_type(segment_reader, "json.my_arr", ColumnType::I64);
         assert_type(segment_reader, "json.my_arr.my_key", ColumnType::Str);
 
-        fn assert_empty(reader: &SegmentReader, field: &str) {
+        fn assert_empty(reader: &dyn SegmentReader, field: &str) {
             let cols = reader.fast_fields().dynamic_column_handles(field).unwrap();
             assert_eq!(cols.len(), 0);
         }

src/lib.rs

@@ -93,7 +93,7 @@
 //!
 //! for (_score, doc_address) in top_docs {
 //!     // Retrieve the actual content of documents given its `doc_address`.
-//!     let retrieved_doc = searcher.doc::<TantivyDocument>(doc_address)?;
+//!     let retrieved_doc = searcher.doc(doc_address)?;
 //!     println!("{}", retrieved_doc.to_json(&schema));
 //! }
 //!
@@ -224,11 +224,11 @@ use once_cell::sync::Lazy;
 use serde::{Deserialize, Serialize};
 
 pub use self::docset::{DocSet, COLLECT_BLOCK_BUFFER_LEN, TERMINATED};
-pub use crate::core::{json_utils, Executor, Searcher, SearcherGeneration};
+pub use crate::core::{json_utils, Executor, Searcher, SearcherContext, SearcherGeneration};
 pub use crate::directory::Directory;
 pub use crate::index::{
     Index, IndexBuilder, IndexMeta, IndexSettings, InvertedIndexReader, Order, Segment,
-    SegmentMeta, SegmentReader,
+    SegmentMeta, SegmentReader, TantivyInvertedIndexReader, TantivySegmentReader,
 };
 pub use crate::indexer::{IndexWriter, SingleSegmentIndexWriter};
 pub use crate::schema::{Document, TantivyDocument, Term};
@@ -548,7 +548,7 @@ pub mod tests {
         index_writer.commit()?;
         let reader = index.reader()?;
         let searcher = reader.searcher();
-        let segment_reader: &SegmentReader = searcher.segment_reader(0);
+        let segment_reader: &dyn SegmentReader = searcher.segment_reader(0);
         let fieldnorms_reader = segment_reader.get_fieldnorms_reader(text_field)?;
         assert_eq!(fieldnorms_reader.fieldnorm(0), 3);
         assert_eq!(fieldnorms_reader.fieldnorm(1), 0);
@@ -556,7 +556,7 @@ pub mod tests {
         Ok(())
     }
 
-    fn advance_undeleted(docset: &mut dyn DocSet, reader: &SegmentReader) -> bool {
+    fn advance_undeleted(docset: &mut dyn DocSet, reader: &dyn SegmentReader) -> bool {
         let mut doc = docset.advance();
         while doc != TERMINATED {
             if !reader.is_deleted(doc) {
@@ -1073,7 +1073,7 @@ pub mod tests {
         }
         let reader = index.reader()?;
         let searcher = reader.searcher();
-        let segment_reader: &SegmentReader = searcher.segment_reader(0);
+        let segment_reader: &dyn SegmentReader = searcher.segment_reader(0);
         {
             let fast_field_reader_res = segment_reader.fast_fields().u64("text");
             assert!(fast_field_reader_res.is_err());

src/postings/json_postings_writer.rs

@@ -3,7 +3,6 @@ use std::io;
 use common::json_path_writer::JSON_END_OF_PATH;
 use stacker::Addr;
 
-use crate::codec::Codec;
 use crate::indexer::indexing_term::IndexingTerm;
 use crate::indexer::path_to_unordered_id::OrderedPathId;
 use crate::postings::postings_writer::SpecializedPostingsWriter;
@@ -53,12 +52,12 @@ impl<Rec: Recorder> PostingsWriter for JsonPostingsWriter<Rec> {
     }
 
     /// The actual serialization format is handled by the `PostingsSerializer`.
-    fn serialize<C: Codec>(
+    fn serialize(
         &self,
         ordered_term_addrs: &[(Field, OrderedPathId, &[u8], Addr)],
         ordered_id_to_path: &[&str],
         ctx: &IndexingContext,
-        serializer: &mut FieldSerializer<C>,
+        serializer: &mut FieldSerializer,
     ) -> io::Result<()> {
         let mut term_buffer = JsonTermSerializer(Vec::with_capacity(48));
         let mut buffer_lender = BufferLender::default();

src/postings/mod.rs

@@ -12,7 +12,9 @@ mod per_field_postings_writer;
 mod postings;
 mod postings_writer;
 mod recorder;
-mod serializer;
+/// Serializer module for the inverted index
+pub mod serializer;
+pub(crate) mod skip;
 mod term_info;
 
 pub(crate) use loaded_postings::LoadedPostings;
@@ -35,7 +37,7 @@ pub(crate) mod tests {
     use super::{InvertedIndexSerializer, Postings};
     use crate::docset::{DocSet, TERMINATED};
     use crate::fieldnorm::FieldNormReader;
-    use crate::index::{Index, SegmentComponent, SegmentReader};
+    use crate::index::{Index, SegmentComponent};
     use crate::indexer::operation::AddOperation;
     use crate::indexer::SegmentWriter;
     use crate::postings::DocFreq;
@@ -248,7 +250,13 @@ pub(crate) mod tests {
             segment_writer.finalize()?;
         }
         {
-            let segment_reader = SegmentReader::open(&segment)?;
+            let segment_reader = crate::codec::Codec::open_segment_reader(
+                segment.index().codec(),
+                segment.index().directory(),
+                segment.meta(),
+                segment.schema(),
+                None,
+            )?;
             {
                 let fieldnorm_reader = segment_reader.get_fieldnorms_reader(text_field)?;
                 assert_eq!(fieldnorm_reader.fieldnorm(0), 8 + 5);

src/postings/postings_writer.rs

@@ -4,7 +4,6 @@ use std::ops::Range;
 
 use stacker::Addr;
 
-use crate::codec::Codec;
 use crate::fieldnorm::FieldNormReaders;
 use crate::indexer::indexing_term::IndexingTerm;
 use crate::indexer::path_to_unordered_id::OrderedPathId;
@@ -49,12 +48,12 @@ fn make_field_partition(
 /// Serialize the inverted index.
 /// It pushes all term, one field at a time, towards the
 /// postings serializer.
-pub(crate) fn serialize_postings<C: Codec>(
+pub(crate) fn serialize_postings(
     ctx: IndexingContext,
     schema: Schema,
     per_field_postings_writers: &PerFieldPostingsWriter,
     fieldnorm_readers: FieldNormReaders,
-    serializer: &mut InvertedIndexSerializer<C>,
+    serializer: &mut InvertedIndexSerializer,
 ) -> crate::Result<()> {
     // Replace unordered ids by ordered ids to be able to sort
     let unordered_id_to_ordered_id: Vec<OrderedPathId> =
@@ -167,12 +166,12 @@ impl PostingsWriter for PostingsWriterEnum {
         }
     }
 
-    fn serialize<C: Codec>(
+    fn serialize(
         &self,
         term_addrs: &[(Field, OrderedPathId, &[u8], Addr)],
         ordered_id_to_path: &[&str],
         ctx: &IndexingContext,
-        serializer: &mut FieldSerializer<C>,
+        serializer: &mut FieldSerializer,
     ) -> io::Result<()> {
         match self {
             PostingsWriterEnum::DocId(writer) => {
@@ -255,12 +254,12 @@ pub(crate) trait PostingsWriter: Send + Sync {
 
     /// Serializes the postings on disk.
     /// The actual serialization format is handled by the `PostingsSerializer`.
-    fn serialize<C: Codec>(
+    fn serialize(
         &self,
         term_addrs: &[(Field, OrderedPathId, &[u8], Addr)],
         ordered_id_to_path: &[&str],
         ctx: &IndexingContext,
-        serializer: &mut FieldSerializer<C>,
+        serializer: &mut FieldSerializer,
     ) -> io::Result<()>;
 
     /// Tokenize a text and subscribe all of its token.
@@ -312,12 +311,12 @@ pub(crate) struct SpecializedPostingsWriter<Rec: Recorder> {
 
 impl<Rec: Recorder> SpecializedPostingsWriter<Rec> {
     #[inline]
-    pub(crate) fn serialize_one_term<C: Codec>(
+    pub(crate) fn serialize_one_term(
         term: &[u8],
         addr: Addr,
         buffer_lender: &mut BufferLender,
         ctx: &IndexingContext,
-        serializer: &mut FieldSerializer<C>,
+        serializer: &mut FieldSerializer,
     ) -> io::Result<()> {
         let recorder: Rec = ctx.term_index.read(addr);
         let term_doc_freq = recorder.term_doc_freq().unwrap_or(0u32);
@@ -358,12 +357,12 @@ impl<Rec: Recorder> PostingsWriter for SpecializedPostingsWriter<Rec> {
         });
     }
 
-    fn serialize<C: Codec>(
+    fn serialize(
         &self,
         term_addrs: &[(Field, OrderedPathId, &[u8], Addr)],
         _ordered_id_to_path: &[&str],
         ctx: &IndexingContext,
-        serializer: &mut FieldSerializer<C>,
+        serializer: &mut FieldSerializer,
     ) -> io::Result<()> {
         let mut buffer_lender = BufferLender::default();
         for (_field, _path_id, term, addr) in term_addrs {

src/postings/recorder.rs

@@ -1,7 +1,6 @@
 use common::read_u32_vint;
 use stacker::{ExpUnrolledLinkedList, MemoryArena};
 
-use crate::codec::Codec;
 use crate::postings::FieldSerializer;
 use crate::DocId;
 
@@ -68,10 +67,10 @@ pub(crate) trait Recorder: Copy + Default + Send + Sync + 'static {
     /// Close the document. It will help record the term frequency.
     fn close_doc(&mut self, arena: &mut MemoryArena);
     /// Pushes the postings information to the serializer.
-    fn serialize<C: Codec>(
+    fn serialize(
         &self,
         arena: &MemoryArena,
-        serializer: &mut FieldSerializer<C>,
+        serializer: &mut FieldSerializer,
         buffer_lender: &mut BufferLender,
     );
     /// Returns the number of document containing this term.
@@ -111,10 +110,10 @@ impl Recorder for DocIdRecorder {
     #[inline]
     fn close_doc(&mut self, _arena: &mut MemoryArena) {}
 
-    fn serialize<C: Codec>(
+    fn serialize(
         &self,
         arena: &MemoryArena,
-        serializer: &mut FieldSerializer<C>,
+        serializer: &mut FieldSerializer,
         buffer_lender: &mut BufferLender,
     ) {
         let buffer = buffer_lender.lend_u8();
@@ -179,10 +178,10 @@ impl Recorder for TermFrequencyRecorder {
         self.current_tf = 0;
     }
 
-    fn serialize<C: Codec>(
+    fn serialize(
         &self,
         arena: &MemoryArena,
-        serializer: &mut FieldSerializer<C>,
+        serializer: &mut FieldSerializer,
         buffer_lender: &mut BufferLender,
     ) {
         let buffer = buffer_lender.lend_u8();
@@ -236,10 +235,10 @@ impl Recorder for TfAndPositionRecorder {
         self.stack.writer(arena).write_u32_vint(POSITION_END);
     }
 
-    fn serialize<C: Codec>(
+    fn serialize(
         &self,
         arena: &MemoryArena,
-        serializer: &mut FieldSerializer<C>,
+        serializer: &mut FieldSerializer,
         buffer_lender: &mut BufferLender,
     ) {
         let (buffer_u8, buffer_positions) = buffer_lender.lend_all();

src/postings/serializer.rs

@@ -1,15 +1,17 @@
+use std::cmp::Ordering;
 use std::io::{self, Write};
 
-use common::{BinarySerializable, CountingWriter};
+use common::{BinarySerializable, CountingWriter, VInt};
 
 use super::TermInfo;
-use crate::codec::postings::PostingsSerializer;
-use crate::codec::Codec;
 use crate::directory::{CompositeWrite, WritePtr};
 use crate::fieldnorm::FieldNormReader;
 use crate::index::Segment;
 use crate::positions::PositionSerializer;
-use crate::schema::{Field, FieldEntry, FieldType, IndexRecordOption, Schema};
+use crate::postings::compression::{BlockEncoder, VIntEncoder as _, COMPRESSION_BLOCK_SIZE};
+use crate::postings::skip::SkipSerializer;
+use crate::query::Bm25Weight;
+use crate::schema::{Field, FieldEntry, IndexRecordOption, Schema};
 use crate::termdict::TermDictionaryBuilder;
 use crate::{DocId, Score};
 
@@ -44,27 +46,24 @@ use crate::{DocId, Score};
 ///
 /// A description of the serialization format is
 /// [available here](https://fulmicoton.gitbooks.io/tantivy-doc/content/inverted-index.html).
-pub struct InvertedIndexSerializer<C: Codec> {
+pub struct InvertedIndexSerializer {
     terms_write: CompositeWrite<WritePtr>,
     postings_write: CompositeWrite<WritePtr>,
     positions_write: CompositeWrite<WritePtr>,
     schema: Schema,
-    codec: C,
 }
 
-use crate::codec::postings::PostingsCodec;
-
-impl<C: Codec> InvertedIndexSerializer<C> {
+impl InvertedIndexSerializer {
     /// Open a new `InvertedIndexSerializer` for the given segment
-    pub fn open(segment: &mut Segment<C>) -> crate::Result<InvertedIndexSerializer<C>> {
+    pub fn open<C: crate::codec::Codec>(
+        segment: &mut Segment<C>,
+    ) -> crate::Result<InvertedIndexSerializer> {
         use crate::index::SegmentComponent::{Positions, Postings, Terms};
-        let codec = segment.index().codec().clone();
         let inv_index_serializer = InvertedIndexSerializer {
             terms_write: CompositeWrite::wrap(segment.open_write(Terms)?),
             postings_write: CompositeWrite::wrap(segment.open_write(Postings)?),
             positions_write: CompositeWrite::wrap(segment.open_write(Positions)?),
             schema: segment.schema(),
-            codec,
         };
         Ok(inv_index_serializer)
     }
@@ -78,20 +77,22 @@ impl<C: Codec> InvertedIndexSerializer<C> {
         field: Field,
         total_num_tokens: u64,
         fieldnorm_reader: Option<FieldNormReader>,
-    ) -> io::Result<FieldSerializer<'_, C>> {
+    ) -> io::Result<FieldSerializer<'_>> {
         let field_entry: &FieldEntry = self.schema.get_field_entry(field);
         let term_dictionary_write = self.terms_write.for_field(field);
         let postings_write = self.postings_write.for_field(field);
         let positions_write = self.positions_write.for_field(field);
-        let field_type: FieldType = (*field_entry.field_type()).clone();
+        let index_record_option = field_entry
+            .field_type()
+            .index_record_option()
+            .unwrap_or(IndexRecordOption::Basic);
         FieldSerializer::create(
-            &field_type,
+            index_record_option,
             total_num_tokens,
             term_dictionary_write,
             postings_write,
             positions_write,
             fieldnorm_reader,
-            &self.codec,
         )
     }
 
@@ -106,40 +107,34 @@ impl<C: Codec> InvertedIndexSerializer<C> {
 
 /// The field serializer is in charge of
 /// the serialization of a specific field.
-pub struct FieldSerializer<'a, C: Codec> {
-    term_dictionary_builder: TermDictionaryBuilder<&'a mut CountingWriter<WritePtr>>,
-    postings_serializer: <C::PostingsCodec as PostingsCodec>::PostingsSerializer,
-    positions_serializer_opt: Option<PositionSerializer<&'a mut CountingWriter<WritePtr>>>,
+pub struct FieldSerializer<'a, W: Write = WritePtr> {
+    term_dictionary_builder: TermDictionaryBuilder<&'a mut CountingWriter<W>>,
+    postings_serializer: PostingsSerializer,
+    positions_serializer_opt: Option<PositionSerializer<&'a mut CountingWriter<W>>>,
     current_term_info: TermInfo,
     term_open: bool,
-    postings_write: &'a mut CountingWriter<WritePtr>,
+    postings_write: &'a mut CountingWriter<W>,
     postings_start_offset: u64,
 }
 
-impl<'a, C: Codec> FieldSerializer<'a, C> {
-    fn create(
-        field_type: &FieldType,
+impl<'a, W: Write> FieldSerializer<'a, W> {
+    /// Creates a new `FieldSerializer` for the given field type.
+    pub fn create(
+        index_record_option: IndexRecordOption,
         total_num_tokens: u64,
-        term_dictionary_write: &'a mut CountingWriter<WritePtr>,
-        postings_write: &'a mut CountingWriter<WritePtr>,
-        positions_write: &'a mut CountingWriter<WritePtr>,
+        term_dictionary_write: &'a mut CountingWriter<W>,
+        postings_write: &'a mut CountingWriter<W>,
+        positions_write: &'a mut CountingWriter<W>,
         fieldnorm_reader: Option<FieldNormReader>,
-        codec: &C,
-    ) -> io::Result<FieldSerializer<'a, C>> {
+    ) -> io::Result<FieldSerializer<'a, W>> {
         total_num_tokens.serialize(postings_write)?;
-        let index_record_option = field_type
-            .index_record_option()
-            .unwrap_or(IndexRecordOption::Basic);
         let term_dictionary_builder = TermDictionaryBuilder::create(term_dictionary_write)?;
         let average_fieldnorm = fieldnorm_reader
             .as_ref()
             .map(|ff_reader| total_num_tokens as Score / ff_reader.num_docs() as Score)
             .unwrap_or(0.0);
-        let postings_serializer = codec.postings_codec().new_serializer(
-            average_fieldnorm,
-            index_record_option,
-            fieldnorm_reader,
-        );
+        let postings_serializer =
+            PostingsSerializer::new(average_fieldnorm, index_record_option, fieldnorm_reader);
         let positions_serializer_opt = if index_record_option.has_positions() {
             Some(PositionSerializer::new(positions_write))
         } else {
@@ -192,6 +187,7 @@ impl<'a, C: Codec> FieldSerializer<'a, C> {
             "Called new_term, while the previous term was not closed."
         );
         self.term_open = true;
+        self.postings_serializer.clear();
         self.current_term_info = self.current_term_info();
         self.term_dictionary_builder.insert_key(term)?;
         self.postings_serializer
@@ -199,6 +195,11 @@ impl<'a, C: Codec> FieldSerializer<'a, C> {
         Ok(())
     }
 
+    /// Starts the postings for a new term without recording term frequencies.
+    pub fn new_term_without_freq(&mut self, term: &[u8]) -> io::Result<()> {
+        self.new_term(term, 0, false)
+    }
+
     /// Serialize the information that a document contains for the current term:
     /// its term frequency, and the position deltas.
     ///
@@ -255,3 +256,234 @@ impl<'a, C: Codec> FieldSerializer<'a, C> {
         Ok(())
     }
 }
+
+struct Block {
+    doc_ids: [DocId; COMPRESSION_BLOCK_SIZE],
+    term_freqs: [u32; COMPRESSION_BLOCK_SIZE],
+    len: usize,
+}
+
+impl Block {
+    fn new() -> Self {
+        Block {
+            doc_ids: [0u32; COMPRESSION_BLOCK_SIZE],
+            term_freqs: [0u32; COMPRESSION_BLOCK_SIZE],
+            len: 0,
+        }
+    }
+
+    fn doc_ids(&self) -> &[DocId] {
+        &self.doc_ids[..self.len]
+    }
+
+    fn term_freqs(&self) -> &[u32] {
+        &self.term_freqs[..self.len]
+    }
+
+    fn clear(&mut self) {
+        self.len = 0;
+    }
+
+    fn append_doc(&mut self, doc: DocId, term_freq: u32) {
+        let len = self.len;
+        self.doc_ids[len] = doc;
+        self.term_freqs[len] = term_freq;
+        self.len = len + 1;
+    }
+
+    fn is_full(&self) -> bool {
+        self.len == COMPRESSION_BLOCK_SIZE
+    }
+
+    fn is_empty(&self) -> bool {
+        self.len == 0
+    }
+
+    fn last_doc(&self) -> DocId {
+        assert_eq!(self.len, COMPRESSION_BLOCK_SIZE);
+        self.doc_ids[COMPRESSION_BLOCK_SIZE - 1]
+    }
+}
+
+/// Serializer for postings lists.
+pub struct PostingsSerializer {
+    last_doc_id_encoded: u32,
+
+    block_encoder: BlockEncoder,
+    block: Box<Block>,
+
+    postings_write: Vec<u8>,
+    skip_write: SkipSerializer,
+
+    mode: IndexRecordOption,
+    fieldnorm_reader: Option<FieldNormReader>,
+
+    bm25_weight: Option<Bm25Weight>,
+    avg_fieldnorm: Score, /* Average number of term in the field for that segment.
+                           * this value is used to compute the block wand information. */
+    term_has_freq: bool,
+}
+
+impl PostingsSerializer {
+    /// Creates a new `PostingsSerializer`.
+    /// * avg_fieldnorm - average field norm for the field being serialized.
+    /// * mode - indexing options for the field being serialized.
+    pub fn new(
+        avg_fieldnorm: Score,
+        mode: IndexRecordOption,
+        fieldnorm_reader: Option<FieldNormReader>,
+    ) -> PostingsSerializer {
+        PostingsSerializer {
+            block_encoder: BlockEncoder::new(),
+            block: Box::new(Block::new()),
+
+            postings_write: Vec::new(),
+            skip_write: SkipSerializer::new(),
+
+            last_doc_id_encoded: 0u32,
+            mode,
+
+            fieldnorm_reader,
+            bm25_weight: None,
+            avg_fieldnorm,
+            term_has_freq: false,
+        }
+    }
+
+    /// Starts the serialization for a new term.
+    /// * term_doc_freq - the number of documents containing the term.
+    pub fn new_term(&mut self, term_doc_freq: u32, record_term_freq: bool) {
+        self.bm25_weight = None;
+
+        self.term_has_freq = self.mode.has_freq() && record_term_freq;
+        if !self.term_has_freq {
+            return;
+        }
+
+        let num_docs_in_segment: u64 =
+            if let Some(fieldnorm_reader) = self.fieldnorm_reader.as_ref() {
+                fieldnorm_reader.num_docs() as u64
+            } else {
+                return;
+            };
+
+        if num_docs_in_segment == 0 {
+            return;
+        }
+
+        self.bm25_weight = Some(Bm25Weight::for_one_term_without_explain(
+            term_doc_freq as u64,
+            num_docs_in_segment,
+            self.avg_fieldnorm,
+        ));
+    }
+
+    fn write_block(&mut self) {
+        {
+            // encode the doc ids
+            let (num_bits, block_encoded): (u8, &[u8]) = self
+                .block_encoder
+                .compress_block_sorted(self.block.doc_ids(), self.last_doc_id_encoded);
+            self.last_doc_id_encoded = self.block.last_doc();
+            self.skip_write
+                .write_doc(self.last_doc_id_encoded, num_bits);
+            // last el block 0, offset block 1,
+            self.postings_write.extend(block_encoded);
+        }
+        if self.term_has_freq {
+            // encode the term frequencies
+            let (num_bits, block_encoded): (u8, &[u8]) = self
+                .block_encoder
+                .compress_block_unsorted(self.block.term_freqs(), true);
+            self.postings_write.extend(block_encoded);
+            self.skip_write.write_term_freq(num_bits);
+            if self.mode.has_positions() {
+                // We serialize the sum of term freqs within the skip information
+                // in order to navigate through positions.
+                let sum_freq = self.block.term_freqs().iter().cloned().sum();
+                self.skip_write.write_total_term_freq(sum_freq);
+            }
+            let mut blockwand_params = (0u8, 0u32);
+            if let Some(bm25_weight) = self.bm25_weight.as_ref() {
+                if let Some(fieldnorm_reader) = self.fieldnorm_reader.as_ref() {
+                    let docs = self.block.doc_ids().iter().cloned();
+                    let term_freqs = self.block.term_freqs().iter().cloned();
+                    let fieldnorms = docs.map(|doc| fieldnorm_reader.fieldnorm_id(doc));
+                    blockwand_params = fieldnorms
+                        .zip(term_freqs)
+                        .max_by(
+                            |(left_fieldnorm_id, left_term_freq),
+                             (right_fieldnorm_id, right_term_freq)| {
+                                let left_score =
+                                    bm25_weight.tf_factor(*left_fieldnorm_id, *left_term_freq);
+                                let right_score =
+                                    bm25_weight.tf_factor(*right_fieldnorm_id, *right_term_freq);
+                                left_score
+                                    .partial_cmp(&right_score)
+                                    .unwrap_or(Ordering::Equal)
+                            },
+                        )
+                        .unwrap();
+                }
+            }
+            let (fieldnorm_id, term_freq) = blockwand_params;
+            self.skip_write.write_blockwand_max(fieldnorm_id, term_freq);
+        }
+        self.block.clear();
+    }
+
+    /// Register that the given document contains the current term.
+    /// * doc_id - the document id.
+    /// * term_freq - the term frequency within the document.
+    pub fn write_doc(&mut self, doc_id: DocId, term_freq: u32) {
+        self.block.append_doc(doc_id, term_freq);
+        if self.block.is_full() {
+            self.write_block();
+        }
+    }
+
+    /// Finish the serialization for this term.
+    pub fn close_term(
+        &mut self,
+        doc_freq: u32,
+        output_write: &mut impl std::io::Write,
+    ) -> io::Result<()> {
+        if !self.block.is_empty() {
+            // we have doc ids waiting to be written
+            // this happens when the number of doc ids is
+            // not a perfect multiple of our block size.
+            //
+            // In that case, the remaining part is encoded
+            // using variable int encoding.
+            {
+                let block_encoded = self
+                    .block_encoder
+                    .compress_vint_sorted(self.block.doc_ids(), self.last_doc_id_encoded);
+                self.postings_write.write_all(block_encoded)?;
+            }
+            // ... Idem for term frequencies
+            if self.term_has_freq {
+                let block_encoded = self
+                    .block_encoder
+                    .compress_vint_unsorted(self.block.term_freqs());
+                self.postings_write.write_all(block_encoded)?;
+            }
+            self.block.clear();
+        }
+        if doc_freq >= COMPRESSION_BLOCK_SIZE as u32 {
+            let skip_data = self.skip_write.data();
+            VInt(skip_data.len() as u64).serialize(output_write)?;
+            output_write.write_all(skip_data)?;
+        }
+        output_write.write_all(&self.postings_write[..])?;
+        self.skip_write.clear();
+        self.postings_write.clear();
+        self.bm25_weight = None;
+        Ok(())
+    }
+
+    fn clear(&mut self) {
+        self.block.clear();
+        self.last_doc_id_encoded = 0;
+    }
+}

src/postings/skip.rs

@@ -14,7 +14,11 @@ use crate::{DocId, Score, TERMINATED};
 //   (requiring a 6th bit), but the biggest doc_id we can want to encode is TERMINATED-1, which can
 //   be represented on 31b without delta encoding.
 fn encode_bitwidth(bitwidth: u8, delta_1: bool) -> u8 {
-    assert!(bitwidth < 32);
+    assert!(
+        bitwidth < 32,
+        "bitwidth needs to be less than 32, but got {}",
+        bitwidth
+    );
     bitwidth | ((delta_1 as u8) << 6)
 }
 

src/query/all_query.rs

@@ -21,7 +21,7 @@ impl Query for AllQuery {
 pub struct AllWeight;
 
 impl Weight for AllWeight {
-    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
+    fn scorer(&self, reader: &dyn SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
         let all_scorer = AllScorer::new(reader.max_doc());
         if boost != 1.0 {
             Ok(box_scorer(BoostScorer::new(all_scorer, boost)))
@@ -30,7 +30,7 @@ impl Weight for AllWeight {
         }
     }
 
-    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {
+    fn explain(&self, reader: &dyn SegmentReader, doc: DocId) -> crate::Result<Explanation> {
         if doc >= reader.max_doc() {
             return Err(does_not_match(doc));
         }

src/query/automaton_weight.rs

@@ -67,7 +67,7 @@ where
     }
 
     /// Returns the term infos that match the automaton
-    pub fn get_match_term_infos(&self, reader: &SegmentReader) -> crate::Result<Vec<TermInfo>> {
+    pub fn get_match_term_infos(&self, reader: &dyn SegmentReader) -> crate::Result<Vec<TermInfo>> {
         let inverted_index = reader.inverted_index(self.field)?;
         let term_dict = inverted_index.terms();
         let mut term_stream = self.automaton_stream(term_dict)?;
@@ -84,7 +84,7 @@ where
     A: Automaton + Send + Sync + 'static,
     A::State: Clone,
 {
-    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
+    fn scorer(&self, reader: &dyn SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
         let max_doc = reader.max_doc();
         let mut doc_bitset = BitSet::with_max_value(max_doc);
         let inverted_index = reader.inverted_index(self.field)?;
@@ -92,16 +92,18 @@ where
         let mut term_stream = self.automaton_stream(term_dict)?;
         while term_stream.advance() {
             let term_info = term_stream.value();
-            let mut block_segment_postings =
-                inverted_index.read_postings_from_terminfo(term_info, IndexRecordOption::Basic)?;
-            block_segment_postings.fill_bitset(&mut doc_bitset);
+            inverted_index.fill_bitset_for_term(
+                term_info,
+                IndexRecordOption::Basic,
+                &mut doc_bitset,
+            )?;
         }
         let doc_bitset = BitSetDocSet::from(doc_bitset);
         let const_scorer = ConstScorer::new(doc_bitset, boost);
         Ok(Box::new(const_scorer))
     }
 
-    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {
+    fn explain(&self, reader: &dyn SegmentReader, doc: DocId) -> crate::Result<Explanation> {
         let mut scorer = self.scorer(reader, 1.0)?;
         if scorer.seek(doc) == doc {
             Ok(Explanation::new("AutomatonScorer", 1.0))

src/query/boolean_query/boolean_weight.rs

@@ -1,11 +1,12 @@
 use std::collections::HashMap;
 
-use crate::codec::{ObjectSafeCodec, SumOrDoNothingCombiner};
+use crate::codec::SumOrDoNothingCombiner;
 use crate::docset::COLLECT_BLOCK_BUFFER_LEN;
 use crate::index::SegmentReader;
 use crate::query::disjunction::Disjunction;
 use crate::query::explanation::does_not_match;
 use crate::query::score_combiner::{DoNothingCombiner, ScoreCombiner};
+use crate::query::term_query::TermScorer;
 use crate::query::weight::for_each_docset_buffered;
 use crate::query::{
     box_scorer, intersect_scorers, AllScorer, BufferedUnionScorer, EmptyScorer, Exclude,
@@ -38,7 +39,7 @@ fn scorer_union<TScoreCombiner>(
     scorers: Vec<Box<dyn Scorer>>,
     score_combiner_fn: impl Fn() -> TScoreCombiner,
     num_docs: u32,
-    codec: &dyn ObjectSafeCodec,
+    reader: &dyn SegmentReader,
 ) -> Box<dyn Scorer>
 where
     TScoreCombiner: ScoreCombiner,
@@ -61,9 +62,7 @@ where
                 None
             };
             if let Some(combiner) = combiner_opt {
-                let scorer =
-                    codec.build_union_scorer_with_sum_combiner(scorers, num_docs, combiner);
-                scorer
+                reader.build_union_scorer_with_sum_combiner(scorers, num_docs, combiner)
             } else {
                 box_scorer(BufferedUnionScorer::build(
                     scorers,
@@ -181,7 +180,7 @@ impl<TScoreCombiner: ScoreCombiner> BooleanWeight<TScoreCombiner> {
 
     fn per_occur_scorers(
         &self,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
         boost: Score,
     ) -> crate::Result<HashMap<Occur, Vec<Box<dyn Scorer>>>> {
         let mut per_occur_scorers: HashMap<Occur, Vec<Box<dyn Scorer>>> = HashMap::new();
@@ -197,7 +196,7 @@ impl<TScoreCombiner: ScoreCombiner> BooleanWeight<TScoreCombiner> {
 
     fn complex_scorer<TComplexScoreCombiner: ScoreCombiner>(
         &self,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
         boost: Score,
         score_combiner_fn: impl Fn() -> TComplexScoreCombiner,
     ) -> crate::Result<Box<dyn Scorer>> {
@@ -245,13 +244,13 @@ impl<TScoreCombiner: ScoreCombiner> BooleanWeight<TScoreCombiner> {
                     should_scorers,
                     &score_combiner_fn,
                     num_docs,
-                    reader.codec(),
+                    reader,
                 )),
                 1 => ShouldScorersCombinationMethod::Required(scorer_union(
                     should_scorers,
                     &score_combiner_fn,
                     num_docs,
-                    reader.codec(),
+                    reader,
                 )),
                 n if num_of_should_scorers == n => {
                     // When num_of_should_scorers equals the number of should clauses,
@@ -267,18 +266,6 @@ impl<TScoreCombiner: ScoreCombiner> BooleanWeight<TScoreCombiner> {
             }
         };
 
-        let exclude_scorer_opt: Option<Box<dyn Scorer>> = if exclude_scorers.is_empty() {
-            None
-        } else {
-            let exclude_scorers_union: Box<dyn Scorer> = scorer_union(
-                exclude_scorers,
-                DoNothingCombiner::default,
-                num_docs,
-                reader.codec(),
-            );
-            Some(exclude_scorers_union)
-        };
-
         let include_scorer = match (should_scorers, must_scorers) {
             (ShouldScorersCombinationMethod::Ignored, must_scorers) => {
                 // No SHOULD clauses (or they were absorbed into MUST).
@@ -347,11 +334,22 @@ impl<TScoreCombiner: ScoreCombiner> BooleanWeight<TScoreCombiner> {
                 }
             }
         };
-        if let Some(exclude_scorer) = exclude_scorer_opt {
-            Ok(box_scorer(Exclude::new(include_scorer, exclude_scorer)))
-        } else {
-            Ok(include_scorer)
+        if exclude_scorers.is_empty() {
+            return Ok(include_scorer);
         }
+
+        let scorer: Box<dyn Scorer> = if exclude_scorers.len() == 1 {
+            let exclude_scorer = exclude_scorers.pop().unwrap();
+            match exclude_scorer.downcast::<TermScorer>() {
+                // Cast to TermScorer succeeded
+                Ok(exclude_scorer) => Box::new(Exclude::new(include_scorer, *exclude_scorer)),
+                // We get back the original Box<dyn Scorer>
+                Err(exclude_scorer) => Box::new(Exclude::new(include_scorer, exclude_scorer)),
+            }
+        } else {
+            Box::new(Exclude::new(include_scorer, exclude_scorers))
+        };
+        Ok(scorer)
     }
 }
 
@@ -380,7 +378,7 @@ fn remove_and_count_all_and_empty_scorers(
 }
 
 impl<TScoreCombiner: ScoreCombiner + Sync> Weight for BooleanWeight<TScoreCombiner> {
-    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
+    fn scorer(&self, reader: &dyn SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
         if self.weights.is_empty() {
             Ok(Box::new(EmptyScorer))
         } else if self.weights.len() == 1 {
@@ -397,7 +395,7 @@ impl<TScoreCombiner: ScoreCombiner + Sync> Weight for BooleanWeight<TScoreCombin
         }
     }
 
-    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {
+    fn explain(&self, reader: &dyn SegmentReader, doc: DocId) -> crate::Result<Explanation> {
         let mut scorer = self.scorer(reader, 1.0)?;
         if scorer.seek(doc) != doc {
             return Err(does_not_match(doc));
@@ -419,7 +417,7 @@ impl<TScoreCombiner: ScoreCombiner + Sync> Weight for BooleanWeight<TScoreCombin
 
     fn for_each(
         &self,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
         callback: &mut dyn FnMut(DocId, Score),
     ) -> crate::Result<()> {
         let mut scorer = self.complex_scorer(reader, 1.0, &self.score_combiner_fn)?;
@@ -429,7 +427,7 @@ impl<TScoreCombiner: ScoreCombiner + Sync> Weight for BooleanWeight<TScoreCombin
 
     fn for_each_no_score(
         &self,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
         callback: &mut dyn FnMut(&[DocId]),
     ) -> crate::Result<()> {
         let mut scorer = self.complex_scorer(reader, 1.0, || DoNothingCombiner)?;
@@ -451,11 +449,11 @@ impl<TScoreCombiner: ScoreCombiner + Sync> Weight for BooleanWeight<TScoreCombin
     fn for_each_pruning(
         &self,
         threshold: Score,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
         callback: &mut dyn FnMut(DocId, Score) -> Score,
     ) -> crate::Result<()> {
         let scorer = self.complex_scorer(reader, 1.0, &self.score_combiner_fn)?;
-        reader.codec().for_each_pruning(threshold, scorer, callback);
+        reader.for_each_pruning(threshold, scorer, callback);
         Ok(())
     }
 }

src/query/boost_query.rs

@@ -67,11 +67,11 @@ impl BoostWeight {
 }
 
 impl Weight for BoostWeight {
-    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
+    fn scorer(&self, reader: &dyn SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
         self.weight.scorer(reader, boost * self.boost)
     }
 
-    fn explain(&self, reader: &SegmentReader, doc: u32) -> crate::Result<Explanation> {
+    fn explain(&self, reader: &dyn SegmentReader, doc: u32) -> crate::Result<Explanation> {
         let underlying_explanation = self.weight.explain(reader, doc)?;
         let score = underlying_explanation.value() * self.boost;
         let mut explanation =
@@ -80,7 +80,7 @@ impl Weight for BoostWeight {
         Ok(explanation)
     }
 
-    fn count(&self, reader: &SegmentReader) -> crate::Result<u32> {
+    fn count(&self, reader: &dyn SegmentReader) -> crate::Result<u32> {
         self.weight.count(reader)
     }
 }

src/query/const_score_query.rs

@@ -63,7 +63,7 @@ impl ConstWeight {
 }
 
 impl Weight for ConstWeight {
-    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
+    fn scorer(&self, reader: &dyn SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
         let inner_scorer = self.weight.scorer(reader, boost)?;
         Ok(box_scorer(ConstScorer::new(
             inner_scorer,
@@ -71,7 +71,7 @@ impl Weight for ConstWeight {
         )))
     }
 
-    fn explain(&self, reader: &SegmentReader, doc: u32) -> crate::Result<Explanation> {
+    fn explain(&self, reader: &dyn SegmentReader, doc: u32) -> crate::Result<Explanation> {
         let mut scorer = self.scorer(reader, 1.0)?;
         if scorer.seek(doc) != doc {
             return Err(TantivyError::InvalidArgument(format!(
@@ -84,7 +84,7 @@ impl Weight for ConstWeight {
         Ok(explanation)
     }
 
-    fn count(&self, reader: &SegmentReader) -> crate::Result<u32> {
+    fn count(&self, reader: &dyn SegmentReader) -> crate::Result<u32> {
         self.weight.count(reader)
     }
 }

src/query/empty_query.rs

@@ -26,11 +26,11 @@ impl Query for EmptyQuery {
 /// It is useful for tests and handling edge cases.
 pub struct EmptyWeight;
 impl Weight for EmptyWeight {
-    fn scorer(&self, _reader: &SegmentReader, _boost: Score) -> crate::Result<Box<dyn Scorer>> {
+    fn scorer(&self, _reader: &dyn SegmentReader, _boost: Score) -> crate::Result<Box<dyn Scorer>> {
         Ok(box_scorer(EmptyScorer))
     }
 
-    fn explain(&self, _reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {
+    fn explain(&self, _reader: &dyn SegmentReader, doc: DocId) -> crate::Result<Explanation> {
         Err(does_not_match(doc))
     }
 }

src/query/exclude.rs

@@ -1,48 +1,71 @@
-use crate::docset::{DocSet, TERMINATED};
+use crate::docset::{DocSet, SeekDangerResult, TERMINATED};
 use crate::query::Scorer;
 use crate::{DocId, Score};
 
-#[inline]
-fn is_within<TDocSetExclude: DocSet>(docset: &mut TDocSetExclude, doc: DocId) -> bool {
-    docset.doc() <= doc && docset.seek(doc) == doc
-}
-
-/// Filters a given `DocSet` by removing the docs from a given `DocSet`.
+/// An exclusion set is a set of documents
+/// that should be excluded from a given DocSet.
 ///
-/// The excluding docset has no impact on scoring.
-pub struct Exclude<TDocSet, TDocSetExclude> {
-    underlying_docset: TDocSet,
-    excluding_docset: TDocSetExclude,
+/// It can be a single DocSet, or a Vec of DocSets.
+pub trait ExclusionSet: Send {
+    /// Returns `true` if the given `doc` is in the exclusion set.
+    fn contains(&mut self, doc: DocId) -> bool;
 }
 
-impl<TDocSet, TDocSetExclude> Exclude<TDocSet, TDocSetExclude>
+impl<TDocSet: DocSet> ExclusionSet for TDocSet {
+    #[inline]
+    fn contains(&mut self, doc: DocId) -> bool {
+        self.seek_danger(doc) == SeekDangerResult::Found
+    }
+}
+
+impl<TDocSet: DocSet> ExclusionSet for Vec<TDocSet> {
+    #[inline]
+    fn contains(&mut self, doc: DocId) -> bool {
+        for docset in self.iter_mut() {
+            if docset.seek_danger(doc) == SeekDangerResult::Found {
+                return true;
+            }
+        }
+        false
+    }
+}
+
+/// Filters a given `DocSet` by removing the docs from an exclusion set.
+///
+/// The excluding docsets have no impact on scoring.
+pub struct Exclude<TDocSet, TExclusionSet> {
+    underlying_docset: TDocSet,
+    exclusion_set: TExclusionSet,
+}
+
+impl<TDocSet, TExclusionSet> Exclude<TDocSet, TExclusionSet>
 where
     TDocSet: DocSet,
-    TDocSetExclude: DocSet,
+    TExclusionSet: ExclusionSet,
 {
     /// Creates a new `ExcludeScorer`
     pub fn new(
         mut underlying_docset: TDocSet,
-        mut excluding_docset: TDocSetExclude,
-    ) -> Exclude<TDocSet, TDocSetExclude> {
+        mut exclusion_set: TExclusionSet,
+    ) -> Exclude<TDocSet, TExclusionSet> {
         while underlying_docset.doc() != TERMINATED {
             let target = underlying_docset.doc();
-            if !is_within(&mut excluding_docset, target) {
+            if !exclusion_set.contains(target) {
                 break;
             }
             underlying_docset.advance();
         }
         Exclude {
             underlying_docset,
-            excluding_docset,
+            exclusion_set,
         }
     }
 }
 
-impl<TDocSet, TDocSetExclude> DocSet for Exclude<TDocSet, TDocSetExclude>
+impl<TDocSet, TExclusionSet> DocSet for Exclude<TDocSet, TExclusionSet>
 where
     TDocSet: DocSet,
-    TDocSetExclude: DocSet,
+    TExclusionSet: ExclusionSet,
 {
     fn advance(&mut self) -> DocId {
         loop {
@@ -50,7 +73,7 @@ where
             if candidate == TERMINATED {
                 return TERMINATED;
             }
-            if !is_within(&mut self.excluding_docset, candidate) {
+            if !self.exclusion_set.contains(candidate) {
                 return candidate;
             }
         }
@@ -61,7 +84,7 @@ where
         if candidate == TERMINATED {
             return TERMINATED;
         }
-        if !is_within(&mut self.excluding_docset, candidate) {
+        if !self.exclusion_set.contains(candidate) {
             return candidate;
         }
         self.advance()
@@ -79,10 +102,10 @@ where
     }
 }
 
-impl<TScorer, TDocSetExclude> Scorer for Exclude<TScorer, TDocSetExclude>
+impl<TScorer, TExclusionSet> Scorer for Exclude<TScorer, TExclusionSet>
 where
     TScorer: Scorer,
-    TDocSetExclude: DocSet + 'static,
+    TExclusionSet: ExclusionSet + 'static,
 {
     #[inline]
     fn score(&mut self) -> Score {

src/query/exist_query.rs

@@ -98,7 +98,7 @@ pub struct ExistsWeight {
 }
 
 impl Weight for ExistsWeight {
-    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
+    fn scorer(&self, reader: &dyn SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
         let fast_field_reader = reader.fast_fields();
         let mut column_handles = fast_field_reader.dynamic_column_handles(&self.field_name)?;
         if self.field_type == Type::Json && self.json_subpaths {
@@ -165,7 +165,7 @@ impl Weight for ExistsWeight {
         Ok(box_scorer(ConstScorer::new(docset, boost)))
     }
 
-    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {
+    fn explain(&self, reader: &dyn SegmentReader, doc: DocId) -> crate::Result<Explanation> {
         let mut scorer = self.scorer(reader, 1.0)?;
         if scorer.seek(doc) != doc {
             return Err(does_not_match(doc));

src/query/mod.rs

@@ -43,7 +43,7 @@ pub use self::boost_query::{BoostQuery, BoostWeight};
 pub use self::const_score_query::{ConstScoreQuery, ConstScorer};
 pub use self::disjunction_max_query::DisjunctionMaxQuery;
 pub use self::empty_query::{EmptyQuery, EmptyScorer, EmptyWeight};
-pub use self::exclude::Exclude;
+pub use self::exclude::{Exclude, ExclusionSet};
 pub use self::exist_query::ExistsQuery;
 pub use self::explanation::Explanation;
 #[cfg(test)]
@@ -53,8 +53,7 @@ pub use self::intersection::{intersect_scorers, Intersection};
 pub use self::more_like_this::{MoreLikeThisQuery, MoreLikeThisQueryBuilder};
 pub use self::phrase_prefix_query::PhrasePrefixQuery;
 pub use self::phrase_query::regex_phrase_query::{wildcard_query_to_regex_str, RegexPhraseQuery};
-pub use self::phrase_query::PhraseQuery;
-pub(crate) use self::phrase_query::PhraseScorer;
+pub use self::phrase_query::{PhraseQuery, PhraseScorer};
 pub use self::query::{EnableScoring, Query, QueryClone};
 pub use self::query_parser::{QueryParser, QueryParserError};
 pub use self::range_query::*;
@@ -63,8 +62,8 @@ pub use self::reqopt_scorer::RequiredOptionalScorer;
 pub use self::score_combiner::{DisjunctionMaxCombiner, ScoreCombiner, SumCombiner};
 pub use self::scorer::{box_scorer, Scorer};
 pub use self::set_query::TermSetQuery;
-pub use self::term_query::TermQuery;
-pub use self::union::BufferedUnionScorer;
+pub use self::term_query::{TermQuery, TermScorer};
+pub use self::union::{BufferedUnionScorer, SimpleUnion};
 #[cfg(test)]
 pub use self::vec_docset::VecDocSet;
 pub use self::weight::Weight;

src/query/more_like_this/more_like_this.rs

@@ -8,7 +8,7 @@ use crate::query::{BooleanQuery, BoostQuery, Occur, Query, TermQuery};
 use crate::schema::document::{Document, Value};
 use crate::schema::{Field, FieldType, IndexRecordOption, Term};
 use crate::tokenizer::{FacetTokenizer, PreTokenizedStream, TokenStream, Tokenizer};
-use crate::{DocAddress, Result, Searcher, TantivyDocument, TantivyError};
+use crate::{DocAddress, Result, Searcher, TantivyError};
 
 #[derive(Debug, PartialEq)]
 struct ScoreTerm {
@@ -129,7 +129,7 @@ impl MoreLikeThis {
         searcher: &Searcher,
         doc_address: DocAddress,
     ) -> Result<Vec<ScoreTerm>> {
-        let doc = searcher.doc::<TantivyDocument>(doc_address)?;
+        let doc = searcher.doc(doc_address)?;
 
         let field_to_values = doc.get_sorted_field_values();
         self.retrieve_terms_from_doc_fields(searcher, &field_to_values)
@@ -167,7 +167,7 @@ impl MoreLikeThis {
         term_frequencies: &mut HashMap<Term, usize>,
     ) -> Result<()> {
         let schema = searcher.schema();
-        let tokenizer_manager = searcher.index().tokenizers();
+        let tokenizer_manager = searcher.tokenizers();
 
         let field_entry = schema.get_field_entry(field);
         if !field_entry.is_indexed() {

src/query/phrase_prefix_query/phrase_prefix_weight.rs

@@ -31,10 +31,10 @@ impl PhrasePrefixWeight {
         }
     }
 
-    fn fieldnorm_reader(&self, reader: &SegmentReader) -> crate::Result<FieldNormReader> {
+    fn fieldnorm_reader(&self, reader: &dyn SegmentReader) -> crate::Result<FieldNormReader> {
         let field = self.phrase_terms[0].1.field();
         if self.similarity_weight_opt.is_some() {
-            if let Some(fieldnorm_reader) = reader.fieldnorms_readers().get_field(field)? {
+            if let Ok(fieldnorm_reader) = reader.get_fieldnorms_reader(field) {
                 return Ok(fieldnorm_reader);
             }
         }
@@ -43,7 +43,7 @@ impl PhrasePrefixWeight {
 
     pub(crate) fn phrase_scorer(
         &self,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
         boost: Score,
     ) -> crate::Result<Option<Box<dyn Scorer>>> {
         let similarity_weight_opt = self
@@ -113,7 +113,7 @@ impl PhrasePrefixWeight {
 }
 
 impl Weight for PhrasePrefixWeight {
-    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
+    fn scorer(&self, reader: &dyn SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
         if let Some(scorer) = self.phrase_scorer(reader, boost)? {
             Ok(scorer)
         } else {

src/query/phrase_query/phrase_scorer.rs

@@ -44,6 +44,13 @@ impl<TPostings: Postings> DocSet for PostingsWithOffset<TPostings> {
     }
 }
 
+/// `PhraseScorer` is a `Scorer` that matches documents that match a phrase query, and scores them
+/// based on the number of times the phrase appears in the document and the fieldnorm of the
+/// document.
+///
+/// It is implemented as an intersection of the postings of each term in the
+/// phrase, where the intersection condition is that the positions of the terms are next to each
+/// other (or within a certain slop).
 pub struct PhraseScorer<TPostings: Postings = StandardPostings> {
     intersection_docset: Intersection<PostingsWithOffset<TPostings>, PostingsWithOffset<TPostings>>,
     num_terms: usize,
@@ -347,6 +354,9 @@ fn intersection_count_with_carrying_slop(
 
 impl<TPostings: Postings> PhraseScorer<TPostings> {
     // If similarity_weight is None, then scoring is disabled.
+    /// Creates a phrase scorer from term postings and phrase matching options.
+    ///
+    /// `slop` controls the maximum positional distance allowed between terms.
     pub fn new(
         term_postings: Vec<(usize, TPostings)>,
         similarity_weight_opt: Option<Bm25Weight>,

src/query/phrase_query/phrase_weight.rs

@@ -27,10 +27,10 @@ impl PhraseWeight {
         }
     }
 
-    fn fieldnorm_reader(&self, reader: &SegmentReader) -> crate::Result<FieldNormReader> {
+    fn fieldnorm_reader(&self, reader: &dyn SegmentReader) -> crate::Result<FieldNormReader> {
         let field = self.phrase_terms[0].1.field();
         if self.similarity_weight_opt.is_some() {
-            if let Some(fieldnorm_reader) = reader.fieldnorms_readers().get_field(field)? {
+            if let Ok(fieldnorm_reader) = reader.get_fieldnorms_reader(field) {
                 return Ok(fieldnorm_reader);
             }
         }
@@ -39,7 +39,7 @@ impl PhraseWeight {
 
     pub(crate) fn phrase_scorer(
         &self,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
         boost: Score,
     ) -> crate::Result<Option<Box<dyn Scorer>>> {
         let similarity_weight_opt = self
@@ -74,12 +74,11 @@ impl PhraseWeight {
             term_infos.push((offset, term_info));
         }
 
-        let scorer = reader.codec().new_phrase_scorer_type_erased(
+        let scorer = inverted_index_reader.new_phrase_scorer(
             &term_infos[..],
             similarity_weight_opt,
             fieldnorm_reader,
             self.slop,
-            &inverted_index_reader,
         )?;
 
         Ok(Some(scorer))
@@ -92,7 +91,7 @@ impl PhraseWeight {
 }
 
 impl Weight for PhraseWeight {
-    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
+    fn scorer(&self, reader: &dyn SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
         if let Some(scorer) = self.phrase_scorer(reader, boost)? {
             Ok(scorer)
         } else {
@@ -100,7 +99,7 @@ impl Weight for PhraseWeight {
         }
     }
 
-    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {
+    fn explain(&self, reader: &dyn SegmentReader, doc: DocId) -> crate::Result<Explanation> {
         let scorer_opt = self.phrase_scorer(reader, 1.0)?;
         if scorer_opt.is_none() {
             return Err(does_not_match(doc));

src/query/phrase_query/regex_phrase_weight.rs

@@ -47,9 +47,9 @@ impl RegexPhraseWeight {
         }
     }
 
-    fn fieldnorm_reader(&self, reader: &SegmentReader) -> crate::Result<FieldNormReader> {
+    fn fieldnorm_reader(&self, reader: &dyn SegmentReader) -> crate::Result<FieldNormReader> {
         if self.similarity_weight_opt.is_some() {
-            if let Some(fieldnorm_reader) = reader.fieldnorms_readers().get_field(self.field)? {
+            if let Ok(fieldnorm_reader) = reader.get_fieldnorms_reader(self.field) {
                 return Ok(fieldnorm_reader);
             }
         }
@@ -58,7 +58,7 @@ impl RegexPhraseWeight {
 
     pub(crate) fn phrase_scorer(
         &self,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
         boost: Score,
     ) -> crate::Result<Option<PhraseScorer<UnionType>>> {
         let similarity_weight_opt = self
@@ -86,7 +86,8 @@ impl RegexPhraseWeight {
                     "Phrase query exceeded max expansions {num_terms}"
                 )));
             }
-            let union = Self::get_union_from_term_infos(&term_infos, reader, &inverted_index)?;
+            let union =
+                Self::get_union_from_term_infos(&term_infos, reader, inverted_index.as_ref())?;
 
             posting_lists.push((offset, union));
         }
@@ -101,13 +102,11 @@ impl RegexPhraseWeight {
 
     /// Add all docs of the term to the docset
     fn add_to_bitset(
-        inverted_index: &InvertedIndexReader,
+        inverted_index: &dyn InvertedIndexReader,
         term_info: &TermInfo,
         doc_bitset: &mut BitSet,
     ) -> crate::Result<()> {
-        let mut segment_postings =
-            inverted_index.read_postings_from_terminfo(term_info, IndexRecordOption::Basic)?;
-        segment_postings.fill_bitset(doc_bitset);
+        inverted_index.fill_bitset_for_term(term_info, IndexRecordOption::Basic, doc_bitset)?;
         Ok(())
     }
 
@@ -167,8 +166,8 @@ impl RegexPhraseWeight {
     /// Use Roaring Bitmaps for sparse terms. The full bitvec is main memory consumer currently.
     pub(crate) fn get_union_from_term_infos(
         term_infos: &[TermInfo],
-        reader: &SegmentReader,
-        inverted_index: &InvertedIndexReader,
+        reader: &dyn SegmentReader,
+        inverted_index: &dyn InvertedIndexReader,
     ) -> crate::Result<UnionType> {
         let max_doc = reader.max_doc();
 
@@ -262,7 +261,7 @@ impl RegexPhraseWeight {
 }
 
 impl Weight for RegexPhraseWeight {
-    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
+    fn scorer(&self, reader: &dyn SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
         if let Some(scorer) = self.phrase_scorer(reader, boost)? {
             Ok(box_scorer(scorer))
         } else {
@@ -270,7 +269,7 @@ impl Weight for RegexPhraseWeight {
         }
     }
 
-    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {
+    fn explain(&self, reader: &dyn SegmentReader, doc: DocId) -> crate::Result<Explanation> {
         let scorer_opt = self.phrase_scorer(reader, 1.0)?;
         if scorer_opt.is_none() {
             return Err(does_not_match(doc));

src/query/query.rs

@@ -146,7 +146,7 @@ pub trait Query: QueryClone + Send + Sync + downcast_rs::Downcast + fmt::Debug {
         let weight = self.weight(EnableScoring::disabled_from_searcher(searcher))?;
         let mut result = 0;
         for reader in searcher.segment_readers() {
-            result += weight.count(reader)? as usize;
+            result += weight.count(reader.as_ref())? as usize;
         }
         Ok(result)
     }

src/query/query_parser/query_parser.rs

@@ -2068,6 +2068,16 @@ mod test {
             format!("Regex(Field(0), {:#?})", expected_regex).as_str(),
             false,
         );
+        let expected_regex2 = tantivy_fst::Regex::new(r".*a").unwrap();
+        test_parse_query_to_logical_ast_helper(
+            "title:(/.*b/ OR /.*a/)",
+            format!(
+                "(Regex(Field(0), {:#?}) Regex(Field(0), {:#?}))",
+                expected_regex, expected_regex2
+            )
+            .as_str(),
+            false,
+        );
 
         // Invalid field
         let err = parse_query_to_logical_ast("float:/.*b/", false).unwrap_err();

src/query/range_query/mod.rs

@@ -19,7 +19,8 @@ pub(crate) fn is_type_valid_for_fastfield_range_query(typ: Type) -> bool {
         | Type::Bool
         | Type::Date
         | Type::Json
-        | Type::IpAddr => true,
-        Type::Facet | Type::Bytes => false,
+        | Type::IpAddr
+        | Type::Bytes => true,
+        Type::Facet => false,
     }
 }

src/query/range_query/range_query.rs

@@ -214,7 +214,7 @@ impl InvertedIndexRangeWeight {
 }
 
 impl Weight for InvertedIndexRangeWeight {
-    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
+    fn scorer(&self, reader: &dyn SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
         let max_doc = reader.max_doc();
         let mut doc_bitset = BitSet::with_max_value(max_doc);
 
@@ -230,15 +230,17 @@ impl Weight for InvertedIndexRangeWeight {
             }
             processed_count += 1;
             let term_info = term_range.value();
-            let mut postings =
-                inverted_index.read_postings_from_terminfo(term_info, IndexRecordOption::Basic)?;
-            postings.fill_bitset(&mut doc_bitset);
+            inverted_index.fill_bitset_for_term(
+                term_info,
+                IndexRecordOption::Basic,
+                &mut doc_bitset,
+            )?;
         }
         let doc_bitset = BitSetDocSet::from(doc_bitset);
         Ok(box_scorer(ConstScorer::new(doc_bitset, boost)))
     }
 
-    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {
+    fn explain(&self, reader: &dyn SegmentReader, doc: DocId) -> crate::Result<Explanation> {
         let mut scorer = self.scorer(reader, 1.0)?;
         if scorer.seek(doc) != doc {
             return Err(does_not_match(doc));
@@ -679,7 +681,7 @@ mod tests {
                 .weight(EnableScoring::disabled_from_schema(&schema))
                 .unwrap();
             let range_scorer = range_weight
-                .scorer(&searcher.segment_readers()[0], 1.0f32)
+                .scorer(searcher.segment_readers()[0].as_ref(), 1.0f32)
                 .unwrap();
             range_scorer
         };

src/query/range_query/range_query_fastfield.rs

@@ -6,8 +6,8 @@ use std::net::Ipv6Addr;
 use std::ops::{Bound, RangeInclusive};
 
 use columnar::{
-    Cardinality, Column, ColumnType, MonotonicallyMappableToU128, MonotonicallyMappableToU64,
-    NumericalType, StrColumn,
+    BytesColumn, Cardinality, Column, ColumnType, MonotonicallyMappableToU128,
+    MonotonicallyMappableToU64, NumericalType, StrColumn,
 };
 use common::bounds::{BoundsRange, TransformBound};
 
@@ -53,7 +53,7 @@ impl FastFieldRangeWeight {
 }
 
 impl Weight for FastFieldRangeWeight {
-    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
+    fn scorer(&self, reader: &dyn SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
         // Check if both bounds are Bound::Unbounded
         if self.bounds.is_unbounded() {
             return Ok(box_scorer(AllScorer::new(reader.max_doc())));
@@ -175,6 +175,25 @@ impl Weight for FastFieldRangeWeight {
                 return Ok(box_scorer(EmptyScorer));
             };
             search_on_u64_ff(column, boost, BoundsRange::new(lower_bound, upper_bound))
+        } else if field_type.is_bytes() {
+            let Some(bytes_column): Option<BytesColumn> =
+                reader.fast_fields().bytes(&field_name)?
+            else {
+                return Ok(Box::new(EmptyScorer));
+            };
+            let dict = bytes_column.dictionary();
+
+            let bounds = self.bounds.map_bound(get_value_bytes);
+            // Get term ids for terms
+            let (lower_bound, upper_bound) =
+                dict.term_bounds_to_ord(bounds.lower_bound, bounds.upper_bound)?;
+            let fast_field_reader = reader.fast_fields();
+            let Some((column, _col_type)) =
+                fast_field_reader.u64_lenient_for_type(None, &field_name)?
+            else {
+                return Ok(Box::new(EmptyScorer));
+            };
+            search_on_u64_ff(column, boost, BoundsRange::new(lower_bound, upper_bound))
         } else {
             assert!(
                 maps_to_u64_fastfield(field_type.value_type()),
@@ -220,7 +239,7 @@ impl Weight for FastFieldRangeWeight {
         }
     }
 
-    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {
+    fn explain(&self, reader: &dyn SegmentReader, doc: DocId) -> crate::Result<Explanation> {
         let mut scorer = self.scorer(reader, 1.0)?;
         if scorer.seek(doc) != doc {
             return Err(TantivyError::InvalidArgument(format!(
@@ -237,7 +256,7 @@ impl Weight for FastFieldRangeWeight {
 ///
 /// Convert into fast field value space and search.
 fn search_on_json_numerical_field(
-    reader: &SegmentReader,
+    reader: &dyn SegmentReader,
     field_name: &str,
     typ: Type,
     bounds: BoundsRange<ValueBytes<Vec<u8>>>,
@@ -1403,6 +1422,66 @@ mod tests {
 
         Ok(())
     }
+
+    #[test]
+    fn test_bytes_field_ff_range_query() -> crate::Result<()> {
+        use crate::schema::BytesOptions;
+
+        let mut schema_builder = Schema::builder();
+        let bytes_field = schema_builder
+            .add_bytes_field("data", BytesOptions::default().set_fast().set_indexed());
+        let schema = schema_builder.build();
+        let index = Index::create_in_ram(schema.clone());
+        let mut index_writer: IndexWriter = index.writer_for_tests()?;
+
+        // Insert documents with lexicographically sortable byte values
+        // Using simple byte sequences that have clear ordering
+        let values: Vec<Vec<u8>> = vec![
+            vec![0x00, 0x10],
+            vec![0x00, 0x20],
+            vec![0x00, 0x30],
+            vec![0x01, 0x00],
+            vec![0x01, 0x10],
+            vec![0x02, 0x00],
+        ];
+
+        for value in &values {
+            let mut doc = TantivyDocument::new();
+            doc.add_bytes(bytes_field, value);
+            index_writer.add_document(doc)?;
+        }
+        index_writer.commit()?;
+
+        let reader = index.reader()?;
+        let searcher = reader.searcher();
+
+        // Test: Range query [0x00, 0x20] to [0x01, 0x00] (inclusive)
+        // Should match: [0x00, 0x20], [0x00, 0x30], [0x01, 0x00]
+        let lower = Term::from_field_bytes(bytes_field, &[0x00, 0x20]);
+        let upper = Term::from_field_bytes(bytes_field, &[0x01, 0x00]);
+        let range_query = RangeQuery::new(Bound::Included(lower), Bound::Included(upper));
+        let count = searcher.search(&range_query, &Count)?;
+        assert_eq!(
+            count, 3,
+            "Expected 3 documents in range [0x00,0x20] to [0x01,0x00]"
+        );
+
+        // Test: Range query > [0x01, 0x00] (exclusive lower bound)
+        // Should match: [0x01, 0x10], [0x02, 0x00]
+        let lower = Term::from_field_bytes(bytes_field, &[0x01, 0x00]);
+        let range_query = RangeQuery::new(Bound::Excluded(lower), Bound::Unbounded);
+        let count = searcher.search(&range_query, &Count)?;
+        assert_eq!(count, 2, "Expected 2 documents > [0x01,0x00]");
+
+        // Test: Range query < [0x00, 0x30] (exclusive upper bound)
+        // Should match: [0x00, 0x10], [0x00, 0x20]
+        let upper = Term::from_field_bytes(bytes_field, &[0x00, 0x30]);
+        let range_query = RangeQuery::new(Bound::Unbounded, Bound::Excluded(upper));
+        let count = searcher.search(&range_query, &Count)?;
+        assert_eq!(count, 2, "Expected 2 documents < [0x00,0x30]");
+
+        Ok(())
+    }
 }
 
 #[cfg(test)]

src/query/term_query/term_scorer.rs

@@ -8,6 +8,10 @@ use crate::query::{Explanation, Scorer};
 use crate::{DocId, Score};
 
 #[derive(Clone)]
+/// Scorer for a single term over a postings list.
+///
+/// `TermScorer` combines postings data, fieldnorms, and BM25 term weight to
+/// produce per-document scores.
 pub struct TermScorer<
     TPostings: Postings = <<StandardCodec as Codec>::PostingsCodec as PostingsCodec>::Postings,
 > {
@@ -17,6 +21,8 @@ pub struct TermScorer<
 }
 
 impl<TPostings: Postings> TermScorer<TPostings> {
+    /// Creates a new term scorer from postings, fieldnorm reader, and BM25
+    /// term weight.
     pub fn new(
         postings: TPostings,
         fieldnorm_reader: FieldNormReader,
@@ -29,14 +35,17 @@ impl<TPostings: Postings> TermScorer<TPostings> {
         }
     }
 
+    /// Returns the term frequency for the current document.
     pub fn term_freq(&self) -> u32 {
         self.postings.term_freq()
     }
 
+    /// Returns the fieldnorm id for the current document.
     pub fn fieldnorm_id(&self) -> u8 {
         self.fieldnorm_reader.fieldnorm_id(self.doc())
     }
 
+    /// Returns the maximum score upper bound for this scorer.
     pub fn max_score(&self) -> Score {
         self.similarity_weight.max_score()
     }
@@ -252,7 +261,9 @@ mod tests {
             let mut block_max_scores_b = vec![];
             let mut docs = vec![];
             {
-                let mut term_scorer = term_weight.term_scorer_for_test(reader, 1.0).unwrap();
+                let mut term_scorer = term_weight
+                    .term_scorer_for_test(reader.as_ref(), 1.0)
+                    .unwrap();
                 while term_scorer.doc() != TERMINATED {
                     let mut score = term_scorer.score();
                     docs.push(term_scorer.doc());
@@ -266,7 +277,9 @@ mod tests {
                 }
             }
             {
-                let mut term_scorer = term_weight.term_scorer_for_test(reader, 1.0).unwrap();
+                let mut term_scorer = term_weight
+                    .term_scorer_for_test(reader.as_ref(), 1.0)
+                    .unwrap();
                 for d in docs {
                     let block_max_score = term_scorer.seek_block_max(d);
                     block_max_scores_b.push(block_max_score);

src/query/term_query/term_weight.rs

@@ -1,9 +1,8 @@
 use crate::docset::{DocSet, COLLECT_BLOCK_BUFFER_LEN};
 use crate::fieldnorm::FieldNormReader;
-use crate::index::SegmentReader;
+use crate::index::{BoxedTermScorer, SegmentReader};
 use crate::query::bm25::Bm25Weight;
 use crate::query::explanation::does_not_match;
-use crate::query::term_query::TermScorer;
 use crate::query::weight::for_each_docset_buffered;
 use crate::query::{box_scorer, AllScorer, AllWeight, EmptyScorer, Explanation, Scorer, Weight};
 use crate::schema::IndexRecordOption;
@@ -17,7 +16,7 @@ pub struct TermWeight {
 }
 
 enum TermOrEmptyOrAllScorer {
-    TermScorer(Box<dyn Scorer>),
+    TermScorer(BoxedTermScorer),
     Empty,
     AllMatch(AllScorer),
 }
@@ -25,7 +24,7 @@ enum TermOrEmptyOrAllScorer {
 impl TermOrEmptyOrAllScorer {
     pub fn into_boxed_scorer(self) -> Box<dyn Scorer> {
         match self {
-            TermOrEmptyOrAllScorer::TermScorer(scorer) => scorer,
+            TermOrEmptyOrAllScorer::TermScorer(scorer) => scorer.into_boxed_scorer(),
             TermOrEmptyOrAllScorer::Empty => box_scorer(EmptyScorer),
             TermOrEmptyOrAllScorer::AllMatch(scorer) => box_scorer(scorer),
         }
@@ -33,17 +32,17 @@ impl TermOrEmptyOrAllScorer {
 }
 
 impl Weight for TermWeight {
-    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
+    fn scorer(&self, reader: &dyn SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>> {
         Ok(self.specialized_scorer(reader, boost)?.into_boxed_scorer())
     }
 
-    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {
+    fn explain(&self, reader: &dyn SegmentReader, doc: DocId) -> crate::Result<Explanation> {
         match self.specialized_scorer(reader, 1.0)? {
-            TermOrEmptyOrAllScorer::TermScorer(mut term_scorer) => {
+            TermOrEmptyOrAllScorer::TermScorer(term_scorer) => {
+                let mut term_scorer = term_scorer.into_boxed_scorer();
                 if term_scorer.doc() > doc || term_scorer.seek(doc) != doc {
                     return Err(does_not_match(doc));
                 }
-                let mut term_scorer = term_scorer.downcast::<TermScorer>().ok().unwrap();
                 let mut explanation = term_scorer.explain();
                 explanation.add_context(format!("Term={:?}", self.term,));
                 Ok(explanation)
@@ -53,7 +52,7 @@ impl Weight for TermWeight {
         }
     }
 
-    fn count(&self, reader: &SegmentReader) -> crate::Result<u32> {
+    fn count(&self, reader: &dyn SegmentReader) -> crate::Result<u32> {
         if let Some(alive_bitset) = reader.alive_bitset() {
             Ok(self.scorer(reader, 1.0)?.count(alive_bitset))
         } else {
@@ -68,11 +67,12 @@ impl Weight for TermWeight {
     /// `DocSet` and push the scored documents to the collector.
     fn for_each(
         &self,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
         callback: &mut dyn FnMut(DocId, Score),
     ) -> crate::Result<()> {
         match self.specialized_scorer(reader, 1.0)? {
-            TermOrEmptyOrAllScorer::TermScorer(mut term_scorer) => {
+            TermOrEmptyOrAllScorer::TermScorer(term_scorer) => {
+                let mut term_scorer = term_scorer.into_boxed_scorer();
                 term_scorer.for_each(callback);
             }
             TermOrEmptyOrAllScorer::Empty => {}
@@ -87,11 +87,12 @@ impl Weight for TermWeight {
     /// `DocSet` and push the scored documents to the collector.
     fn for_each_no_score(
         &self,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
         callback: &mut dyn FnMut(&[DocId]),
     ) -> crate::Result<()> {
         match self.specialized_scorer(reader, 1.0)? {
-            TermOrEmptyOrAllScorer::TermScorer(mut term_scorer) => {
+            TermOrEmptyOrAllScorer::TermScorer(term_scorer) => {
+                let mut term_scorer = term_scorer.into_boxed_scorer();
                 let mut buffer = [0u32; COLLECT_BLOCK_BUFFER_LEN];
                 for_each_docset_buffered(&mut term_scorer, &mut buffer, callback);
             }
@@ -118,15 +119,13 @@ impl Weight for TermWeight {
     fn for_each_pruning(
         &self,
         threshold: Score,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
         callback: &mut dyn FnMut(DocId, Score) -> Score,
     ) -> crate::Result<()> {
         let specialized_scorer = self.specialized_scorer(reader, 1.0)?;
         match specialized_scorer {
             TermOrEmptyOrAllScorer::TermScorer(term_scorer) => {
-                reader
-                    .codec()
-                    .for_each_pruning(threshold, term_scorer, callback);
+                reader.for_each_pruning(threshold, term_scorer.into_boxed_scorer(), callback);
             }
             TermOrEmptyOrAllScorer::Empty => {}
             TermOrEmptyOrAllScorer::AllMatch(_) => {
@@ -164,13 +163,16 @@ impl TermWeight {
     #[cfg(test)]
     pub(crate) fn term_scorer_for_test(
         &self,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
         boost: Score,
     ) -> Option<super::TermScorer> {
         let scorer = self.specialized_scorer(reader, boost).unwrap();
         match scorer {
-            TermOrEmptyOrAllScorer::TermScorer(scorer) => {
-                let term_scorer = scorer.downcast::<super::TermScorer>().ok()?;
+            TermOrEmptyOrAllScorer::TermScorer(term_scorer) => {
+                let term_scorer = term_scorer
+                    .into_boxed_scorer()
+                    .downcast::<super::TermScorer>()
+                    .ok()?;
                 Some(*term_scorer)
             }
             _ => None,
@@ -179,7 +181,7 @@ impl TermWeight {
 
     fn specialized_scorer(
         &self,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
         boost: Score,
     ) -> crate::Result<TermOrEmptyOrAllScorer> {
         let field = self.term.field();
@@ -209,12 +211,12 @@ impl TermWeight {
         Ok(TermOrEmptyOrAllScorer::TermScorer(term_scorer))
     }
 
-    fn fieldnorm_reader(&self, segment_reader: &SegmentReader) -> crate::Result<FieldNormReader> {
+    fn fieldnorm_reader(
+        &self,
+        segment_reader: &dyn SegmentReader,
+    ) -> crate::Result<FieldNormReader> {
         if self.scoring_enabled {
-            if let Some(field_norm_reader) = segment_reader
-                .fieldnorms_readers()
-                .get_field(self.term.field())?
-            {
+            if let Ok(field_norm_reader) = segment_reader.get_fieldnorms_reader(self.term.field()) {
                 return Ok(field_norm_reader);
             }
         }

src/query/union/buffered_union.rs

@@ -275,8 +275,8 @@ where
         let mut is_hit = false;
         let mut min_new_target = TERMINATED;
 
-        for scorer in self.scorers.iter_mut() {
-            match scorer.seek_danger(target) {
+        for docset in self.scorers.iter_mut() {
+            match docset.seek_danger(target) {
                 SeekDangerResult::Found => {
                     is_hit = true;
                     break;

src/query/union/simple_union.rs

@@ -12,7 +12,11 @@ pub struct SimpleUnion<TDocSet> {
 }
 
 impl<TDocSet: DocSet> SimpleUnion<TDocSet> {
-    pub(crate) fn build(mut docsets: Vec<TDocSet>) -> SimpleUnion<TDocSet> {
+    /// Builds a `SimpleUnion` from multiple docsets.
+    ///
+    /// Exhausted docsets are filtered out, and the union is initialized at the
+    /// smallest current doc id across remaining docsets.
+    pub fn build(mut docsets: Vec<TDocSet>) -> SimpleUnion<TDocSet> {
         docsets.retain(|docset| docset.doc() != TERMINATED);
         let mut docset = SimpleUnion { docsets, doc: 0 };
 

src/query/weight.rs

@@ -32,10 +32,10 @@ pub trait Weight: Send + Sync + 'static {
     /// `boost` is a multiplier to apply to the score.
     ///
     /// See [`Query`](crate::query::Query).
-    fn scorer(&self, reader: &SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>>;
+    fn scorer(&self, reader: &dyn SegmentReader, boost: Score) -> crate::Result<Box<dyn Scorer>>;
 
     /// Returns an [`Explanation`] for the given document.
-    fn explain(&self, reader: &SegmentReader, doc: DocId) -> crate::Result<Explanation> {
+    fn explain(&self, reader: &dyn SegmentReader, doc: DocId) -> crate::Result<Explanation> {
         let mut scorer = self.scorer(reader, 1.0)?;
         if scorer.doc() > doc || scorer.seek(doc) != doc {
             return Err(does_not_match(doc));
@@ -44,7 +44,7 @@ pub trait Weight: Send + Sync + 'static {
     }
 
     /// Returns the number documents within the given [`SegmentReader`].
-    fn count(&self, reader: &SegmentReader) -> crate::Result<u32> {
+    fn count(&self, reader: &dyn SegmentReader) -> crate::Result<u32> {
         let mut scorer = self.scorer(reader, 1.0)?;
         if let Some(alive_bitset) = reader.alive_bitset() {
             Ok(scorer.count(alive_bitset))
@@ -57,7 +57,7 @@ pub trait Weight: Send + Sync + 'static {
     /// `DocSet` and push the scored documents to the collector.
     fn for_each(
         &self,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
         callback: &mut dyn FnMut(DocId, Score),
     ) -> crate::Result<()> {
         let mut scorer = self.scorer(reader, 1.0)?;
@@ -69,7 +69,7 @@ pub trait Weight: Send + Sync + 'static {
     /// `DocSet` and push the scored documents to the collector.
     fn for_each_no_score(
         &self,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
         callback: &mut dyn FnMut(&[DocId]),
     ) -> crate::Result<()> {
         let mut docset = self.scorer(reader, 1.0)?;
@@ -92,7 +92,7 @@ pub trait Weight: Send + Sync + 'static {
     fn for_each_pruning(
         &self,
         threshold: Score,
-        reader: &SegmentReader,
+        reader: &dyn SegmentReader,
         callback: &mut dyn FnMut(DocId, Score) -> Score,
     ) -> crate::Result<()> {
         let mut scorer = self.scorer(reader, 1.0)?;