## Motivation
Today every segment component — postings, fast fields, field norms, store — is hardcoded into several places. Adding a new per-segment data structure means forking Tantivy and editing each of those sites.
This PR introduces a `SegmentPlugin` trait that lets a custom component participate in the full segment lifecycle — write, serialize, merge, garbage collection, space usage — through the same interface the built-ins use, without touching Tantivy internals. The four built-in components are themselves reimplemented as plugins.
We (ParadeDB) plan on using this trait for 1) additional segment metadata for partitioning 2) custom vector index.
## Plugin Trait
Two traits. The first is the `SegmentPlugin` factory:
```rust
pub trait SegmentPlugin: Send + Sync + 'static {
/// File extensions this component owns, e.g. ["idx", "pos", "term"] for postings.
fn extensions(&self) -> &[&str];
/// Create a writer for the indexing path.
fn create_writer(&self, ctx: &PluginWriterContext) -> crate::Result<Box<dyn PluginWriter>>;
/// Merge this component across several source segments into the target segment.
fn merge(&self, ctx: PluginMergeContext) -> crate::Result<()>;
/// Report on-disk space usage, keyed by component name. Has a default impl.
fn space_usage(&self, reader: &SegmentReader)
-> crate::Result<BTreeMap<String, ComponentSpaceUsage>>;
}
```
A `SegmentPlugin` owns one or more file extensions and knows how to (a) build a writer for the indexing path and (b) merge itself across segments.
The second trait is the segment writer:
```rust
pub trait PluginWriter: Send + Any {
/// Called once per document, in doc-id order, for every plugin writer.
fn add_document(&mut self, doc_id: DocId, doc: &TantivyDocument, schema: &Schema)
-> crate::Result<()> { Ok(()) }
/// Serialize accumulated data to segment files (honoring an optional doc-id remap).
fn serialize(&mut self, segment: &Segment, doc_id_map: Option<&DocIdMapping>) -> crate::Result<()>;
fn close(self: Box<Self>) -> crate::Result<()>;
fn mem_usage(&self) -> usize;
fn as_any(&self) -> &dyn Any; // downcast support, Rust 1.86
fn as_any_mut(&mut self) -> &mut dyn Any;
}
```
The write path no longer has any by-name wiring: `SegmentWriter` hands every document to every plugin writer's `add_document`, and `finalize()` calls `serialize` then close on each.
## Key Design Decisions
1. The index — not the segment — owns the plugin set. The set of custom plugins is recorded once, at index creation, in `IndexMeta`. `#[serde(default)]` makes this backward compatible.
2. Plugins are registered, like tokenizers — and re-registration is enforced fail-closed. Plugins are not serialized; they're re-attached on every `Index::open` via `register_plugin`, exactly like custom tokenizers. To prevent consumers from accidentally forgetting to register a plugin, we validate the registered plugin set against the persisted set when the index is first used for a write/merge/GC operation.
3. Registration order is the write/merge order. Built-ins come first (field norms → postings → fast fields → store), then custom plugins.
4. The read side needs no plugin hook. Custom component data is read back through the existing public surface `SegmentReader::open_read`.
5. Backwards compatibility — behavior for existing indexes is unchanged.
Decouple the low-cardinality sub-aggregation threshold from the fused
count-lane threshold, rely on the full-column invariant in the linear
resolver, and explicitly test the single-bucket resolver.
* chore: downgrade info! logs to debug! in managed_directory.rs
* chore: downgrade info! logs to debug! in file_watcher.rs
* chore: downgrade info! logs to debug! in index_writer.rs
* chore: downgrade info! logs to debug! in prepared_commit.rs
* chore: downgrade info! logs to debug! in segment_updater.rs
`BooleanWeight::for_each_pruning` handed a union of term scorers straight
to `block_wand`, whatever the weight's score combiner. Block-WAND prunes on
the sum of the per-term block maxima and scores the survivors the same way,
so it silently replaces the combiner with a sum. A `DisjunctionMaxQuery`
collected through `TopDocs` therefore scored a document matching in two
fields as the sum of both instead of the better of the two, and ignored the
tie breaker.
The union only reaches that path when every clause reads term frequencies,
which is why the bug hid: a query asking for `Basic` postings falls back to
`BufferedUnionScorer`, which does honor the combiner.
`ScoreCombiner` now declares whether Block-WAND may drive it. Only
`SumCombiner` opts in; everything else falls back to the plain union, which
still prunes on the threshold but makes no assumption about how the scores
combine.
The intersection specialization needs no such fallback: `Intersection::score`
sums its children whatever the combiner (the combiner only shapes how should
clauses combine), so `block_wand_intersection`'s summing already matches the
unpruned scorer for every combiner. A second test pins that equivalence by
comparing `for_each_pruning` against a manual scorer walk on a must+must
weight built with `DisjunctionMaxCombiner`.
Avoid histogram decoding for single-bucket ranges and use precomputed
boundaries for small ranges. Split hot counters into lanes to reduce
write dependencies on low-cardinality grids.
`Streamer::term_ord()` derived the ordinal by counting `delta_reader.advance()`
calls from a seed taken only when the stream had a key lower bound. An automaton
search has no key bounds, so the seed was 0 — but the reader it is handed *is*
block-pruned by that automaton (`get_block_iterator_for_range_and_automaton`).
Every block skipped ahead of the first match went uncounted, so `term_ord()`
returned the term's position among the blocks actually scanned rather than its
ordinal in the dictionary.
The error is silent and grows with how deep the first match sits: on a 262k-term
dictionary, a regex matching only the last key reported ordinal 999 instead of
262142. Callers that resolve those ordinals back to terms therefore act on a
different term entirely — `build_allowed_term_ids_for_str` builds the terms
aggregation's allowed-ordinal bitset this way, so an `include` regex could make
the aggregation count unrelated terms while returning the expected bucket count.
Broad patterns matching from the start of the dictionary hid it, since nothing is
pruned ahead of the first match.
Each slice handed to the reader now carries the ordinal of its first term, and
the streamer resets to it on entering a slice instead of incrementing across the
gap. Blocks merged into one slice stay contiguous, so counting within a slice is
still correct.
`Dictionary::sorted_ords_to_term_cb` already tracked `BlockAddr::first_ordinal`
explicitly, which is why ordinal->term resolution was unaffected.
Map a configured string missing value to its dictionary ordinal when the
term is already present. This coalesces real and missing documents before
the segment-level cutoff, preserving counts and sub-aggregation results.
Keep synthetic sentinels for absent terms and non-string columns, and add
regression coverage with a segment size of one.
Extract full and non-full field key construction into dedicated helpers.
Rename the surviving document buffer and avoid copying IDs on the full-field path.
Filter documents without values after each single-valued field so later
decoding and key construction only process viable documents. Use exact
block lengths to validate aligned results and keep unsupported missing-value
combinations on the general path.
Track valid document positions in a reusable bitset instead of compacting
parallel document and index buffers after each optional field. Decode each
field against the original block and consult the mask before writing keys or
populating buckets.
This favors simpler collection logic over shrinking later sparse decodes.
Fan out dynamic/JSON physical column combinations during aggregation tree
construction so each segment collector handles one typed accessor per
field and applies missing values exactly once.
Use a generic key codec and shared term bucket maps for both packed and
unpacked keys. Keep optional and multivalued fields eligible for packed
storage, short-circuit sparse documents, and account for spilled composite
keys.
Paul Masurel
Pascal Seitz
Paul Masurel
Ming
PSeitz-dd
trinity-1686a
dependabot[bot]
mmustafasenoglu
Marc Bachmann
Xuanwo
ildis