mirror of
https://github.com/quickwit-oss/tantivy.git
synced 2026-10-07 04:12:42 +00:00
Ming
0401b45781
## Motivation
Today every segment component — postings, fast fields, field norms, store — is hardcoded into several places. Adding a new per-segment data structure means forking Tantivy and editing each of those sites.
This PR introduces a `SegmentPlugin` trait that lets a custom component participate in the full segment lifecycle — write, serialize, merge, garbage collection, space usage — through the same interface the built-ins use, without touching Tantivy internals. The four built-in components are themselves reimplemented as plugins.
We (ParadeDB) plan on using this trait for 1) additional segment metadata for partitioning 2) custom vector index.
## Plugin Trait
Two traits. The first is the `SegmentPlugin` factory:
```rust
pub trait SegmentPlugin: Send + Sync + 'static {
/// File extensions this component owns, e.g. ["idx", "pos", "term"] for postings.
fn extensions(&self) -> &[&str];
/// Create a writer for the indexing path.
fn create_writer(&self, ctx: &PluginWriterContext) -> crate::Result<Box<dyn PluginWriter>>;
/// Merge this component across several source segments into the target segment.
fn merge(&self, ctx: PluginMergeContext) -> crate::Result<()>;
/// Report on-disk space usage, keyed by component name. Has a default impl.
fn space_usage(&self, reader: &SegmentReader)
-> crate::Result<BTreeMap<String, ComponentSpaceUsage>>;
}
```
A `SegmentPlugin` owns one or more file extensions and knows how to (a) build a writer for the indexing path and (b) merge itself across segments.
The second trait is the segment writer:
```rust
pub trait PluginWriter: Send + Any {
/// Called once per document, in doc-id order, for every plugin writer.
fn add_document(&mut self, doc_id: DocId, doc: &TantivyDocument, schema: &Schema)
-> crate::Result<()> { Ok(()) }
/// Serialize accumulated data to segment files (honoring an optional doc-id remap).
fn serialize(&mut self, segment: &Segment, doc_id_map: Option<&DocIdMapping>) -> crate::Result<()>;
fn close(self: Box<Self>) -> crate::Result<()>;
fn mem_usage(&self) -> usize;
fn as_any(&self) -> &dyn Any; // downcast support, Rust 1.86
fn as_any_mut(&mut self) -> &mut dyn Any;
}
```
The write path no longer has any by-name wiring: `SegmentWriter` hands every document to every plugin writer's `add_document`, and `finalize()` calls `serialize` then close on each.
## Key Design Decisions
1. The index — not the segment — owns the plugin set. The set of custom plugins is recorded once, at index creation, in `IndexMeta`. `#[serde(default)]` makes this backward compatible.
2. Plugins are registered, like tokenizers — and re-registration is enforced fail-closed. Plugins are not serialized; they're re-attached on every `Index::open` via `register_plugin`, exactly like custom tokenizers. To prevent consumers from accidentally forgetting to register a plugin, we validate the registered plugin set against the persisted set when the index is first used for a write/merge/GC operation.
3. Registration order is the write/merge order. Built-ins come first (field norms → postings → fast fields → store), then custom plugins.
4. The read side needs no plugin hook. Custom component data is read back through the existing public surface `SegmentReader::open_read`.
5. Backwards compatibility — behavior for existing indexes is unchanged.
69 lines
2.8 KiB
Rust
69 lines
2.8 KiB
Rust
use std::fmt::{Display, Formatter};
|
|
|
|
/// Enum describing each component of a tantivy segment.
|
|
///
|
|
/// Each component is stored in its own file,
|
|
/// using the pattern `segment_uuid`.`component_extension`,
|
|
/// except the delete component that takes an `segment_uuid`.`delete_opstamp`.`component_extension`
|
|
#[derive(Debug, Clone, Hash, Eq, PartialEq)]
|
|
pub enum SegmentComponent {
|
|
/// Postings (or inverted list). Sorted lists of document ids, associated with terms
|
|
Postings,
|
|
/// Positions of terms in each document.
|
|
Positions,
|
|
/// Column-oriented random-access storage of fields.
|
|
FastFields,
|
|
/// Stores the sum of the length (in terms) of each field for each document.
|
|
/// Field norms are stored as a special u64 fast field.
|
|
FieldNorms,
|
|
/// Dictionary associating `Term`s to `TermInfo`s which is
|
|
/// simply an address into the `postings` file and the `positions` file.
|
|
Terms,
|
|
/// Row-oriented, compressed storage of the documents.
|
|
/// Accessing a document from the store is relatively slow, as it
|
|
/// requires to decompress the entire block it belongs to.
|
|
Store,
|
|
/// Temporary storage of the documents, before streamed to `Store`.
|
|
TempStore,
|
|
/// Bitset describing which document of the segment is alive.
|
|
/// (It was representing deleted docs but changed to represent alive docs from v0.17)
|
|
Delete,
|
|
/// A custom component defined by a [`SegmentPlugin`](crate::SegmentPlugin).
|
|
/// The string is the file extension for this component.
|
|
Custom(String),
|
|
}
|
|
|
|
impl TryFrom<&str> for SegmentComponent {
|
|
type Error = String;
|
|
|
|
fn try_from(value: &str) -> Result<Self, Self::Error> {
|
|
match value {
|
|
"idx" => Ok(SegmentComponent::Postings),
|
|
"pos" => Ok(SegmentComponent::Positions),
|
|
"term" => Ok(SegmentComponent::Terms),
|
|
"store" => Ok(SegmentComponent::Store),
|
|
"temp" => Ok(SegmentComponent::TempStore),
|
|
"fast" => Ok(SegmentComponent::FastFields),
|
|
"fieldnorm" => Ok(SegmentComponent::FieldNorms),
|
|
"del" => Ok(SegmentComponent::Delete),
|
|
other => Ok(SegmentComponent::Custom(other.to_string())),
|
|
}
|
|
}
|
|
}
|
|
|
|
impl Display for SegmentComponent {
|
|
fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
|
|
match self {
|
|
SegmentComponent::Postings => write!(f, "idx"),
|
|
SegmentComponent::Positions => write!(f, "pos"),
|
|
SegmentComponent::FastFields => write!(f, "fast"),
|
|
SegmentComponent::FieldNorms => write!(f, "fieldnorm"),
|
|
SegmentComponent::Terms => write!(f, "term"),
|
|
SegmentComponent::Store => write!(f, "store"),
|
|
SegmentComponent::TempStore => write!(f, "temp"),
|
|
SegmentComponent::Delete => write!(f, "del"),
|
|
SegmentComponent::Custom(ext) => write!(f, "{ext}"),
|
|
}
|
|
}
|
|
}
|