mirror of
https://github.com/quickwit-oss/tantivy.git
synced 2026-10-07 04:12:42 +00:00
Ming
0401b45781
## Motivation
Today every segment component — postings, fast fields, field norms, store — is hardcoded into several places. Adding a new per-segment data structure means forking Tantivy and editing each of those sites.
This PR introduces a `SegmentPlugin` trait that lets a custom component participate in the full segment lifecycle — write, serialize, merge, garbage collection, space usage — through the same interface the built-ins use, without touching Tantivy internals. The four built-in components are themselves reimplemented as plugins.
We (ParadeDB) plan on using this trait for 1) additional segment metadata for partitioning 2) custom vector index.
## Plugin Trait
Two traits. The first is the `SegmentPlugin` factory:
```rust
pub trait SegmentPlugin: Send + Sync + 'static {
/// File extensions this component owns, e.g. ["idx", "pos", "term"] for postings.
fn extensions(&self) -> &[&str];
/// Create a writer for the indexing path.
fn create_writer(&self, ctx: &PluginWriterContext) -> crate::Result<Box<dyn PluginWriter>>;
/// Merge this component across several source segments into the target segment.
fn merge(&self, ctx: PluginMergeContext) -> crate::Result<()>;
/// Report on-disk space usage, keyed by component name. Has a default impl.
fn space_usage(&self, reader: &SegmentReader)
-> crate::Result<BTreeMap<String, ComponentSpaceUsage>>;
}
```
A `SegmentPlugin` owns one or more file extensions and knows how to (a) build a writer for the indexing path and (b) merge itself across segments.
The second trait is the segment writer:
```rust
pub trait PluginWriter: Send + Any {
/// Called once per document, in doc-id order, for every plugin writer.
fn add_document(&mut self, doc_id: DocId, doc: &TantivyDocument, schema: &Schema)
-> crate::Result<()> { Ok(()) }
/// Serialize accumulated data to segment files (honoring an optional doc-id remap).
fn serialize(&mut self, segment: &Segment, doc_id_map: Option<&DocIdMapping>) -> crate::Result<()>;
fn close(self: Box<Self>) -> crate::Result<()>;
fn mem_usage(&self) -> usize;
fn as_any(&self) -> &dyn Any; // downcast support, Rust 1.86
fn as_any_mut(&mut self) -> &mut dyn Any;
}
```
The write path no longer has any by-name wiring: `SegmentWriter` hands every document to every plugin writer's `add_document`, and `finalize()` calls `serialize` then close on each.
## Key Design Decisions
1. The index — not the segment — owns the plugin set. The set of custom plugins is recorded once, at index creation, in `IndexMeta`. `#[serde(default)]` makes this backward compatible.
2. Plugins are registered, like tokenizers — and re-registration is enforced fail-closed. Plugins are not serialized; they're re-attached on every `Index::open` via `register_plugin`, exactly like custom tokenizers. To prevent consumers from accidentally forgetting to register a plugin, we validate the registered plugin set against the persisted set when the index is first used for a write/merge/GC operation.
3. Registration order is the write/merge order. Built-ins come first (field norms → postings → fast fields → store), then custom plugins.
4. The read side needs no plugin hook. Custom component data is read back through the existing public surface `SegmentReader::open_read`.
5. Backwards compatibility — behavior for existing indexes is unchanged.
38 lines
1.3 KiB
Rust
38 lines
1.3 KiB
Rust
use serde::{Deserialize, Serialize};
|
|
|
|
/// Configuration for a plugin-defined field type.
|
|
///
|
|
/// Custom field types are opaque to tantivy's built-in components: they are never indexed,
|
|
/// stored, or turned into fast fields by the built-ins. A [`SegmentPlugin`] consumes the
|
|
/// field's values by matching on [`type_name`](Self::type_name) in the schema and writes its
|
|
/// own segment files.
|
|
///
|
|
/// `type_name` identifies the custom type; `params` carries opaque,
|
|
/// type-specific configuration that the consuming plugin interprets.
|
|
#[derive(Clone, Debug, PartialEq, Serialize, Deserialize, Default)]
|
|
pub struct CustomOptions {
|
|
type_name: String,
|
|
#[serde(default, skip_serializing_if = "serde_json::Value::is_null")]
|
|
params: serde_json::Value,
|
|
}
|
|
|
|
impl CustomOptions {
|
|
/// Creates a new `CustomOptions` for the given custom type name and parameters.
|
|
pub fn new<T: Into<String>>(type_name: T, params: serde_json::Value) -> CustomOptions {
|
|
CustomOptions {
|
|
type_name: type_name.into(),
|
|
params,
|
|
}
|
|
}
|
|
|
|
/// The name identifying this custom type. Plugins claim a type by matching on it.
|
|
pub fn type_name(&self) -> &str {
|
|
&self.type_name
|
|
}
|
|
|
|
/// Opaque, type-specific configuration interpreted by the consuming plugin.
|
|
pub fn params(&self) -> &serde_json::Value {
|
|
&self.params
|
|
}
|
|
}
|