Intermediate writes are an implementation detail. From Tantivy side we only decide when a file is finished. This change is done to remove flush overhead for VecWriter. On finalization we just move the Vec to the directory.
Rename TerminatingWrite to FinishableWrite to reflect these semantics.
Treat a fieldless exists leaf as match-all and make the lenient set parser
stop when parsing no longer consumes input. Add public QueryParser proptests
covering both regressions.
Findings and original fixes by Oleksii Syniakov in osyniakov/tantivy#4.
Co-authored-by: Oleksii Syniakov <1282756+osyniakov@users.noreply.github.com>
Silence the 7 warnings reported by `cargo clippy --all --tests`:
- Replace manual `% n == 0` parity checks with `is_multiple_of`
(`manual_is_multiple_of`) in index_writer, doc_predicate_query and
seek_danger tests. `is_multiple_of` is stable for unsigned integers as
of 1.87, which matches the crate MSRV.
- Prefix the intentionally unused `other_segment_meta` binding with an
underscore. The binding is kept rather than dropped because the
inventory only tracks live `SegmentMeta`s, so dropping it would
invalidate the `inventory.all()` assertion in the same test.
No behavior change.
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
Use the default seek-based implementation until the optimized path can handle invalid child states safely. A child miss followed by another child hit previously allowed refill to consume an unaligned scorer.
Add shared seek_danger contract proptests, a regression for #3086, and a source inventory guard for future overrides.
Serialize vector-backed term entries as a map so Quickwit nodes running
different versions can exchange Postcard aggregation results. Add a legacy
fixture covering nested terms and histogram aggregations.
Use total floating-point ordering for aggregation keys so ordering is
consistent with bitwise equality for NaNs and signed zero. This also
removes fallible key comparisons from term aggregation sorting.
* Add DocPredicateQuery and a generic FunctionPredicate implementation
Introduces the DocPredicate/SegmentDocPredicate abstraction and
DocPredicateQuery, a query that walks a segment's documents by
repeatedly evaluating a per-segment predicate (with a scorer_danger
fast path for boolean intersections). FunctionPredicate is a first,
generic implementation built from a plain per-segment factory
function, with no dependency on fast fields or any other segment
data structure.
* CR comments
* Better explain
* Added the capacity to build scorer and seek_danger in a joint manner.
* Added some partial support of scorer_danger for PhraseWeight
* Bugfix following CR
* CR: introducing OccurWeights
* CR comment
---------
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
BooleanWeight::new just default min_should_match to 1.
This is misleading because I think someone who passes a single must clause would expect the result
to match that clause.
Due to min_should_match, it actually returns nothing.
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
* Make aggregation column block accessor private
Move ColumnBlockAccessor out of tantivy-columnar and into the aggregation implementation. Keep its behavior and tests intact while removing it from the public columnar API.
* Refactor aggregation value block access
* Following comment
Removing Fused from the element that are private to the module.
Re-added optimization in stats for single doc requests to avoid perf regression
Renamed Fused -> Flattened
---------
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
commit() only guarantees documents are persisted to the Directory; an
already-open IndexReader only picks up the change according to its
ReloadPolicy. With the default OnCommitWithDelay, the reload happens
asynchronously on a background thread and is not guaranteed to be done
by the time commit() returns, on any Directory implementation
(including RamDirectory), so a search immediately after commit() can
still miss the documents just committed.
Document this explicitly on ReloadPolicy::OnCommitWithDelay,
IndexReader::reload(), and IndexWriter::commit(), and add a runnable
example showing the ReloadPolicy::Manual + explicit reader.reload()
pattern for deterministic read-your-writes.
See #1824
Normalize numerical values before constructing intermediate term keys so equal values merge even when their physical column types differ between segments.
Add a regression test covering u64 and i64 columns with a shared value.
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
Adds an end-to-end test that indexes real JSON documents and asserts hit
counts, complementing the existing logical-AST assertions. The test fails
without the fix (phrase prefix returns 0 hits instead of 2).
Phrase prefix (`"..."*`) and slop (`"..."~N`) were silently dropped on JSON
fields. `generate_literals_for_json_object` hard-coded `slop: 0` and
`prefix: false`, so a query like `data.name:"foo bar"*` degraded to an exact
phrase, even though the underlying `PhrasePrefixQuery` already supports
JSON-path terms.
Thread `slop`/`prefix` through the JSON branch, exactly as the `Str` branch
already does, and add the "phrase prefix requires at least two terms" guard for
the single-token case (mirroring `generate_literals_for_str`).
Tests added as JSON analogues of the existing text-field tests:
- test_phrase_prefix_on_json_field
- test_phrase_prefix_too_short_on_json_field
- test_phrase_slop_on_json_field
This aligns JSON fields with the phrase `~`/`*` behavior already documented for
`QueryParser`.
## Motivation
Today every segment component — postings, fast fields, field norms, store — is hardcoded into several places. Adding a new per-segment data structure means forking Tantivy and editing each of those sites.
This PR introduces a `SegmentPlugin` trait that lets a custom component participate in the full segment lifecycle — write, serialize, merge, garbage collection, space usage — through the same interface the built-ins use, without touching Tantivy internals. The four built-in components are themselves reimplemented as plugins.
We (ParadeDB) plan on using this trait for 1) additional segment metadata for partitioning 2) custom vector index.
## Plugin Trait
Two traits. The first is the `SegmentPlugin` factory:
```rust
pub trait SegmentPlugin: Send + Sync + 'static {
/// File extensions this component owns, e.g. ["idx", "pos", "term"] for postings.
fn extensions(&self) -> &[&str];
/// Create a writer for the indexing path.
fn create_writer(&self, ctx: &PluginWriterContext) -> crate::Result<Box<dyn PluginWriter>>;
/// Merge this component across several source segments into the target segment.
fn merge(&self, ctx: PluginMergeContext) -> crate::Result<()>;
/// Report on-disk space usage, keyed by component name. Has a default impl.
fn space_usage(&self, reader: &SegmentReader)
-> crate::Result<BTreeMap<String, ComponentSpaceUsage>>;
}
```
A `SegmentPlugin` owns one or more file extensions and knows how to (a) build a writer for the indexing path and (b) merge itself across segments.
The second trait is the segment writer:
```rust
pub trait PluginWriter: Send + Any {
/// Called once per document, in doc-id order, for every plugin writer.
fn add_document(&mut self, doc_id: DocId, doc: &TantivyDocument, schema: &Schema)
-> crate::Result<()> { Ok(()) }
/// Serialize accumulated data to segment files (honoring an optional doc-id remap).
fn serialize(&mut self, segment: &Segment, doc_id_map: Option<&DocIdMapping>) -> crate::Result<()>;
fn close(self: Box<Self>) -> crate::Result<()>;
fn mem_usage(&self) -> usize;
fn as_any(&self) -> &dyn Any; // downcast support, Rust 1.86
fn as_any_mut(&mut self) -> &mut dyn Any;
}
```
The write path no longer has any by-name wiring: `SegmentWriter` hands every document to every plugin writer's `add_document`, and `finalize()` calls `serialize` then close on each.
## Key Design Decisions
1. The index — not the segment — owns the plugin set. The set of custom plugins is recorded once, at index creation, in `IndexMeta`. `#[serde(default)]` makes this backward compatible.
2. Plugins are registered, like tokenizers — and re-registration is enforced fail-closed. Plugins are not serialized; they're re-attached on every `Index::open` via `register_plugin`, exactly like custom tokenizers. To prevent consumers from accidentally forgetting to register a plugin, we validate the registered plugin set against the persisted set when the index is first used for a write/merge/GC operation.
3. Registration order is the write/merge order. Built-ins come first (field norms → postings → fast fields → store), then custom plugins.
4. The read side needs no plugin hook. Custom component data is read back through the existing public surface `SegmentReader::open_read`.
5. Backwards compatibility — behavior for existing indexes is unchanged.
Decouple the low-cardinality sub-aggregation threshold from the fused
count-lane threshold, rely on the full-column invariant in the linear
resolver, and explicitly test the single-bucket resolver.
Paul Masurel
Pascal Seitz
jinhelin
Divyesh Kakadiya
DavIvek
Ming
PSeitz-dd