Treat a fieldless exists leaf as match-all and make the lenient set parser
stop when parsing no longer consumes input. Add public QueryParser proptests
covering both regressions.
Findings and original fixes by Oleksii Syniakov in osyniakov/tantivy#4.
Co-authored-by: Oleksii Syniakov <1282756+osyniakov@users.noreply.github.com>
Adds an end-to-end test that indexes real JSON documents and asserts hit
counts, complementing the existing logical-AST assertions. The test fails
without the fix (phrase prefix returns 0 hits instead of 2).
Phrase prefix (`"..."*`) and slop (`"..."~N`) were silently dropped on JSON
fields. `generate_literals_for_json_object` hard-coded `slop: 0` and
`prefix: false`, so a query like `data.name:"foo bar"*` degraded to an exact
phrase, even though the underlying `PhrasePrefixQuery` already supports
JSON-path terms.
Thread `slop`/`prefix` through the JSON branch, exactly as the `Str` branch
already does, and add the "phrase prefix requires at least two terms" guard for
the single-token case (mirroring `generate_literals_for_str`).
Tests added as JSON analogues of the existing text-field tests:
- test_phrase_prefix_on_json_field
- test_phrase_prefix_too_short_on_json_field
- test_phrase_slop_on_json_field
This aligns JSON fields with the phrase `~`/`*` behavior already documented for
`QueryParser`.
## Motivation
Today every segment component — postings, fast fields, field norms, store — is hardcoded into several places. Adding a new per-segment data structure means forking Tantivy and editing each of those sites.
This PR introduces a `SegmentPlugin` trait that lets a custom component participate in the full segment lifecycle — write, serialize, merge, garbage collection, space usage — through the same interface the built-ins use, without touching Tantivy internals. The four built-in components are themselves reimplemented as plugins.
We (ParadeDB) plan on using this trait for 1) additional segment metadata for partitioning 2) custom vector index.
## Plugin Trait
Two traits. The first is the `SegmentPlugin` factory:
```rust
pub trait SegmentPlugin: Send + Sync + 'static {
/// File extensions this component owns, e.g. ["idx", "pos", "term"] for postings.
fn extensions(&self) -> &[&str];
/// Create a writer for the indexing path.
fn create_writer(&self, ctx: &PluginWriterContext) -> crate::Result<Box<dyn PluginWriter>>;
/// Merge this component across several source segments into the target segment.
fn merge(&self, ctx: PluginMergeContext) -> crate::Result<()>;
/// Report on-disk space usage, keyed by component name. Has a default impl.
fn space_usage(&self, reader: &SegmentReader)
-> crate::Result<BTreeMap<String, ComponentSpaceUsage>>;
}
```
A `SegmentPlugin` owns one or more file extensions and knows how to (a) build a writer for the indexing path and (b) merge itself across segments.
The second trait is the segment writer:
```rust
pub trait PluginWriter: Send + Any {
/// Called once per document, in doc-id order, for every plugin writer.
fn add_document(&mut self, doc_id: DocId, doc: &TantivyDocument, schema: &Schema)
-> crate::Result<()> { Ok(()) }
/// Serialize accumulated data to segment files (honoring an optional doc-id remap).
fn serialize(&mut self, segment: &Segment, doc_id_map: Option<&DocIdMapping>) -> crate::Result<()>;
fn close(self: Box<Self>) -> crate::Result<()>;
fn mem_usage(&self) -> usize;
fn as_any(&self) -> &dyn Any; // downcast support, Rust 1.86
fn as_any_mut(&mut self) -> &mut dyn Any;
}
```
The write path no longer has any by-name wiring: `SegmentWriter` hands every document to every plugin writer's `add_document`, and `finalize()` calls `serialize` then close on each.
## Key Design Decisions
1. The index — not the segment — owns the plugin set. The set of custom plugins is recorded once, at index creation, in `IndexMeta`. `#[serde(default)]` makes this backward compatible.
2. Plugins are registered, like tokenizers — and re-registration is enforced fail-closed. Plugins are not serialized; they're re-attached on every `Index::open` via `register_plugin`, exactly like custom tokenizers. To prevent consumers from accidentally forgetting to register a plugin, we validate the registered plugin set against the persisted set when the index is first used for a write/merge/GC operation.
3. Registration order is the write/merge order. Built-ins come first (field norms → postings → fast fields → store), then custom plugins.
4. The read side needs no plugin hook. Custom component data is read back through the existing public surface `SegmentReader::open_read`.
5. Backwards compatibility — behavior for existing indexes is unchanged.
It applies the same logic on floats as for u64 or i64.
In all case, the idea is (for the inverted index) to coerce number
to their canonical representation, before indexing and before searching.
That way a document with the float 1.0 will be searchable when the user
searches for 1.
Note that contrary to the columnar, we do not attempt to coerce all of the
terms associated to a given json path to a single numerical type.
We simply rely on this "point-wise" canonicalization.
(a OR b) OR (c OR d) can be simplified to (a OR b OR c OR d)
(a AND b) AND (c AND d) can be simplified to (a AND b AND c AND d)
This directly affects how queries are executed
remove unused SumWithCoordsCombiner
the number of fields is unused and private
* support ff range queries on json fields
* fix term date truncation
* use inverted index range query for phrase prefix queries
* rename to InvertedIndexRangeQuery
* fix column filter, add mixed column test
As preparation of #2023 and #1709
* Use Term to pass parameters
* merge u64 and ip fast field range query
Side note: I did not rename range_query_u64_fastfield, because then git can't track the changes.
* feat(query): Make `BooleanQuery` supports `minimum_number_should_match`. see issue #2398
In this commit, a novel scorer named DisjunctionScorer is introduced, which performs the union of inverted chains with the minimal required elements. BTW, it's implemented via a min-heap. Necessary modifications on `BooleanQuery` and `BooleanWeight` are performed as well.
* fixup! fix test
* fixup!: refactor code.
1. More meaningful names.
2. Add Cache for `Disjunction`'s scorers, and fix bug.
3. Optimize `BooleanWeight::complex_scorer`
Thanks
Paul Masurel <paul@quickwit.io>
* squash!: come up with better variable naming.
* squash!: fix naming issues.
* squash!: fix typo.
* squash!: Remove CombinationMethod::FullIntersection
* move query parser to nom
* add suupport for term grouping
* initial work on infallible parser
* fmt
* add tests and fix minor parsing bugs
* address review comments
* add support for lenient queries in tantivy
* make lenient parser report errors
* allow mixing occur and bool in query
* tokenizer-api: reduce Tokenizer overhead
Previously a new `Token` for each text encountered was created, which
contains `String::with_capacity(200)`
In the new API the token_stream gets mutable access to the tokenizer,
this allows state to be shared (in this PR Token is shared).
Ideally the allocation for the BoxTokenStream would also be removed, but
this may require some lifetime tricks.
* simplify api
* move lowercase and ascii folding buffer to global
* empty Token text as default
* Expose phrase-prefix queries via the built-in query parser
This proposes the less-than-imaginative syntax `field:"phrase ter"*` to
perform a phrase prefix query against `field` using `phrase` and `ter` as the
terms. The aim of this is to make this type of query more discoverable and
simplify manual testing.
I did consider exposing the `max_expansions` parameter similar to how slop is
handled, but I think that this is rather something that should be configured via
the querser parser (similar to `set_field_boost` and `set_field_fuzzy`) as
choosing it requires rather intimiate knowledge of the backing index.
* Prevent construction of zero or one term phrase-prefix queries via the query parser.
* Add example using phrase-prefix search via surface API to improve feature discoverability.
* Change in the query grammar.
Quotation mark can now be used for phrase queries.
The delimiter is part of the `UserInputLeaf`.
That information is meant to be used in Quickwit to solve #3364.
This PR also adds support for quotation marks escaping in phrase
queries.
* Apply suggestions from code review
Applied this command to the code, making it a bit shorter and slightly
more readable.
```
cargo +nightly clippy --all-features --benches --tests --workspace --fix -- -A clippy::all -W clippy::uninlined_format_args
cargo +nightly fmt --all
```
* Make nightly Clippy mostly happy.
* Document how to produce TermSetQuery queries using QueryParser.
* Enable construction of queries using FuzzyTermQuery via the QueryParser
* Use FxHashMap instead of HashMap in the QueryParser as these hash tables are not exposed to DoS attacks.
* Use a struct instead of a tuple to improve readability.
Luca Cominardi
Pascal Seitz
DavIvek
Ming
Darkheir
PSeitz-dd
Paul Masurel
trinity Pointard
Bruce Mitchener
PSeitz
落叶乌龟
trinity-1686a
Caleb Hattingh
François Massot
Adam Reichold
Paul Masurel
Yuri Astrakhan
Adrien Guillo
boraarslan