* jitexpr
* Added possible necessary conditions to jitexpr.
A function makes it possible to infer a necessary query from an expression to match.
We can then accelerate queries involving a calculated field by not even evaluating the expression
on docs that trivially do not match.
* CR comment
* CR comment
* Fixing unit tests
---------
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
RegexPhraseWeight::phrase_scorer compiled every phrase term's regex again for each segment. Determinizing a wide pattern (e.g. an alternation of typo or morphology expansions) costs milliseconds to tens of milliseconds, so on a multi-segment index the query was dominated by recompiling the same automata. The regexes are now compiled in RegexPhraseQuery::regex_phrase_weight and shared through Arc.
An invalid pattern is now reported when the weight is built, instead of when the first segment is scored.
Use scalar decoding for ranges overlapping the final partial load.
Clarify the 64-value aggregation block specialization and name the
generic decoding chunk size.
Apply nightly formatting to the touched benchmark and columnar code.
Avoid formatting keys and finalizing sub-aggregations for buckets
discarded by the final size limit. Preallocate the retained bucket vector
and reuse the cutoff helper with intermediate tuple keys.
Preserve final-key ordering and finalize the ordering metric when sorting
by a sub-aggregation. Cover mixed key types, empty sums, and discarded
document counts in tests.
Resolve retained string ordinals in sorted batches per field instead
of decoding dictionary blocks separately for each tuple component.
Reuse decoding for repeated ordinals while preserving tuple order
and existing numeric and missing-value handling.
Probe the accumulated map only for incoming keys rather than rehashing every accumulated key on each merge. This removes quadratic work when folding many disjoint multi-terms results. The same helper serves range and composite buckets; existing bucket values still merge left-to-right and the wire representation and pruning rules are unchanged.
Cover overlapping/disjoint tuple keys, empty inputs, recursive range subaggregations, postcard round trips, error/count bookkeeping, and merging after pruning.
Same benchmark and configuration as 0062d2f0d (Apple M4 Max, rustc 1.98.0):
Median milliseconds for 10 / 100 / 1000 inputs:
shared 8: 0.006365 / 0.0656 / 0.6449
disjoint 16: 0.0126 / 0.1486 / 1.6882
disjoint 160: 0.1336 / 1.5558 / 20.4019
At 1000 inputs this is 52.4x and 61.6x faster for the disjoint cases, with unchanged measured peak allocation. Shared-key control remains within approximately 2% of baseline.
Validation: 308 aggregation tests pass with default features and 308 with quickwit; changed-file rustfmt and git diff checks pass. Clippy --lib --bench agg_bench passes with the pre-existing clippy::drop_non_drop warning allowed (unchanged drop(add_document) in the benchmark).
Run the existing many-segment aggregation benchmarks at 100 and 1,000
segments with one million total documents. Add multi-terms and nested
terms cases for status/Zipf and high-cardinality/Zipf combinations,
including top-500 requests, to expose merge costs across many segments.
The nested top-500 case limits outer buckets, while multi-terms limits
tuples globally.
Run both groups with: cargo bench --bench agg_bench -- _segments
The Str path and the non string path are very different.
This PR isolates them as different segment aggregation collector.
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
Intermediate writes are an implementation detail. From Tantivy side we only decide when a file is finished. This change is done to remove flush overhead for VecWriter. On finalization we just move the Vec to the directory.
Rename TerminatingWrite to FinishableWrite to reflect these semantics.
Treat a fieldless exists leaf as match-all and make the lenient set parser
stop when parsing no longer consumes input. Add public QueryParser proptests
covering both regressions.
Findings and original fixes by Oleksii Syniakov in osyniakov/tantivy#4.
Co-authored-by: Oleksii Syniakov <1282756+osyniakov@users.noreply.github.com>
Add MIT license metadata to the jitexpr crate manifest, matching
tantivy and the other workspace subcrates, and include a copy of
the MIT license text.
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
The coverage job pinned nightly-2025-12-01, which is rustc 1.93. Since
`--all-features --workspace` pulls in jitexpr -> cranelift 0.134.4, and
those crates declare rust-version = "1.94.0", cargo refused to build:
error: rustc 1.93.1 is not supported by the following packages:
cranelift@0.134.4 requires rustc 1.94.0
Bump the pin to nightly-2026-08-31 (1.100.0-nightly), which builds
jitexpr cleanly.
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
Silence the 7 warnings reported by `cargo clippy --all --tests`:
- Replace manual `% n == 0` parity checks with `is_multiple_of`
(`manual_is_multiple_of`) in index_writer, doc_predicate_query and
seek_danger tests. `is_multiple_of` is stable for unsigned integers as
of 1.87, which matches the crate MSRV.
- Prefix the intentionally unused `other_segment_meta` binding with an
underscore. The binding is kept rather than dropped because the
inventory only tracks live `SegmentMeta`s, so dropping it would
invalidate the `inventory.all()` assertion in the same test.
No behavior change.
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
JITModule never releases its executable allocation on drop; the only
way to reclaim it is the consuming, unsafe JITModule::free_memory.
CompiledFn previously stored the module in a plain field, so every
compiled expression's executable memory leaked for the remainder of
the process once the CompiledFn was dropped.
Wrap the module in ManuallyDrop and add an explicit Drop impl for
CompiledFn that calls free_memory. This is safe because CompiledFn is
only ever constructed behind an Arc and never exposes its entry point
or other raw pointers into the module outside &self-bounded calls, so
drop only runs once no call into the module can be in flight or happen
afterward.
Verified empirically: 200k compile+drop cycles peaked at ~3.3 GB RSS
before this fix and ~6 MB after.
Use the default seek-based implementation until the optimized path can handle invalid child states safely. A child miss followed by another child hit previously allowed refill to consume an unaligned scorer.
Add shared seek_danger contract proptests, a regression for #3086, and a source inventory guard for future overrides.
dependabot[bot]
Paul MasurelandPaul Masurel
palmoni5
David Yaffe
Pascal Seitz
PSeitzandPaul Masurel
David Yaffe
PSeitz-dd