`TermOrdMemo` named the technique, not what it holds: the results of the
expression per input term ord. Rename it `TermOrdResults`, the field
`term_ord_results`, the limit `TERM_ORD_RESULTS_MAX_NUM_TERMS`, and the
slot states `RESULT_UNKNOWN`/`RESULT_NULL`/`RESULT_VALUE`. No behavior
change.
- Replace `TermOrdMemo::get`/`set` by `get_or_eval`, and store the result
for a doc without an input value in slot 0 (term ord `o` in slot `o + 1`)
instead of a separate field.
- Index the memo directly: a term ord out of the input dictionary is a
corrupted column, which panics like the rest of the module.
- Destructure the source once per block, and evaluate misses with a free
`eval_single_input`, instead of a per-doc `memoized_result` method.
- Share the cardinality tail of the two `load_block` paths, and implement
`InputColumn::load_block` on top of `load_block_values`.
- Drop the test-only memo threshold: the limit test builds a dictionary of
`TERM_ORD_MEMO_MAX_NUM_TERMS + 1` terms, and the other tests exercise the
memo for their small dictionaries.
jitexpr_agg_bench: terms on (REGEXP_EXTRACT service ..) 7.7 ms -> 6.8 ms,
on (UPPER service) 8.0 ms -> 6.8 ms; the other cases are unchanged.
A jitexpr value source evaluated its expression once per doc. For an
expression of a text input, e.g. `(REGEXP_EXTRACT service "^([^-]+)" 1u64)`,
the result only depends on the term of the doc, and text inputs are
dictionary encoded with typically few distinct terms per segment (service
names, hosts, statuses). Evaluating the expression per doc made a terms
aggregation over such a calculated field ~100x slower than over the
physical column; on a CloudPrem searcher, the expression accounted for 91%
of the CPU of such a query.
Memoize the results of an expression of a single text input whose
dictionary has at most 65,536 terms, per input term ord. The expression is
evaluated on the first doc of each term, and later docs read the encoded
result from two dense arrays allocated zeroed when the source is opened
(9 bytes per term), plus a slot for docs without a value. Terms are only
resolved for the term ords evaluated for the first time. jitexpr functions
are deterministic, and `Str` results are interned in the order of the docs
as before, so the values and term ords are unchanged. `to_f64` clears the
memo, whose results are encoded for the previous result type. Other
expressions keep the per-doc path.
benches/jitexpr_agg_bench.rs (2M docs, 4 segments, 200 services):
terms on (REGEXP_EXTRACT service ..) 140.6 ms -> 7.7 ms (18x)
terms on (UPPER service) 305.7 ms -> 8.0 ms (38x)
terms on (REGEXP_EXTRACT request_id ..) 1.441 s -> 1.444 s (unique terms, no memo)
terms on physical service 1.08 ms -> 1.07 ms
A jitexpr field that can only return a string (e.g. REGEXP_EXTRACT) used
to fail in aggregations requiring a number (stats, histogram, range,
percentiles, ...). It is now parsed as f64. The doc has no value if the
string cannot be parsed, or is not finite.
- jitexpr: lower the Str -> F64 coercion with a native parse function.
- jitexpr: accept Str in CompiledFn::coerce_result_to_f64.
- jitexpr: add infer_result_types.
- JitExprValueSourceProvider: coerce string-only expressions when the
aggregation accepts f64 but not strings.
* Add the plain codec format to the columnar
Adds StrColumnCodec, the column key code of plain columns, the V3
columnar version, and the plain column storage: OnPair16 blocks of at
most 65,536 values and 1 MiB, block index, and reader.
The columnar writer does not produce plain columns yet.
* less result moar panic
* CR Comment. Making PlainColumn Inner type private
* CR comment: checked sorted index in block index.
`SnippetGenerator::set_max_num_chars` and `search_fragments` document the
limit as a number of chars (not bytes), but fragments were cut as soon as
their byte length went over it. For non-ASCII text, snippets were therefore
much shorter than requested: about half for Cyrillic or Greek, a third for
CJK.
Count the chars of the fragment instead. Since a char is at least one byte,
the chars only need counting once the byte length exceeds the limit, so ASCII
text does no extra work.
Co-authored-by: ybai08 <271969568+ybai08@users.noreply.github.com>
* Export PhraseScorer from tantivy::query
PhraseScorer already has a public constructor and the Postings trait is public, but the type was only reachable inside the crate. Exporting it lets a custom Query/Weight score a phrase over any Postings, for example a union of several terms at one position, without copying the phrase matching code.
Also documents the struct, the constructor and phrase_count.
* Document that PhraseScorer::new needs two postings
Intersection::new asserts at least two docsets, so a custom Weight whose phrase degenerates to one term must branch before building the scorer.
---------
Co-authored-by: KleiKodesh <211884665+KleiKodesh@users.noreply.github.com>
The metric reducers folded values one at a time through recurrences that
carry state across every iteration: Kahan compensated summation in
`collect_stats` (sum/avg/min/max/count/stats) and Welford's online variance
in `extended_stats`. These strict dependency chains block both CPU
pipelining and auto-vectorization.
`collect_stats` now reduces each block of values as a slice into 4
independent Kahan (sum, delta) lanes plus 4 min/max lanes, combined at the
end of the block.
`extended_stats` accumulates each block into 4 independent
`IntermediateExtendedStats` lanes and folds them into the bucket with the
existing `merge_fruits` (Chan parallel-variance combination), the same
operation used to merge segment results. The lanes carry the bucket's sigma,
since `merge_fruits` replaces an empty bucket with the incoming lane.
Blocks with fewer than two values per lane are collected value by value,
where setting up and combining lanes costs more than it saves.
Only the floating-point summation order changes, as it already does across
segment merges. All aggregation tests pass within the existing `2e-12`
tolerance, including the rounding-sensitive `test_aggregation_level1`, and a
new test covers sigma through the lane path.
`VecTermBuckets` and `LazyVecTermBuckets` used `get_unchecked_mut` on
the term id. The bucket vector is sized from the column's stored
`max_value`, which is read from disk and never checked against the
decoded values. A corrupt or crafted column could cause an
out-of-bounds write.
Use plain indexing instead. A corrupt column now panics, like
`PagedTermMap` already does.
agg_bench (terms without sub-aggregations, 4 index types, 2 rounds):
all differences vs main are within noise (-3.8% to +1.5%).
Replace the `LAZY_BUCKET_ID_GENERATION` const bool on `VecTermBuckets`
with two focused types:
- `VecTermBuckets<B, const LANES: usize = 1>`: eager bucket-id assignment,
with `LANES` count lanes. `LANES == 1` is the former scalar-eager map;
`LANES == 8` (via the `VecTermBucketsWithLanes` alias) the lane map.
- `LazyVecTermBuckets<B>`: lazy bucket-id assignment on first occurrence,
single counter. Lazy only applies above
`LAZY_BUCKET_ID_GENERATION_THRESHOLD`, which never overlaps the very-low
cardinality lane regime, so it needs no lanes.
This removes the const bool and its awkward `LAZY && !B::ASSIGNS_ID`
edge case: each `term_entry` now does exactly one thing (eager rotates
across lanes; lazy assigns on first use), so both hot loops are simpler.
`VecTermBucketsWithLanes` is kept as a type alias
(`= VecTermBuckets<B, LANES>`) so the lane call site and the existing
lane unit test are unchanged. Dispatch in term_agg and multi_terms now
selects `LazyVecTermBuckets` for the lazy branch.
No behavior change: all term and multi-terms aggregation tests pass.
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
* Hardening unsafe in substring_count
* Remove unsafe in StrColumn::ord_to_str
ord_to_term may leave arbitrary bytes in the buffer when the term is not
found or on error, which left an invalid UTF-8 String behind.
The buffer is now only put back into the String after UTF-8 validation.
On not found or error, the output is left empty.
* Make CompactDoc::node_data crate private
extract_str relies on string values in node_data being valid UTF-8.
With a public field, safe code could break this invariant.
* Remove unsafe in Term::get_json_path
The public Term API (append_bytes, set_bytes, Term::wrap, serde, ...) can
put arbitrary bytes in the json path. Invalid UTF-8 sequences are now
replaced by U+FFFD.
* LZ4 doc store: never hand back a buffer with uninitialized bytes
compress/decompress now work on a local Vec while it contains
uninitialized bytes, and clear it on error. On error or panic, the
caller's buffer is left empty. Buffer reuse is preserved.
* Prepare ValueSource for computed text sources
- `ValueSource::term_dictionary()` lets a source resolve its own term
ords (`ValueSourceDictionary`, implemented by the sstable dictionary).
Terms and cardinality resolve ords through it; a registered name never
uses the physical dictionary of the same field.
- `ValueSourceProvider::for_segment` receives the column types the
aggregation accepts.
- composite and multi_terms reject registered sources explicitly.
* CR comment
* CR comment
* CR comment
---------
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
`VersatileBuffer<A>` (tantivy-common) keeps an allocation and lends it
as an empty `Vec<T>`, for any `T` with the size and alignment of `A`.
The allocation goes back to the buffer when the guard is dropped,
including on panic. If the guard is leaked, the buffer starts again
from an empty `Vec`.
This makes it possible to reuse a buffer of values borrowing data with
a short lifetime, without self-referential structs.
The jitexpr predicate uses it for its argument slots. It replaces the
`ClearOnDrop` guard, which transmuted a `Vec<VariableValue<'static>>`
to a shorter lifetime.
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
`Intersection::seek` used to build a `Vec<&mut dyn DocSet>` on every call
to reuse `go_to_first_doc`. It now shares the leapfrog loop of `advance`,
which uses `seek_danger` on the secondary docsets and does not allocate.
Added a proptest mixing `seek` and `advance` on a 3-way intersection.
* Add keyless streaming to the term dictionary streamers
The sstable streamer rebuilds the key of each entry it reads. It does
this also for the entries that the automaton rejects. Many callers do
not read the key. They only use term infos or term ordinals.
Add `without_keys()` to the streamer builders. The resulting streamer
does not rebuild keys. It gives term ordinals and values only.
- `KeyTracking` (in tantivy-common) is the buffer for the current key.
`Vec<u8>` keeps the key. `WithoutKeys` does nothing.
- The streamer types get a generic parameter `K = Vec<u8>`. Existing
type names keep the keyed behavior. `key()` and `next()` exist only
on the keyed streamer. The keyless streamer has `next_without_key()`.
- At the lower bound, the streamer does not know the previous key. It
uses the lower bound prefix that the comparator found as the
previous key. It then checks the upper bound on this prefix and the
suffix, with the new `DeltaKeyComparator::compare_prefix_and_suffix`.
- The FST streamer uses the same `KeyTracking`. Thus the
`TermDictionary` facade has `without_keys()` on both backends.
These callers now use `without_keys()`: AutomatonWeight,
InvertedIndexRangeWeight, the quickwit warmup paths and the terms
aggregation include/exclude regex.
* CR comment
---------
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
* Make ValueSource::load_block take &mut self
Value sources were held as `Arc<dyn ValueSource>` and cloned into their
collector, so `load_block` could only take `&self`. A computed source
could not keep per-segment state (caches, scratch buffers) across blocks.
Collectors were built in two phases: `build_nodes` pushed each
`XxxAggReqData` into per-kind vectors of `AggregationsSegmentCtx`, and
each collector builder cloned its request data out of those vectors by
index. The ctx copy had to stay around for name lookups, memory
accounting, and for the collectors that read their data back by index.
Each request node is turned into exactly one collector, so nothing
actually shares a value source. This change makes that ownership
explicit:
- `ValueSource::load_block` takes `&mut self`,
`ValueSourceProvider::for_segment` returns `Box<dyn ValueSource>`.
- The request tree (`AggNode { data: AggNodeData, children }`) owns the
request data. Building collectors consumes the tree, moving the
request data into the collectors instead of cloning it.
- `AggregationsSegmentCtx` only keeps the shared collect-time state
(context and block accessor). `PerRequestAggSegCtx`, `AggKind`,
`idx_in_req_data` and the `push_*`/`get_*_req_data` helpers are gone.
- Cardinality, percentiles, top_hits and missing_term collectors own
their request data instead of reading it from the ctx by index.
- Histogram requests are normalized once, when building the tree. The
flattened terms x histogram path is split into a borrow-only plan step
and a consuming build step, and no longer clones the histogram request
at finalization.
- Request data memory is charged exactly once per node. Previously it
was charged at every nesting level, and range, histogram, filter and
composite builders charged their request data a second time.
- `FilterAggReqData::evaluator` no longer needs an `Rc`, and the
filter, composite and multi_terms request data no longer derive Clone.
* CR comment
* CR comment
---------
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
* Changing the way aggregation access their value.
They now get values via a ValueSource abstraction.
The aggregation collector also gets the possibility to
register ValueSourceProvider describing value columns that are computed
on the fly.
Finally, segment aggregation that require a full column
now manipulates a Arc<dyn ColumnValue> directly.
* CR comments
* Clippy
* Fixing regression
---------
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
Sorted segment merges copied every document's positions for a term into a vector before sorting by mapped document ID. Common terms with many positions therefore consumed memory proportional to documents × positions, outside the indexing writer budget.
Stream shuffled postings through a min-heap with one cursor per input segment and one reusable positions buffer. The mapping preserves each segment's document order; deleted/filtered postings are skipped. The stacked path continues to stream directly.
RegexQuery can be built from a compiled Regex, but RegexPhraseQuery always compiled its patterns with Regex::new and its default DFA state limit. from_regexes takes the compiled regexes directly, so a caller can build them with its own limits, or reuse ones it already compiled.
* Expose the Levenshtein automaton of FuzzyTermQuery
Callers that need the terms a fuzzy query expands to (e.g. to highlight them) had to copy the private DfaWrapper and depend on the same levenshtein_automata version as tantivy, or their expansion could silently diverge from the query's. FuzzyTermQuery::automaton returns the automaton the query matches with, built from the same cached LevenshteinAutomatonBuilder.
* Export DfaWrapper outside of tests
The re-export was still behind #[cfg(test)], so FuzzyTermQuery::automaton returned a type other crates could not name. A doctest now uses it from outside the crate.
Snippet only kept a copy of the fragment text, so callers that need to extend the fragment or map it back to the source (e.g. to keep punctuation that the tokenizer leaves outside the last token) had to search for it, which picks the wrong occurrence when the same text appears earlier.