On OTel logs (~1 KB stored per doc, a stored JSON object of attributes):
- Doc store: CompactDoc serializes its stored fields by walking node_data
directly (strings and bytes are already VInt-len + bytes there, the doc
store encoding after the type byte). New hidden Document::
serialize_stored_fields hook, default false: other document types keep the
generic serializer. Byte-identical to BinaryDocumentSerializer (proptest on
random nested values of every type, stored/unstored fields); IP addresses,
pre-tokenized text and custom values fall back to the generic path.
StoreWriter::store: 987 -> 583 ns/doc.
- CompactDoc: decode ValueAddr / VInt from slices instead of io::Read; exact
size_hint on object/array iterators (the generic serializer then streams
entries instead of collecting them).
- CompactDoc::add_value: one thread-local scratch stack for child addresses
instead of a Vec per nested object; leaves skip it. Doc building: -7.7%.
- SimpleTokenizer: byte lookup table for pure-ASCII text, same tokens as the
char path (proptest on ASCII and arbitrary Unicode). Tokenizer: -14%.
Indexing 2M OTel docs, one thread (harness): -6% CPU with zstd 8 / 1 MB
blocks, -12% with zstd 3 / 4 MB blocks.
* Prepare ValueSource for computed text sources
- `ValueSource::term_dictionary()` lets a source resolve its own term
ords (`ValueSourceDictionary`, implemented by the sstable dictionary).
Terms and cardinality resolve ords through it; a registered name never
uses the physical dictionary of the same field.
- `ValueSourceProvider::for_segment` receives the column types the
aggregation accepts.
- composite and multi_terms reject registered sources explicitly.
* CR comment
* CR comment
* CR comment
---------
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
`VersatileBuffer<A>` (tantivy-common) keeps an allocation and lends it
as an empty `Vec<T>`, for any `T` with the size and alignment of `A`.
The allocation goes back to the buffer when the guard is dropped,
including on panic. If the guard is leaked, the buffer starts again
from an empty `Vec`.
This makes it possible to reuse a buffer of values borrowing data with
a short lifetime, without self-referential structs.
The jitexpr predicate uses it for its argument slots. It replaces the
`ClearOnDrop` guard, which transmuted a `Vec<VariableValue<'static>>`
to a shorter lifetime.
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
`Intersection::seek` used to build a `Vec<&mut dyn DocSet>` on every call
to reuse `go_to_first_doc`. It now shares the leapfrog loop of `advance`,
which uses `seek_danger` on the secondary docsets and does not allocate.
Added a proptest mixing `seek` and `advance` on a 3-way intersection.
* Add keyless streaming to the term dictionary streamers
The sstable streamer rebuilds the key of each entry it reads. It does
this also for the entries that the automaton rejects. Many callers do
not read the key. They only use term infos or term ordinals.
Add `without_keys()` to the streamer builders. The resulting streamer
does not rebuild keys. It gives term ordinals and values only.
- `KeyTracking` (in tantivy-common) is the buffer for the current key.
`Vec<u8>` keeps the key. `WithoutKeys` does nothing.
- The streamer types get a generic parameter `K = Vec<u8>`. Existing
type names keep the keyed behavior. `key()` and `next()` exist only
on the keyed streamer. The keyless streamer has `next_without_key()`.
- At the lower bound, the streamer does not know the previous key. It
uses the lower bound prefix that the comparator found as the
previous key. It then checks the upper bound on this prefix and the
suffix, with the new `DeltaKeyComparator::compare_prefix_and_suffix`.
- The FST streamer uses the same `KeyTracking`. Thus the
`TermDictionary` facade has `without_keys()` on both backends.
These callers now use `without_keys()`: AutomatonWeight,
InvertedIndexRangeWeight, the quickwit warmup paths and the terms
aggregation include/exclude regex.
* CR comment
---------
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
* Make ValueSource::load_block take &mut self
Value sources were held as `Arc<dyn ValueSource>` and cloned into their
collector, so `load_block` could only take `&self`. A computed source
could not keep per-segment state (caches, scratch buffers) across blocks.
Collectors were built in two phases: `build_nodes` pushed each
`XxxAggReqData` into per-kind vectors of `AggregationsSegmentCtx`, and
each collector builder cloned its request data out of those vectors by
index. The ctx copy had to stay around for name lookups, memory
accounting, and for the collectors that read their data back by index.
Each request node is turned into exactly one collector, so nothing
actually shares a value source. This change makes that ownership
explicit:
- `ValueSource::load_block` takes `&mut self`,
`ValueSourceProvider::for_segment` returns `Box<dyn ValueSource>`.
- The request tree (`AggNode { data: AggNodeData, children }`) owns the
request data. Building collectors consumes the tree, moving the
request data into the collectors instead of cloning it.
- `AggregationsSegmentCtx` only keeps the shared collect-time state
(context and block accessor). `PerRequestAggSegCtx`, `AggKind`,
`idx_in_req_data` and the `push_*`/`get_*_req_data` helpers are gone.
- Cardinality, percentiles, top_hits and missing_term collectors own
their request data instead of reading it from the ctx by index.
- Histogram requests are normalized once, when building the tree. The
flattened terms x histogram path is split into a borrow-only plan step
and a consuming build step, and no longer clones the histogram request
at finalization.
- Request data memory is charged exactly once per node. Previously it
was charged at every nesting level, and range, histogram, filter and
composite builders charged their request data a second time.
- `FilterAggReqData::evaluator` no longer needs an `Rc`, and the
filter, composite and multi_terms request data no longer derive Clone.
* CR comment
* CR comment
---------
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
* Changing the way aggregation access their value.
They now get values via a ValueSource abstraction.
The aggregation collector also gets the possibility to
register ValueSourceProvider describing value columns that are computed
on the fly.
Finally, segment aggregation that require a full column
now manipulates a Arc<dyn ColumnValue> directly.
* CR comments
* Clippy
* Fixing regression
---------
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
Sorted segment merges copied every document's positions for a term into a vector before sorting by mapped document ID. Common terms with many positions therefore consumed memory proportional to documents × positions, outside the indexing writer budget.
Stream shuffled postings through a min-heap with one cursor per input segment and one reusable positions buffer. The mapping preserves each segment's document order; deleted/filtered postings are skipped. The stacked path continues to stream directly.
RegexQuery can be built from a compiled Regex, but RegexPhraseQuery always compiled its patterns with Regex::new and its default DFA state limit. from_regexes takes the compiled regexes directly, so a caller can build them with its own limits, or reuse ones it already compiled.
* Expose the Levenshtein automaton of FuzzyTermQuery
Callers that need the terms a fuzzy query expands to (e.g. to highlight them) had to copy the private DfaWrapper and depend on the same levenshtein_automata version as tantivy, or their expansion could silently diverge from the query's. FuzzyTermQuery::automaton returns the automaton the query matches with, built from the same cached LevenshteinAutomatonBuilder.
* Export DfaWrapper outside of tests
The re-export was still behind #[cfg(test)], so FuzzyTermQuery::automaton returned a type other crates could not name. A doctest now uses it from outside the crate.
Snippet only kept a copy of the fragment text, so callers that need to extend the fragment or map it back to the source (e.g. to keep punctuation that the tokenizer leaves outside the last token) had to search for it, which picks the wrong occurrence when the same text appears earlier.
Following up on #3135, the regexes are compiled on first use and kept in the query, so a query reused across searchers, or cloned, determinizes each pattern once. RegexPhraseQuery::regexes exposes them for callers that inspect the patterns before searching, e.g. to count a phrase's per-segment expansions against max_expansions, instead of compiling them a second time.
* jitexpr
* Added possible necessary conditions to jitexpr.
A function makes it possible to infer a necessary query from an expression to match.
We can then accelerate queries involving a calculated field by not even evaluating the expression
on docs that trivially do not match.
* CR comment
* CR comment
* Fixing unit tests
---------
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
RegexPhraseWeight::phrase_scorer compiled every phrase term's regex again for each segment. Determinizing a wide pattern (e.g. an alternation of typo or morphology expansions) costs milliseconds to tens of milliseconds, so on a multi-segment index the query was dominated by recompiling the same automata. The regexes are now compiled in RegexPhraseQuery::regex_phrase_weight and shared through Arc.
An invalid pattern is now reported when the weight is built, instead of when the first segment is scored.
Avoid formatting keys and finalizing sub-aggregations for buckets
discarded by the final size limit. Preallocate the retained bucket vector
and reuse the cutoff helper with intermediate tuple keys.
Preserve final-key ordering and finalize the ordering metric when sorting
by a sub-aggregation. Cover mixed key types, empty sums, and discarded
document counts in tests.
Resolve retained string ordinals in sorted batches per field instead
of decoding dictionary blocks separately for each tuple component.
Reuse decoding for repeated ordinals while preserving tuple order
and existing numeric and missing-value handling.
Probe the accumulated map only for incoming keys rather than rehashing every accumulated key on each merge. This removes quadratic work when folding many disjoint multi-terms results. The same helper serves range and composite buckets; existing bucket values still merge left-to-right and the wire representation and pruning rules are unchanged.
Cover overlapping/disjoint tuple keys, empty inputs, recursive range subaggregations, postcard round trips, error/count bookkeeping, and merging after pruning.
Same benchmark and configuration as 0062d2f0d (Apple M4 Max, rustc 1.98.0):
Median milliseconds for 10 / 100 / 1000 inputs:
shared 8: 0.006365 / 0.0656 / 0.6449
disjoint 16: 0.0126 / 0.1486 / 1.6882
disjoint 160: 0.1336 / 1.5558 / 20.4019
At 1000 inputs this is 52.4x and 61.6x faster for the disjoint cases, with unchanged measured peak allocation. Shared-key control remains within approximately 2% of baseline.
Validation: 308 aggregation tests pass with default features and 308 with quickwit; changed-file rustfmt and git diff checks pass. Clippy --lib --bench agg_bench passes with the pre-existing clippy::drop_non_drop warning allowed (unchanged drop(add_document) in the benchmark).
The Str path and the non string path are very different.
This PR isolates them as different segment aggregation collector.
Co-authored-by: Paul Masurel <paul.masurel@datadoghq.com>
François Massot
nadav-govari
Paul MasurelandPaul Masurel
Pascal Seitz
Luca Cominardi
François Massot
Walter Woodall
palmoni5
dayaffe
David Yaffe